Methods and systems for improved user and / or patient experience in mammography

Abstract

The present invention is entitled methods and systems for improved user and / or patient experience in mammography. The present invention provides various methods and systems for breast positioning assistance during mammography and image-guided interventional procedures. In one example, a vision system is used to evaluate one or more of patient position, breast position, and breast anatomy to determine whether the patient and breast are adjusted to a desired position that is preferred for a desired view and imaging procedure. In addition, based on the evaluation, real-time feedback can be provided to guide the user in positioning the breast and / or the patient before acquiring an x-ray image.

Description

Technical Field

[0001] Embodiments of the subject matter disclosed herein relate to mammography and biopsy procedures, and more particularly, to breast positioning assistance for workflow and user / patient experience improvements during mammography and breast biopsy procedures. Background Art

[0002] Mammography is a medical imaging procedure used to detect one or more breast cancers. Accurate interpretation of mammographic images (also known as mammograms) and detection of breast cancer relies on the generation of high-quality mammograms. A key factor affecting the quality of mammograms is breast positioning. Failure to properly position the breast can result in mammographic artifacts and tissue rejection, and therefore missed cancers. The technician's training and experience level can significantly affect image quality. For example, a technician with less / moderate training and / or experience may not properly position the breast, and therefore, recall rates and missed cancers may be higher.

[0003] Furthermore, previous methods for position assessment involve technicians reviewing the acquired x-ray images. As a result, radiation doses are delivered to the patient even if the patient is not well positioned. Even during x-ray image review, some technicians may not correctly assess the x-ray images, which also increases recall and reduces confidence in the diagnosis. Summary of the invention

[0004] In one embodiment, a method for an x-ray mammography system includes: evaluating one or more of a patient position of a patient, a partial view of the patient, and a breast anatomy of the patient via visual sensing using one or more cameras coupled to the x-ray mammography system; detecting one or more of a patient positioning error and a breast positioning error based on the evaluation; and providing real-time feedback to a user via a user interface of the mammography system based on the detection. In this manner, the visual sensing system can be used to evaluate breast position in real time and provide feedback in real time to guide a technician to position the breast to achieve high quality images. By guiding the technician based on visual sensing, real-time guidance for correct positioning can be provided, which results in improved positioning and high quality images, and thus increased sensitivity for cancer detection.

[0005] It should be understood that the above brief description is provided to introduce in a simplified form selected concepts that are further described in the detailed description. It is not meant to identify key or essential features of the claimed subject matter, the scope of which is solely defined by the claims that follow the detailed description. Furthermore, the claimed subject matter is not limited to implementations that solve any disadvantages noted above or in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The present disclosure will be better understood by reading the following description of non-limiting embodiments with reference to the accompanying drawings, in which:

[0007] Figure 1A is a schematic diagram of a mammography system including a visual sensing system according to an embodiment of the present disclosure;

[0008] Figure 1B is a schematic diagram of a front view of a portion of a mammography system including a vision sensor for detecting one or more of breast position and breast morphology according to an embodiment of the present disclosure.

[0009] Figure 1C According to the embodiments of the present disclosure Figure 1B Schematic diagram of a perspective view of a mammography system including a vision sensor.

[0010] Figure 1D According to the embodiments of the present disclosure Figure 1B A schematic diagram of an enlarged perspective view of a portion of a gantry of a mammography system showing exemplary positioning of vision sensors for detecting one or more of breast position and breast morphology.

[0011] Figure 2 is a block diagram of a mammography system including a visual monitoring system according to an embodiment of the present disclosure, showing a plurality of objects sensed by the visual system and a plurality of actuators controlled based on the sensed objects;

[0012] Figure 3 is a high-level flow chart illustrating a method for assessing one or more of breast position, patient position, user position, and user morphology based on an operating mode according to an embodiment of the present disclosure;

[0013] Figure 4A is a high-level flow chart illustrating a method for assessing breast position and patient position during mammography and providing real-time feedback for adjusting breast position and patient position to improve mammographic image quality in accordance with an embodiment of the present disclosure;

[0014] Figure 4B yes Figure 4A Continuation of ;

[0015] Figure 5 is a high-level flow chart illustrating a method for assessing breast position and breast morphology during digital breast tomosynthesis (DBT) and providing real-time feedback for adjusting breast position and patient position to improve DBT image quality in accordance with an embodiment of the present disclosure;

[0016] Figure 6is a high-level flow chart illustrating a method for assessing breast position and patient position at selected angles within an angular range of an x-ray system during DBT imaging and providing real-time feedback for adjusting breast position and patient position to improve DBT image quality in accordance with an embodiment of the present disclosure; and

[0017] Figure 7 is a high-level flow chart illustrating a method for reducing glare from one or more camera images acquired using one or more visual sensors coupled to a mammography system in accordance with an embodiment of the present disclosure.

[0018] Fig. 8A is a diagram showing exemplary positioning of an x-ray source and a vision sensor for cranio-caudal (CC) view assessment of an x-ray mammography system according to an embodiment of the present disclosure.

[0019] Figure 8B is a diagram showing exemplary positioning of an x-ray source and a vision sensor for mediolateral oblique (MLO) view evaluation of an x-ray mammography system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] The following description relates to various embodiments of x-ray systems for use in mammography and biopsy procedures. Figure 1A An exemplary embodiment of an x-ray system is shown that includes a vision system that includes one or more cameras to detect one or more of accessories associated with each procedure, body parts involved in the procedure, one or more objects in the system's surroundings, a patient, and a user. Based on the detection, the x-ray system including the vision system can assess one or more of breast morphology, breast position, patient morphology, patient position, user morphology, and user position. Figure 1B An exemplary embodiment of a vision system for assessing breast position is shown. Specifically, the vision system may include a camera coupled to a gantry of an x-ray system such that the camera's field of view is aligned with the x-ray field of view. An exemplary position of the camera is Figure 1C and Figure 1D Further shown in . Figure 2 A block diagram of one or more objects and motions detected by a vision system including a camera, and one or more actuators of an x-ray system adjusted based on the detected one or more objects and motions is shown. A controller of an x-ray mammography system may be configured to evaluate one or more of breast position and breast morphology, patient position and patient morphology, and user position and user morphology based on an operating mode of the x-ray system, such as Figure 3 Specifically, the vision system may include at least a first vision sensor and a second sensor for assessing patient position and breast position, respectively. Fig. 8A and Figure 8B An exemplary position of the second visual sensor relative to the x-ray source for the CC view and the MLO view is shown in FIG. In addition, the controller may include instructions for evaluating the breast position and the patient position relative to the x-ray system and providing real-time feedback to the user during the mammography procedure, such as Figure 4A and Figure 4B In addition, Figure 5 and Figure 6 Exemplary methods for breast position and patient position assessment during DBT and real-time feedback for breast position adjustment are described. In addition, the controller may include instructions for reducing glare within a region of interest of one or more camera images obtained using a vision system, such as Figure 7 shown.

[0021] During imaging procedures (such as mammography or DBT imaging procedures), and during image-guided interventional procedures (such as DBT-guided biopsy, CESM biopsy, stereotactic biopsy, etc.), positioning the breast plays an important role in obtaining high-quality images showing various areas of the breast. In addition, depending on the view, some of the landmarks used for positioning may be different. Typically, before obtaining a mammogram, it is not possible to determine whether the breast is positioned to provide a high-quality mammogram. The inventors of this article have identified the above-mentioned problems and provide methods and systems for improving breast positioning before initiating acquisition. Specifically, methods and systems for evaluating breast position and patient body position before imaging, and providing real-time feedback for improving breast and patient position are provided. In one embodiment, a first visual sensor for evaluating the patient's body position captures an image of the patient and the x-ray system, and a second visual sensor for evaluating the breast position captures an image of the compressed breast. The camera images obtained from the first and second vision sensors are then input into an artificial intelligence (AI) based image processing model that evaluates the input camera images for a desired breast positioning framework (i.e., based on the view for inclusion of breast anatomical landmarks), and further evaluates patient position, and provides real-time feedback to the technician / user through a user interface for one or more of patient position and breast position correction. Details of evaluating one or more of patient position and breast position to improve breast position for imaging with a mammography system are further described below.

[0022] See also Figure 1A, a mammographic system 100 for performing mammographic procedures is shown, according to an exemplary embodiment, and includes an x-ray system 10. The x-ray system 10 may be a tomosynthesis system, such as a digital breast tomosynthesis ("DBT") system. The x-ray system 10 may be used to perform one or more procedures, including digital tomosynthesis imaging and DBT-guided breast biopsy. In addition, the x-ray system 10 may be used to perform mammographic imaging procedures in which one or more views including cranio-caudal (CC views) and medio-lateral oblique (MLO views) of the breast are obtained. The x-ray system may also be used to perform other x-ray screening and diagnostic imaging procedures, including CESM and contrast-enhanced DBT (CE-DBT) diagnostic imaging, as well as interventional procedures, including CESM-guided biopsy and stereotactic procedures.

[0023] The X-ray system 10 includes a support structure 42 to which a radiation source 16, a radiation detector 18, and a collimator 20 are attached. The radiation source 16 is housed in a gantry 15 that is movably coupled to the support structure 42. Specifically, the gantry 15 can be mounted to the support structure 42 so that the gantry 15 including the radiation source 16 can rotate about an axis 58 relative to the radiation detector 18. The rotation angle range of the gantry 15 that houses the radiation source 16 indicates rotation up to a desired angle in either direction around a vertical axis perpendicular to a horizontal detection surface of the detector 18. For example, the rotation angle range of the radiation source 16 may be -θ to +θ, where θ may cause the angle range to be a limited angle range of less than 360 degrees. An exemplary x-ray system may have an angle range of ±11 degrees, which may allow the gantry to rotate from -11 degrees to +11 degrees (i.e., rotation of the radiation source) around the rotation axis of the gantry. The angular range may vary according to manufacturing specifications. For example, the angular range of a DBT system may be approximately ±11 degrees to ±60 degrees, depending on manufacturing specifications.

[0024] The radiation source 16 is directed toward the volume or object to be imaged and is configured to emit radiation rays at a desired time and acquire one or more images. The radiation detector 18 is configured to receive the radiation rays via the surface 24. The detector 18 can be any of a variety of different detectors, such as an X-ray detector, a digital radiography detector, or a flat panel detector. The collimator 20 is disposed adjacent to the radiation source 16 and is configured to adjust the irradiation area of ​​the imaging object.

[0025] In some exemplary embodiments, the system 10 may also include a patient shield 36 mounted to the radiation source 16 via a mask rail 38 so that a body part (e.g., head) of the patient is not directly exposed to the radiation. The system 10 may also include a compression paddle 40 that may be capable of moving upward and downward relative to the support structure 42 along a vertical axis 60. Thus, the compression paddle 40 may be adjusted to be positioned closer to the radiation detector 18 by moving the compression paddle 40 downward toward the detector 18, and the distance between the detector 18 and the compression paddle 40 may be increased by moving the compression paddle 40 upward away from the detector along the vertical axis 60. The movement of the compression paddle 40 may be adjusted by a user via a compression paddle actuator (not shown) included in the X-ray system 10. The compression paddle 40 may hold a body part (such as a breast) in position against the surface 24 of the radiation detector 18. The compression paddle 40 may compress the body part and hold the body part stationary in position while optionally providing a hole to allow insertion of a biopsy needle, such as a core needle or a vacuum-assisted core needle. Thus, compression paddle 40 can be used to compress the body part to minimize the thickness through which x-rays pass and help reduce movement of the body part due to patient movement. X-ray system 10 can also include a subject support (not shown) on which the body part can be positioned.

[0026] The mammography system 100 may also include a workstation 43 including a controller 44 including a memory and at least one processor. The controller 44 may be communicatively coupled to one or more components of the x-ray system 10, including one or more of the radiation source 16, the radiation detector 18, the compression paddle 40, and the biopsy device. In one exemplary embodiment, communication between the controller and the x-ray system 10 may occur via a wireless communication system. In other exemplary embodiments, the controller 44 may be in electrical communication with one or more components of the x-ray system via a cable 47. Additionally, in one exemplary embodiment, as shown in FIG. Figure 1A As shown, the controller 44 is integrated into the workstation 43. In other exemplary embodiments, the controller 44 may be integrated into one or more of the various components of the system 10 disclosed above. In addition, the controller 44 may include processing circuitry that executes stored program logic, and may be any of a variety of computers, processors, controllers, or combinations thereof that are usable and compatible with various types of equipment and devices used in the X-ray system 10.

[0027] Workstation 43 may include radiation shielding 48 that protects an operator of system 10 from radiation rays emitted by radiation source 16. Workstation 43 may also include a display 56, keyboard 52, mouse 54, and / or other suitable user input devices that facilitate control of system 10 via user interface 50.

[0028] In addition, the x-ray system 10 may include a second control station (not shown) including a second user interface having a second display portion with appropriate input features to facilitate control of the system 10 and viewing one or more images captured by one or more of the vision system and the x-ray system 10. The second control station may be positioned proximate to the x-ray system and may be coupled (wired or wirelessly) to the x-ray system 10. Specifically, the second control station may be positioned so that a user may view the second display portion and / or the second user interface while adjusting the breast and / or patient position. Thus, the positioning of the second control station may allow the user to simultaneously view real-time camera feedback and adjust the patient and / or breast position. Figure 1B and Figure 1C An exemplary second control station is shown.

[0029] Through its processor and controller, the controller 44 can regulate the operation and function of the X-ray system 10. For example, the controller 44 can provide timing control regarding when the X-ray source 16 emits X-rays, and can also regulate how the detector 18 reads and transmits information or signals after the X-rays hit the detector 18, and how the X-ray source 16 and the detector 18 move relative to each other and relative to the body part. The controller 44 can also control the manner in which information (including images 42 and data acquired during operation) is processed, displayed, stored, and manipulated. The different processing steps performed by the controller 44 include receiving one or more signals from one or more sensors, receiving user input, evaluating the received signals / inputs, image processing, determining reconstruction errors, outputting operating parameters including error indications, and adjusting one or more actuators of the x-ray system to control the operation of the x-ray system, which can be provided by a set of instructions stored in a non-transitory memory of the processor. Information can also be stored in one or more non-transitory memories of the controller 44 for later retrieval and use.

[0030] In addition, as described above, the radiation detector 18 receives radiation rays emitted by the radiation source 16. Specifically, during imaging using the X-ray system, a projection image of the imaged body part can be obtained at the detector 18. In some exemplary embodiments, data received by the radiation detector 18 (such as projection image data) can be transmitted from the radiation detector 18 to the controller 44 electronically and / or wirelessly. The controller 44 can then reconstruct one or more scanned images based on the projection image data, for example by implementing a reconstruction algorithm. The reconstructed image can be displayed to the user on the user interface 50 via the display screen 56.

[0031] The radiation source 16 together with the radiation detector 18 form part of an X-ray system 10 that provides X-ray images for the purpose of screening for abnormalities, diagnosis, dynamic imaging, and one or more of image-guided biopsies. For example, the X-ray system 10 can be operated in a mammography mode to screen for abnormalities. During mammography, the patient's breast is positioned and compressed between the detector 18 and the compression paddle 40. Therefore, the volume of the X-ray system 10 between the compression paddle 40 and the detector 18 is the imaging volume. The radiation source 16 then emits radiation rays onto the compressed breast, and a projection image of the breast is formed on the detector 18. The projection image can then be reconstructed by the controller 44 and displayed on the interface 56 via the display portion 50.

[0032] During mammography, gantry 15 may be adjusted at different angles to obtain images of different orientations, such as craniocaudal (CC) images and mediolateral oblique (MLO) views. Additionally, during acquisition of mammographic views (CC and MLO views), gantry 15, compression paddles 40, and detector 18 may be rotated as a single unit about axis 58. In other examples, gantry 15 may be rotated about axis 58 while compression paddles 40 and detector 18 remain stationary.

[0033] In addition, the X-ray system 10 can be operated in a tomosynthesis mode for performing digital breast tomosynthesis (DBT). During tomosynthesis, the X-ray system 10 can be operated to direct low-dose radiation to the imaging volume (between the compression paddle 40 and the detector 18) at various angles within the angular range of the X-ray system 10. During tomosynthesis, similar to mammography, the breast is compressed between the compression paddle 40 and the detector 18. The radiation source 16 is then rotated from -θ to +θ, and multiple projection images of the compressed breast are obtained at regular angular intervals within the angular range. For example, if the angular range of the x-ray system is ±11 degrees, the detector can capture 22 projection images approximately every degree of sweep during the angular sweep of the gantry. The multiple projection images are then processed by the controller 44 to generate a plurality of DBT image slices. The processing may include applying one or more reconstruction algorithms to reconstruct a three-dimensional DBT image of the breast.

[0034] Additionally, the x-ray system 10 may be configured to perform a DBT-guided biopsy procedure. Thus, in some exemplary embodiments, the system 10 may further include a biopsy device (not shown) including a biopsy needle for extracting a tissue sample for further analysis.

[0035] In an exemplary embodiment, the biopsy device may include a biopsy table (not shown) positioned above the detector 18 of the x-ray system 10. For example, the biopsy table may be configured to slide above the detector 18. During the setting of the biopsy device, the user may remove the compression paddle 40 of the x-ray system 10 and slide the biopsy table above the detector 18. When the biopsy device is positioned on the x-ray system 10, a suitable compression paddle (not shown) for biopsy, such as a horizontal approach biopsy paddle (without a hole) or a vertical approach biopsy paddle (with a hole), may be selected according to the type of biopsy and coupled to the x-ray system 10.

[0036] The biopsy device may also include a biopsy tool interface having a biopsy tool display. The biopsy tool interface may be coupled to the biopsy table via a communication port. In one embodiment, the biopsy tool interface may be communicatively coupled to the x-ray system controller 44 so that a user may be able to adjust the position of the x-ray system, such as adjusting the gantry to a parked position, via the biopsy tool interface. In other embodiments, the biopsy tool interface may be coupled to a biopsy device controller that sends information to and receives information from the x-ray system controller 44. In some other embodiments, in addition or alternatively, adjustment and control of the biopsy device may be performed by a biopsy device control module of the x-ray system controller 44.

[0037] The biopsy device may include a biopsy tool that can be directly coupled to the biopsy table. The biopsy tool may include a biopsy gun holder for mounting a biopsy gun. In addition, the biopsy gun holder may include a mechanical stop for adjusting the position of the biopsy needle. The biopsy needle may include an outer sleeve, an inner sleeve positioned in the outer sleeve, and an opening for receiving a portion of tissue from a biopsy lesion or target. The sleeves form a cutting device, wherein the outer sleeve is configured to slide or rotate above the inner sleeve, and / or the inner sleeve is configured to slide or rotate within the outer sleeve.

[0038] Specifically, before the needle is inserted, the breast is positioned between a compression paddle (not shown) and the top surface of the table. In some examples, a breast pad can be positioned on the surface, the breast is positioned between the compression paddle and the pad and is compressed by moving the compression paddle toward the surface. When the breast is positioned, a first set of target images is obtained by scanning the compressed breast with the X-ray system 10 at various angles within its angular range to identify the target for biopsy. The first set of target images can be a three-dimensional image (DBT image) or a two-dimensional full-field digital mammography image acquired and reconstructed by the X-ray system. The user can locate the area of ​​interest and identify the target position for biopsy by selecting the target position from the first set of images. The target position can be identified by the x-coordinate, y-coordinate, and z-coordinate within the DBT volume between the compression paddle and the biopsy table surface or pad (if used). Based on the target position coordinates selected by the user, the biopsy device controller can adjust the position of the mechanical stop of the biopsy gun holder so that when the needle is inserted into the compressed breast via the biopsy gun, the needle stops moving when the needle tip reaches a desired position relative to the target position (referred to as a pre-fire position). Although the present example illustrates adjusting the biopsy device via the biopsy device controller, it should be understood that in some embodiments, the x-ray system controller 44 may command control of the biopsy device.

[0039] Once the biopsy tool and biopsy gun are in the target position, the user / radiologist can drive the needle through the biopsy gun until it reaches the mechanical stop. Once fully inserted, the needle is in the target position (i.e., the position where the notch of the needle is in front of the lesion for puncture). Subsequently, a second set of images is obtained when the biopsy needle is in the pre-fired position. The user can then initiate the firing of the biopsy needle via the biopsy gun. Once the biopsy needle is fired, at least one biopsy sample can be removed from a body part (such as a patient's breast) by using suction and / or a cutting mechanism formed by an inner sleeve and an outer sleeve. The sample is moved along the suction tube by suction, and the suction tube is connected to a collection chamber with a separate pre-marked chamber to depict the order or position of each sample from the biopsy procedure. An alternative method of marking each sample to allow for position identification and / or order identification can also be adopted. In addition, after the needle is fired, a third set of images can be obtained when the needle is in the post-fired position.

[0040] The mammography system 100 may further include one or more visual sensors for sensing one or more components and accessories of the mammography system 100. The one or more visual sensors may include a first visual sensor 101 and a workstation visual sensor 102, such as Figure 1AAs shown. The first visual sensor 101 can be configured to sense one or more components and accessories associated with the x-ray system 10. In addition, any of the first visual sensors 101 can be configured to sense the form and movement of one or more users and patients, while the workstation visual sensor 102 can be used to monitor the user's position and / or movement at the workstation. Although this example shows two cameras for implementing visual sensing for the mammography system 100, it should be understood that the visual system may include additional cameras or fewer cameras, as further described below. In addition, one or more visual sensors may include a second visual sensor ( Figure 1A The second visual sensor may be used to capture a camera image of the compressed breast in the imaging volume. Thus, the camera image obtained using the second visual sensor may be used to assess the breast position before acquiring the x-ray projection image. Figure 1B , Figure 1C and Figure 1D Details of the position and adjustment of the second visual sensor for breast position assessment are described in.

[0041] Figure 1B An exemplary embodiment of a vision system is shown that includes a camera for monitoring breast position and breast morphology relative to the field of view of an x-ray system 150. Specifically, a vision sensor system 153 including a second vision sensor 154 is shown. The x-ray system 150 is similar to Figure 1A The x-ray system 10 described above is described, and therefore, for the sake of brevity, descriptions of similar components and elements will not be repeated here. In short, the x-ray system 150 includes a gantry 158, which includes a radiation source 160, a radiation detector 168, and a collimator (not shown) coupled to a support structure 170. The x-ray system 150 also includes a compression paddle 164, which is used to maintain a body part, such as a breast, in an appropriate position against the top surface of the radiation detector 168. The compression paddle 164 can be coupled to a support rail 172 of the gantry 158 and can be moved upward and downward along the support rail in a direction away from and toward the radiation detector 168. The movement of the gantry 158 and the compression paddle 164 can be based on a controller (not shown) (such as, for example, a controller) that is communicatively coupled to the x-ray system 150 via corresponding actuators. Figure 1A The rack 158 can be rotated clockwise and counterclockwise to a desired degree about the vertical axis of the support structure. The rotational movement of the rack 158 is indicated by the double-ended arrow 186. In addition, the rack height can be adjusted by moving the rack 158 vertically upward and downward, as indicated by the double-ended arrow 185.

[0042] The second visual sensor 154 may be coupled to the gantry 158 such that the field of view of the second visual sensor 154 is aligned relative to the field of view of the x-ray system 150, specifically relative to the radiation source located within the gantry 158. The second visual sensor 154 may be used to monitor the patient's breast position and breast anatomical landmarks during a procedure performed by the x-ray system 150, such as mammographic imaging, DBT imaging, or image-guided biopsy. In one example, the second visual sensor 154 may be configured as an RGB-D camera that combines depth information with RGB color information. In addition, the second visual sensor 154 may be communicatively coupled to a controller of the x-ray system 150, and one or more camera images and camera image sequences captured by the second visual sensor 154 may be stored in a non-transitory memory of the controller. In addition, the second visual sensor 154 may be configured to capture movement and / or action. For example, a camera image sequence over a duration may be obtained, which may be used for action recognition.

[0043] In some embodiments, the second visual sensor 154 can be positioned so as to monitor and assess a local portion of the patient in addition to monitoring and assessing the position of the compressed breast. That is, the second visual sensor can also be used to capture a local portion of the patient (e.g., the patient's shoulder) and thus provide a local view of the patient in addition to providing a view of the compressed breast.

[0044] It should be appreciated that in some embodiments, the second visual sensor may be positioned such that an imaging volume including the compression paddles, the detector, and the compressed breast is visualized via the second visual sensor 154 .

[0045] Taken together, the second visual sensor 154 can be adjusted (e.g., by adjusting the position of the visual sensor 154) to view and / or capture one or more of the imaging volume (including the compression paddles, the detector, and the compressed breast), the imaging volume and the edge volume, and the imaging volume and a localized portion of the patient.

[0046] In addition, the x-ray system 150 may include a second control station 194 including a second user interface 195 having a second display portion 196 having appropriate input features to facilitate control of the system 150 and viewing one or more images captured by one or more of the vision system 153 and the x-ray system 150. The second control station 194 may be positioned proximate to the x-ray system 150 and is shown coupled to the x-ray system 10. As described herein, real-time feedback for adjusting one or more of patient position, breast position, and x-ray system position (e.g., gantry position) may be provided to the user via the second control station 194. That is, provided to the user via the second display 196 of the second user interface 195. As such, the user may easily view feedback via the second display portion 195 of the second user interface 195 while adjusting the breast and / or patient position. Thus, positioning the second control station 194 proximate to the x-ray system 150 may allow the user to simultaneously view real-time camera feedback and adjust the patient and / or breast position.

[0047] Although the present example shows the second control station 194 positioned on one side of the x-ray system 150, a third control station similar to the second control station 194 may be positioned on the opposite side of the x-ray system. In addition, the second control station 194 may be removably coupled to the x-ray system 150. Thus, when desired, the second control station may be removed and positioned on the opposite side.

[0048] In addition, in one example, the camera images captured by the second visual sensor 154 may be pre-processed to reduce glare and extract relevant image data, and the processed image data may be used as input to an artificial intelligence-based deep learning model (which is stored in the controller, in an edge device connected to the controller, in a cloud in communication with the controller, or in any appropriate combination thereof), the artificial intelligence-based deep learning model including a neural network such as a convolutional neural network that is used to assess breast position based on breast anatomical landmarks detected during positioning the breast for various imaging and image-guided procedures utilizing the x-ray system 150. Although some of the breast anatomical landmarks (also referred to as breast structures) may vary depending on the view (e.g., CC or MLO view) and procedure (e.g., mammography or DBT), there may be some common breast anatomical landmarks or breast structures that are evaluated for breast positioning at each view. As described below in Figures 2 to 6 Details of assessing breast position based on anatomical landmarks identified or detected using the second vision sensor 154 are further discussed in.

[0049] Additionally, second visual sensor 154 may be positioned on gantry 158 such that a field of view of second visual sensor 154 includes compression paddle 164 and an imaging volume between compression paddle 164 and detector 168. One or more images from the camera may be used to detect edges of the compression paddle, and detect one or more of contours, openings, depressions, bends, and textures on portions of the compression paddle between the edges of the compression paddle. Based on the detected compression paddle properties, the vision system may determine the type of compression paddle used with the mammography system.

[0050] Relative to Figure 1C Further illustrated is an exemplary positioning of the second visual sensor 154. Specifically, Figure 1C A perspective view of x-ray system 150 is shown. A second visual sensor 154 is positioned on the housing of radiation source 160 via a pair of rails such that the camera can be moved along axis 155 in forward and rearward directions as indicated by arrow 155. Figure 1D An enlarged portion 190 of an x-ray system including a visual sensor 154 is shown. The second visual sensor 154 is mounted on a pair of rails 157 and is movable along an axis 155 and is rotatable about an axis 156 that is perpendicular to the axis 155. In one example, the camera position can be adjusted via rotation along 156 and / or movement along 155 so that the field of view of the first camera 154 is aligned with the field of view of the radiation source 160. In one example, the second visual sensor 154 can be integrated with the x-ray system 150. In other examples, the visual system 153 can be a temporary system that can be removably coupled to the x-ray system when needed.

[0051] Return to Figure 1A , the one or more visual sensors are communicatively coupled to the controller 44. Figure 2 Details of various components of the mammography system 100 that may be sensed and monitored by the first vision sensor 101 and the workstation vision sensor 102 are further elaborated, as well as various actuators of the mammography system 100 that may be adjusted in response to the sensing and monitoring by the camera.

[0052] Go to Figure 2, a block diagram of a mammography system 200 is shown. The mammography system 200 may be a non-limiting example of the mammography system 100 of FIG. 1 . In short, the mammography system 200 may be used to perform one or more of mammography (such as conventional mammograms), digital breast tomosynthesis, and biopsy procedures (such as stereotactic biopsy or DBT-guided biopsy). The mammography system 200 may include a medical vision system 250 having at least a visual sensor system 252 and a processing system 256 to perform one or more of sensing, monitoring, and analysis of one or more accessories and components associated with the mammography system 200. In addition, the mammography system 200 may be automatically adjusted, controlled, and set via the processing system 256 based on input from the visual sensor system 252 and analysis of the input by the processing system 256 to improve workflow, breast positioning, and patient positioning during one or more of mammography and biopsy procedures performed using the mammography system 200. The processing system 256 may be a non-limiting example of the controller 44 of FIG1 and may be configured to receive signals from one or more sensor systems (including the visual sensor system 252) of the mammography system 200, as further described below. In addition to providing real-time feedback including one or more alerts and indications to one or more of a user and a patient via a user interface 286 of the mammography system 200, the processor may be further configured to analyze the data received from the sensor systems and adjust the operation of the mammography system 200 via one or more x-ray system actuators 276, as further described below.

[0053] Mammography system 200 may include x-ray system 210 and medical vision system 250. X-ray system 210 may be Figure 1A The x-ray system 10 or Figure 1B 210. In one exemplary embodiment, the x-ray system 210 may be configured as a medical imaging modality for performing a mammography procedure to image and analyze a body part of a patient, such as a breast. In another exemplary embodiment, the x-ray system 210 may be configured to perform a biopsy procedure, such as an x-ray guided biopsy, to obtain a tissue sample from a body part of a patient. In addition, the x-ray system 210 may be converted from a mammography system for obtaining medical scan images to a biopsy system for performing a biopsy procedure for extracting tissue for evaluation. When the x-ray system 210 is used to perform a biopsy procedure, a biopsy device 212 may be coupled to the x-ray system 210. The biopsy device 212 may be Figure 1AExamples of biopsy devices are described, and may include a biopsy table, a biopsy tool, a biopsy tool interface, a biopsy gun holder, and a biopsy gun. In addition, the biopsy device 212 may include one or more biopsy device accessories 214 coupled to one or more of the x-ray system 210 and the biopsy device 212. The one or more biopsy device accessories 214 may include: a needle 216 for entering a body part and extracting a tissue portion for further analysis by a clinician; an adapter 218; a biopsy pad 220 for use as a support for the body part and for obtaining a desired positioning of the body part relative to the needle 216 during a biopsy; a compression paddle 222 for supporting the body part and holding the body part to reduce movement during a biopsy; and one or more phantoms 224 for performing a quality check prior to a mammogram or biopsy. The compression paddle 222 is relative to Figure 1A The described compression paddle 40 or Figure 1B An example of compression paddle 164 in FIG. The mammography system may be configured to monitor one or more of a component (such as biopsy device 212 ), different components of biopsy device 212 , and accessories (such as biopsy device accessories 214 ) via medical vision system 250 .

[0054] In one exemplary embodiment, vision system 250 may be used to detect the presence of compression paddle 222. Additionally, upon detecting the compression paddle, vision system 250 may generate one or more images of compression paddle 222. The one or more images of compression paddle 222 may then be used to identify / classify the type of compression paddle based on one or more attributes of compression paddle 222.

[0055] The one or more attributes of the compression paddle 222 may include: the distance between the left and right edges of the paddle, the presence or absence of one or more openings / recesses on the top surface between the left and right edges of the paddle, the presence or absence of a texture difference in the middle of the paddle, the presence or absence of one or more bends on the surface between the left and right edges of the paddle, and the shape of the paddle. Based on the detected attributes, the vision system 250 may identify the type of compression paddle attached to the system. For example, the side edges of the compression paddle may be identified, and the distance between the edges may be calculated. In addition, based on the one or more images of the compression paddle, the vision system may determine whether one or more other attributes of the surface of the compression paddle between the edges may be detected, such as a contour, an opening, a recess, a bend, and / or a texture. The edge distance and other attributes of the compression paddle may be used to identify the type of compression paddle currently used with the mammography system. In some examples, based on the identified type of compression paddle, the vision system 250 may determine an operating mode. For example, if a compression paddle specific to biopsy (e.g., a biopsy paddle) is detected, the vision system may determine that the user intends to operate the mammography system in a biopsy mode.

[0056] Furthermore, in addition or alternatively, the visual system 250 may be used to monitor changes in the position of the compression paddle 222. For example, during a mammography or biopsy examination, the user may move the compression paddle in order to adjust the position of the breast. The visual system 250 may detect the movement of the compression paddle 222, and after compression, based on the changed compression paddle position, the controller may command adjustment of the collimator so that the breast is within the field of view of the x-ray system.

[0057] Furthermore, in some embodiments, in addition to or as an alternative to the user confirming the compression paddle position via the user interface, the final (locked) position of the compression paddle may be determined using a visual system. The final compression paddle position may indicate that the breast is in a position for imaging. In one example, upon confirming the final position of the compression paddle, the breast position of the compressed breast may be assessed, and therefore, a breast position assessment interface may be initiated and a breast position assessment may be performed using the visual system 250. Upon confirming that the compressed breast is in a desired position for imaging using the visual system 250, the mammography system may automatically start x-ray image acquisition of the compressed breast. In another example, upon confirming the final position of the compression paddle, one or more of the breast position and the patient position may be assessed using the visual system 250. Upon confirming that one or more of the breast position, the patient position, and the user position are in a corresponding desired position for imaging, the mammography system may automatically start x-ray image acquisition.

[0058] Furthermore, in some embodiments, in addition to assessing one or more of breast position, patient position, and user position, after x-ray image acquisition, the acquired image may be further assessed to determine that a plurality of anatomical landmarks of the breast are captured in the x-ray image. Thus, in addition to assessing breast position using a visual sensor prior to initiating image acquisition, a breast position assessment may be performed on the acquired x-ray image. In this way, a second assessment of breast position after image acquisition may be utilized to determine whether the desired tissue portion is imaged, and thus the number of patient recalls may be reduced.

[0059] The mammography system 200 may also be configured to monitor an environment 226 surrounding the x-ray system 210 using a medical vision system 250. The environment 226 may include one or more of a workstation 228, a specimen 230 (such as a body part) for imaging, a wheelchair 232 according to the needs of the patient, a biopsy couch 234 according to the type of biopsy performed and the physiological condition of the patient, a patient 236, and a user 237. In addition, the mammography system 200 may be configured to monitor one or more of a process, movement, and motion relative to the x-ray system 210 and the environment 226 via the medical vision system 250. The movement and / or motion may include patient movement, user movement, and movement of one or more accessories of the x-ray system.

[0060] As described above, the medical vision system 250 includes a vision sensor system 252, which includes one or more cameras 254 and an image processing system 256, which includes a processor 258 and a non-transitory memory 260. The vision sensor system 252 can be communicatively coupled to the image processing system 256. In particular, the processing system 256 can receive one or more signals from the one or more cameras 254 of the vision system. The one or more cameras of the vision system 252 can be similar to those with respect to Figure 1A The cameras 101 and 102, and therefore, the one or more cameras 254, can sense the mammography system 200 and its components, accessories, and environment. Data from the one or more cameras 254 can be sent to a processing system 256 for further analysis and storage.

[0061] In an exemplary embodiment, the visual sensor system 252 including one or more cameras 254 may include two visual sensors (two cameras 101 and 102), such as Figure 1A In another exemplary embodiment, the visual sensor system 252 may include a Figure 1B Another vision sensor of the x-ray system 150 , such as a second vision sensor 154 .

[0062] The processing system 256 includes a processor 258 configured to execute machine-readable instructions stored in a non-transitory memory 260. The processor 258 may be single-core or multi-core, and the program executed thereon may be configured to perform parallel or distributed processing. In some embodiments, the processor 258 may optionally include separate components distributed in two or more devices, which may be remotely located and / or configured for coordinated processing. In some embodiments, one or more aspects of the processor 258 may be virtualized and executed by a remotely accessible networked computing device configured with a cloud computing configuration. According to other embodiments, the processor 258 may include other electronic components capable of performing processing functions, such as a digital signal processor, a field programmable gate array (FPGA), or a graphics board. According to other embodiments, the processor 258 may include multiple electronic components capable of performing processing functions. For example, the processor 258 may include two or more electronic components selected from a list of electronic components, including: a central processing unit, a digital signal processor, a field programmable gate array, and a graphics board. In other embodiments, the processor 258 may be configured to include a graphics processing unit (GPU) having a parallel computing architecture and parallel processing capabilities. The non-transitory memory 260 may store a neural network module 262, camera image data 264, an accessory monitoring module 266, a user input monitoring module 268, an environmental monitoring module 270, a workflow monitoring module 272, and a patient monitoring module 274. The neural network module 262 may include a deep learning module including a plurality of parameters (including weights, biases, activation functions) and instructions for implementing the one or more deep neural networks to receive image data from the visual sensor system 252, identify one or more objects corresponding to one or more of x-ray system components and accessories, further identify one or more environmental parameters, and further identify one or more processes and actions related to one or more of mammography and biopsy. For example, the neural network module 262 may store instructions for implementing the deep learning module including one or more neural networks, such as a convolutional neural network (CNN). The neural network module 262 may include trained and / or untrained neural networks, and may also include various data or metadata related to the one or more neural networks stored therein. Non-transitory memory 260 may also store a training module (not shown) that includes instructions for training one or more of the deep neural networks stored in neural network module 262. Training may be performed using a training data set that includes camera images of a compressed breast, camera images of a phantom, and camera images reconstructed from x-ray images of a compressed breast.

[0063] In addition, using the input from the vision sensor system 252, the deep learning module can identify breast position, breast anatomical landmarks, patient position, patient morphology, user position, and user morphology. In addition, using the input from the vision sensor system 252, the deep learning module can evaluate the breast position as well as the patient position, and determine one or more errors based on this evaluation. The one or more errors may include breast positioning errors, patient positioning errors, and user errors, and control the mammography system based on the errors (e.g., prevent image acquisition in response to detecting one or more errors) and provide real-time feedback based on the detected errors (e.g., when the breast is positioned, the vision sensor can obtain one or more images, which can be used to analyze the breast position and morphology, and provide real-time feedback based on this analysis).

[0064] In one example, the first deep learning model may include parameters of the expected patient position, including the whole-body patient position and the patient position relative to the x-ray system, where the expected patient position is based on the imaging mode of the x-ray system and the expected view to be acquired by the imaging system. In addition, the second deep learning model may include parameters of the expected breast position, which include one or more compressed breast features included in the imaging volume of the x-ray system, where the expected breast position is based on the imaging mode of the x-ray system and the expected view to be acquired by the imaging system. Before initiating x-ray image acquisition, the vision sensor system 252 can be used to evaluate the current patient position (relative to the first model) and the current breast position (relative to the second model), and provide real-time feedback to the user via the user interface based on this evaluation. When the current patient position is consistent with the first model and the current breast position is consistent with the second model, x-ray image acquisition can be initiated. The details of evaluating the breast position and patient position before image acquisition will be further elaborated in detail with respect to Figures 2 to 8B the method described below.

[0065] The non-transitory memory 260 can also store camera image data 264. The camera image data 264 can include images captured by the vision sensor system 252. For example, the images captured by the vision sensor may include: one or more mammography systems including the x-ray system 210 (including its components and accessories), the environment 226, and images of the processes and / or actions associated with the x-ray system 210 and the environment 226. The camera image data 264 can also include patient monitoring images, user monitoring images, and compressed breast images.

[0066] The non-transitory memory 260 can also store an attachment monitoring module 266, which includes instructions for monitoring and analyzing the presence and current position of one or more attachments 214 and the biopsy device 212.

[0067] The non-transitory memory 260 may also store a user input monitoring module 268 that includes instructions for monitoring and analyzing user input made via the user interface.

[0068] The non-transitory memory 260 may also store an environment monitoring module 270 including instructions for monitoring and analyzing the environment 226, and may store a workflow monitoring module 272 including instructions for monitoring and analyzing one or more processes and actions 238. In addition, the non-transitory memory 260 may store a patient monitoring module 274 for monitoring and analyzing one or more of patient presence, patient location, and patient movement into and out of an examination room. In addition, the non-transitory memory 260 may store a user monitoring module for monitoring and analyzing one or more of user presence, user location, and user movement into and out of an examination room.

[0069] The non-transitory memory 260 may also store medical image data 275. The medical image data 275 may include a scanned image of a body part captured by the x-ray system 210.

[0070] Upon sensing and analyzing one or more of the x-ray system 210, the environment 226, and the processes and actions 238, the image processing system 256 may output instructions based on the sensing and analysis to one or more x-ray system actuators 276. The x-ray system actuators 276 may include an image acquisition actuator 278 for controlling the radiation output from a radiation source (such as the radiation source 16 in FIG. 1 ), a gantry motion actuator 280 for controlling the gantry position of the x-ray system 210, and a bed position actuator for adjusting the biopsy bed position, for example, based on the presence or absence of an object such as a wheelchair 232 in the environment 226. The gantry motion actuator 280 may include one or more actuators for adjusting one or more of gantry elevation, gantry rotation, and gantry angulation, wherein gantry elevation motion includes movement of the gantry in an upward or downward direction along a vertical axis of the x-ray system 210, gantry rotation is rotation of the detector and x-ray generating tube about an axis of rotation, and gantry angulation is rotation of the x-ray tube while the detector remains within an angular range of rotation.

[0071] The x-ray system actuator 276 may also include a biopsy device actuator for adjusting the operation of the biopsy device, such as firing a biopsy needle, based on, for example, one or more inconsistencies between the sensed user input and the actual x-ray system configuration, as described in further detail below. The x-ray system brake 276 may also include a compression paddle brake 285 for adjusting the movement of the compression paddle 222.

[0072] Additionally, upon sensing and analyzing one or more of the x-ray system 210, breast position, patient position, environment 226, and process and action 238, the image processing system 256 may output one or more alerts, including real-time feedback, via the user interface 286. The user interface 286 may be Figure 1A 286. The one or more alerts output by the processing system 256 via the user interface 286 may include one or more of a visual alert 288 and an audible alert 290. Other types of alerts, such as tactile alerts, are also within the scope of the present disclosure. In addition, the processing system 256 may be configured to update an image acquisition input 292 on the user interface 286 and adjust one or more of the x-ray system settings, configurations, and operations accordingly. In addition, the processing system may be configured to update an attachment information input 294 on the user interface 286 and adjust the x-ray system settings, configurations, and operations accordingly.

[0073] It should be understood that Figure 2 The image processing system 256 shown is for illustration and not limitation. Another suitable image processing system may include more, fewer, or different components.

[0074] Go to Figure 3 , shows a high-level flowchart illustrating a method 300 for assessing one or more of breast position, breast anatomy, patient position, patient morphology, user position, and user morphology. Specifically, prior to initiating image acquisition, an imaging system (such as, Figure 1A Method 300 is implemented during operation of an x-ray system to assess the position of a body part to be imaged relative to the x-ray imaging system and the position of a patient relative to the x-ray system. Method 300 may be implemented by an image processing system such as Figure 1A The method may be implemented by a controller 44 in the image processing system, an edge device connected to the image processing system, a cloud communicating with the image processing system, or any suitable combination thereof. Figures 1A to 1D The method 300 is described with reference to the systems and components of the present invention, but it should be understood that the method 300 can be implemented with other systems and components without departing from the scope of the present disclosure. In addition, although the positioning relative to the breast and relative to Figures 1A to 1D The x-ray mammography system described herein describes method 300 and other methods herein, but it should be understood that the methods and systems described herein for positioning assessment (of body parts and patients) can be implemented using other imaging modalities (such as, x-ray based imaging modalities including CT, DXA, SPECT, etc.) and other modalities (such as, MRI, etc.).

[0075] The method 300 starts at 302. At 302, the method 300 includes determining an operating mode of an x-ray imaging system. The operating mode may be a current operating mode that a user (i.e., a technician) intends to perform with the x-ray system. In one example, the operating mode may be determined based on user input. In another example, a visual sensing system may be used to determine the operating mode based on one or more accessories detected by the visual system.

[0076] Next, at 304, method 300 includes confirming whether the current operating mode is any one of a quality check (QC) mode and a cleaning mode. If the answer at 304 is yes, method 300 proceeds to 305. At 305, method 300 includes evaluating one or more of a user position and a user form. The user position includes the position of the user relative to the x-ray system. The user position may also include the movement of the user from one position to another within a time frame, the user's posture (e.g., sitting, standing, etc.). The user position may be relative to a reference axis of the x-ray system (e.g., Figure 1A The vertical axis 60 in the image is determined and can be based on information from a first visual sensor (such as, Figure 1A The first visual sensor may be, for example, a three-dimensional depth sensing camera. In one embodiment, the visual sensor may not be directly coupled to the x-ray system, and may be positioned (e.g., within a room housing the x-ray system) such that when the user is within a threshold perimeter of the x-ray system, the user and the x-ray system are in the field of view of the visual sensor. The threshold perimeter may be based on the size of the examination room in which the x-ray system is located. In another embodiment, the first visual sensor may be coupled to the x-ray system to estimate and monitor the user's position and morphology. Although this example depicts one visual sensor for user position and user morphology assessment, it should be understood that two or more visual sensors may be used. For example, the visual sensor 101 and the workstation visual sensor 102 may be used to monitor user morphology and position.

[0077] The input from the first visual sensor can be used by the visual sensor data processor to perform tracking of one or more users in the field of view of the visual sensor and evaluate the user position based on the tracking. In one example, the input (e.g., depth information) can be used to perform skeletal tracking, in which multiple joints of the user are identified and analyzed to determine the user's movement, posture, position, etc. For example, the position of the joints during skeletal tracking can be used to determine the above-mentioned user parameters.

[0078] In addition to the position of the user relative to the x-ray system, the first visual sensor can also be used to estimate user morphological parameters, including the user's height. In one example, similar to the determination of the user's position, the user's height can be estimated based on the skeletal tracking data. In another example, the highest point and the lowest point of a plurality of point clouds (data sets representing the user) of the user can be used to estimate the user's height. The point cloud can be obtained from one or more of a two-dimensional image and a three-dimensional image obtained using the visual sensor.

[0079] Next, method 300 proceeds to 306. At 306, method 300 includes adjusting the x-ray system for QC or cleaning based on the user's position and morphology. In one example, the controller may adjust one or more features of the x-ray system for cleaning. The one or more features may include the compression paddle holder position, the detector position, the gantry position, and the control station (also referred to herein as the workstation) position on the gantry rail. The above features may be adjusted to the corresponding cleaning position based on the user's height for faster and more effective cleaning. That is, the x-ray system may be adapted to fit the user for QC and / or cleaning based on the user's position and the user's morphology. As described above, based on the camera image of the user from the visual system, the controller may assess the user's height and automatically adjust the position of the compression paddle and / or detector based on the user's height for QC and / or cleaning. Therefore, the efficiency of the procedure is improved. In addition, the user's position may be used to determine the order of adjustment. For example, when it is detected that the user is closer to the x-ray system, the gantry position and the compression paddle position may be adjusted for cleaning, and the control station position may be adjusted when the user moves closer to the control station.

[0080] If a QC procedure is to be performed, the x-ray system may adjust the one or more features including compression paddle position, gantry position, and workstation position based on the user's height and position. After adjusting one or more features of the x-ray system based on the user's position and morphology, method 300 ends.

[0081] Returning to 304, if the QC or cleaning mode is not confirmed, the method proceeds to 307. At 307, the method 300 includes confirming whether the x-ray system is used to perform a mammography, DBT, or image-guided biopsy procedure. That is, the method 300 includes determining whether the desired operating mode of the x-ray system is a mammography mode, a DBT mode, or an image-guided biopsy mode.

[0082] In one example, an operating mode of the x-ray system may be determined based on an indication made by a user on a user interface of the x-ray system. In one example, a controller may determine the operating mode based on a user launching an application interface corresponding to the operating mode. For example, determining the operating mode based on the user indication may include determining an image-guided biopsy mode in response to a user launching an image-guided biopsy interface, determining a DBT mode in response to a user launching a DBT imaging interface, and determining a mammography mode in response to a user launching a mammography interface. It should be understood that other user indication modes (such as a user selecting a corresponding icon on a user interface, entering a desired mode on a user interface, etc.) may be used to confirm the operating mode and are within the scope of the present disclosure. In addition to or as an alternative to the user indication, the operating mode may be determined based on a visual sensing system detecting the presence of one or more accessories associated with a corresponding procedure. For example, in response to detecting a biopsy device positioned on a detector surface of the x-ray system, the controller may determine that the desired operating mode is an image-guided biopsy mode. In this case, the application interface for image-guided biopsy may be automatically launched when the presence of the biopsy device is detected.

[0083] If the desired operating mode is a mammography mode, the method 300 proceeds to 308. At 308, the method 300 includes assessing the patient morphology and adjusting the x-ray system based on the patient morphology. The patient morphology may include a patient size, which includes one or more of a patient height, a patient weight, and a patient body mass index. In one example, the patient size may be determined based on a user indication via a user interface. In another example, the patient size may be estimated by utilizing a second visual sensor. Adjusting the x-ray system based on the patient morphology may include adjusting one or more of a gantry position (e.g., the gantry may be adjusted for elevation based on the patient's height) and x-ray collimation (e.g., the collimation plates may be controlled to match a region of the breast) based on the patient morphology.

[0084] After adjusting the x-ray system, method 300 proceeds to 314. At 314, method 300 includes evaluating breast position based on one or more breast anatomical structures using the second visual sensor and evaluating patient position for one or more mammographic views using the first visual sensor. Figure 4A and Figure 4B Details of assessing breast position and patient positioning for mammographic views are discussed in .

[0085] If the desired operating mode is DBT mode, the method 300 proceeds to 310 to assess the patient morphology and adjust the x-ray system based on the assessment. Step 310 is similar to step 308 and will not be repeated.

[0086] Next, method 300 proceeds to 316. At 316, method 300 includes evaluating, using the second visual sensor, a plurality of breast positions based on one or more breast anatomies, and evaluating, using the first visual sensor, a plurality of patient positions at a plurality of angles within the tomosynthesis angular range. Figure 5 and Figure 6 Details of evaluating the multiple breast positions and the multiple patient positions are discussed. Briefly, prior to initiating x-ray image acquisition, the breast and patient positions are evaluated by moving the gantry to several angles corresponding to the x-ray source position during DBT acquisition and utilizing an RGB-D camera to ensure that the breast is positioned as desired. For example, while the abdomen or shoulder or contralateral breast of the patient may not be visible at a first initial DBT position, these additional structures may be visible at different DBT angles. Thus, during DBT, the breast position and / or patient position at more than one x-ray tube angle is evaluated.

[0087] If the desired mode of operation is an image-guided biopsy mode, the method proceeds to 312. At 312, the method 300 includes assessing patient morphology, and adjusting one or more of a couch position and an x-ray system based on the patient morphology. The patient morphology may include a patient size, which may include one or more of a patient height, a patient weight, and a patient body mass index, which may be determined and / or estimated as described above at 308. Additionally, the biopsy couch position may be adjusted (e.g., via a biopsy couch actuator) to adjust one or more of a biopsy couch height and an inclination based on the patient morphology. Additionally, adjusting the x-ray system based on the patient morphology may include adjusting one or more of a gantry position and x-ray collimation based on the patient morphology. In this manner, the patient morphology may be assessed, and the x-ray system may be adjusted to automatically set up the x-ray system for the patient prior to assessing one or more of a patient position and a breast position.

[0088] 318. At 318, method 300 includes evaluating, using the second visual sensor, a biopsy device position based on one or more anatomical structures of the breast, using the second visual sensor, and evaluating, using the first visual sensor, a patient position for biopsy. The evaluation of the biopsied breast position may be based on the type of imaging used to guide the biopsy. For example, if a DBT-guided biopsy is performed, the evaluation of the biopsied breast position and the patient body position may be similar to the DBT positioning evaluation and feedback, as described below in Figure 5 and Figure 6 In addition to breast position and patient body positioning, biopsy device position can also be assessed, and real-time feedback can be provided to reduce imaging artifacts due to metallic objects associated with the biopsy device.

[0089] Go to Figure 4A, shows a high-level flowchart that shows a method 400 for evaluating one or more of a breast position and a patient position and providing real-time feedback to a user to obtain a desired breast position for mammography. Specifically, method 400 may be performed before initiating x-ray image acquisition to ensure that the desired breast position is achieved. Breast position evaluation may be performed based on input from a second vision sensor (such as the second vision sensor 154 in Figure 1B ), and patient position evaluation may be performed based on input from one or more of the second vision sensor and a first vision sensor (such as the camera in Figure 1A ). In some examples, the second vision sensor may also be used to capture a local portion of the patient (e.g., the patient's shoulder), and thus provides a local view of the patient in addition to providing a view of the compressed breast. Method 400 may be implemented by an image processing system (such as the controller 44 in Figure 1A ), an edge device connected to the image processing system, a cloud communicating with the image processing system, or any suitable combination thereof. Method 400 is described with reference to the system and components of Figures 1A to 1D , but it should be understood that method 400 may be implemented with other systems and components without departing from the scope of the present disclosure. Additionally, the real-time feedback may be provided via a display portion of a user interface that is coupled to and positioned near the x-ray mammography system such that the user can view the real-time feedback and images displayed on the user interface while making the necessary breast, patient, and x-ray system adjustments to obtain the desired breast positioning for acquisition. An exemplary user interface positioned near the x-ray system is shown at the user interface 195 in Figure 1B and Figure 1C , and the real-time feedback and images may be displayed on the display portion 196 of the user interface 195. In addition to this, the real-time feedback and images may also be displayed at the workstation user interface 50 via the display portion 50.

[0090] Furthermore, it should be understood that the systems and methods described herein may allow the user to take the necessary actions when needed. For example, in some examples, if the user wants to continue with image acquisition using the current position, even if the real-time feedback may indicate additional adjustments, options may be provided via the user interface, for example, to allow the user to initiate image acquisition using the current position.

[0091] Method 400 begins at 402. At 402, method 400 includes confirming whether the breast to be imaged is compressed between a compression paddle (such as the compression paddle 164 in Figure 1B ) and a detector (such as the detector in Figure 1B, between detectors 168 in. In one example, breast compression can be determined based on compression feedback from one or more of a force sensor and a pressure sensor coupled to a compression paddle. In another example, breast compression can be determined based on user confirmation of compression via a user interface. In yet another example, breast compression can be determined based on a final (locked) position of the compression paddle in addition to or as an alternative to user confirmation. For example, the compression paddle position can be determined based on input from one or more of the first visual sensor and the second visual sensor. The final compression paddle position can indicate that the breast is compressed for subsequent positioning assessment and later for x-ray mammographic imaging. If breast compression is confirmed, the answer at 402 is yes, and method 400 proceeds to 406.

[0092] If breast compression is not confirmed, the answer at 402 is no, and method 400 proceeds to 404. At 404, method 400 includes evaluating breast morphology and patient morphology based on input from one or more of the first visual sensor and the second visual sensor. The evaluation of breast morphology and patient morphology may be performed to identify one or more preferred compression paddle types based on the breast morphology and patient morphology and the type of imaging procedure to be performed before initiating compression. Thus, evaluating breast morphology may include estimating a breast size of the patient, which may include breast thickness, and evaluating patient morphology may include estimating a patient's height. In one example, different views of the patient, such as a front view and a side view of the patient, may be obtained from the first visual sensor and the second visual sensor, and the imaging processor may estimate the breast size and patient height using camera images of the different views. The imaging processor may utilize the breast size, patient height, and imaging mode to generate an image of the patient's height and the imaging mode of the patient via a user interface (such as, Figure 1B and Figure 1C The user interface 195) indicates to the user one or more preferred compression paddles for the desired imaging mode and patient.

[0093] Additionally, in some embodiments, before compression is initiated, real-time guidance may be provided to the user regarding the initial positioning of the patient relative to the x-ray system (e.g., sitting, standing, stooping, etc.) based on the patient's morphology and desired view (e.g., CC view or MLO view for mammography). The real-time guidance may be voice feedback and / or instructions on the user interface.

[0094] Continuing to 405, method 400 can monitor breast compression. Method 400 then returns.

[0095] Returning to 406, upon confirming breast compression, method 400 includes adjusting a first visual sensor position and a second visual sensor position based on a desired view of the mammogram, wherein the first visual sensor is used for patient position assessment and the second visual sensor is used for one or more of breast position assessment and patient position assessment.

[0096] In one example, the first visual sensor position can be adjusted so that the field of view of the first visual sensor includes the patient's entire body, the compressed breast, and the x-ray imaging system. In another example, the first visual sensor position can be adjusted so that the patient's upper body portion (e.g., from the top of the head to the waist), the compressed breast, and a portion of the x-ray system (including the imaging volume (the imaging volume includes the compression paddle, the breast, and the detector), the gantry, and a portion of the support column) are in the field of view of the first visual sensor. Further, in other examples, additionally, the first visual sensor position can be adjusted to obtain a view for measuring the distance between the support column and the patient's shoulder along a horizontal imaging plane (e.g., parallel to the detector surface). For example, the view can be a side view of the patient's body and the x-ray imaging system. Furthermore, it should be understood that the first visual sensor position can be based on the patient's position (sitting, standing, or bending down).

[0097] The second visual sensor may be positioned to obtain a field of view of an x-ray beam generated from an x-ray source. The second visual sensor is positioned on the gantry, and therefore, in one example, the gantry may be adjusted based on a desired view for mammography so that the second visual sensor captures an image consistent with the desired view. For example, if the desired view is a cranio-caudal (CC) view, the second camera and the gantry may be adjusted so that the x-ray source is in an inboard position relative to the detector (wherein the vertical axis of the x-ray source, such as vertical axis 60, is perpendicular to the detection surface), and the vertical axis of the second visual sensor is perpendicular to the detection surface, so that the second visual sensor captures a CC view of the compressed breast. In addition, the second visual sensor position may be adjusted so that the x-ray field of view is within the camera field of view. In one example, the viewing angle of the second visual sensor may be greater than the cone angle of the x-ray source. Fig. 8A Exemplary positioning of the x-ray source and second vision sensor for CC view evaluation is shown.

[0098] Go to Fig. 8A , showing an x-ray system (such as, Figure 1A A portion 800 of the x-ray system 10). Fig. 8A Many of the elements correspond to those described above. Figures 1A to 1DLike numbered elements described herein; for the sake of brevity, such elements will not be described again. Portion 800 shows an x-ray source 802 of an x-ray system at an inboard position of the CC view, wherein the vertical axis 60 is perpendicular to the detector 168, a second visual sensor 154, whose vertical axis (not shown) is also perpendicular to the detector 168, and a compressed breast 811 positioned between the detector 168 and the compression paddle 164. The breast position may be assessed prior to initiating an x-ray mammography scan. Thus, the visual sensor 154 may be adjusted such that the viewing angle α2 of the visual sensor includes the compressed breast 811 and is greater than the cone angle α1 of the cone beam 812 of the x-ray source 802. Additionally, the camera field of view 816 (including the horizontal field of view and the vertical field of view) of the visual sensor 154 may be greater than the projected field of view 814 of the detector.

[0099] Back to Figure 4A In step 406, if the desired view is a medial-lateral oblique (MLO) view, the gantry may be adjusted such that the x-ray source is angled (e.g., 45 degrees) relative to the medial position, and the vertical axis of the second visual sensor is also angled relative to the medial position at the same angle, such that the second visual sensor captures the MLO view of the compressed breast before initiating the x-ray mammography scan. Additionally, for the MLO view, the camera position may be the camera position described for the CC view, wherein the x-ray field of view is within the camera field of view. In one example, the viewing angle of the second visual sensor may be greater than the cone angle of the x-ray source. Figure 8B Exemplary positioning of an x-ray source and a second vision sensor for MLO view evaluation is shown.

[0100] Go to Figure 8B , showing an x-ray system (such as, Figure 1A A portion 850 of the x-ray system 10). Figure 8B Many of the elements correspond to those described above. Figures 1A to 1C 802, the second visual sensor 154 having a vertical axis 860 also at 45 degrees relative to the vertical axis 60, and a compression breast 811 of the x-ray system. The breast position can be assessed before initiating an x-ray mammography scan. Therefore, the visual sensor 154 can be adjusted so that the visual sensor's viewing angle α2 includes the compressed breast 811 and is greater than the cone angle α1 of the cone beam 812 of the x-ray source 802. In addition, the camera field of view 822 (including the horizontal field of view and the vertical field of view) of the visual sensor 154 can be greater than the projected field of view 820 of the detector.

[0101] Return to Figure 4A At step 406, after adjusting the first camera position and the second camera position based on the desired view (CC view or MLO view), the method 400 proceeds to 408. At 408, the method 400 includes evaluating the initial camera image for glare. For example, the examination room lights and the camera lens may generate glare on the camera image, which may reduce the subsequent assessment of the breast position. Therefore, after positioning the camera for the desired view, one or more initial camera images may be obtained and evaluated for glare. Depending on the amount of glare and the location of the glare, one or more camera image capture parameters including camera position, lighting, and polarization, as well as camera image processing parameters may be adjusted to reduce the glare. Figure 7 Exemplary methods for assessing glare and performing glare reduction adjustments are described.

[0102] Continuing to 410, method 400 includes determining whether the desired view is a CC view or an MLO view. For example, the desired view may be determined based on a user indication.

[0103] If the desired view is a CC view, the method 400 proceeds to 412. At 412, the method 400 includes obtaining a first camera image from the first visual sensor and obtaining a second camera image from the second visual sensor. In one example, the first camera image and the second camera image may be used as inputs to an artificial intelligence-based deep learning model for assessing compressed breast position and patient position in the CC view. In another example, in addition or alternatively, one or more image processing algorithms and segmentation algorithms that are not based on artificial intelligence, such as mathematical morphology, active contour algorithms, etc., may be used.

[0104] Next, at 414, method 400 includes determining whether a pectoral muscle of the compressed breast is identified in the second camera image. Specifically, method 400 may determine whether the pectoral muscle is included in the compressed breast between the detector and the compression paddle. In one example, the second camera image may include an RGB image and a depth image. The pectoral muscle may be identified based on segmentation of one or more of the RGB image and the depth image. For example, a mask region-convolutional neural network (Mask R-CNN) deep learning algorithm may be used to detect the pectoral muscle region.

[0105] If the pectoralis muscle is not identified, the answer at 414 is no, and the method 400 proceeds to 416. At 416, the method includes assessing the patient position based on the input from the first camera. Assessing the patient position includes assessing the patient shoulder distance from the x-ray system, the patient head position, the patient foot position, the patient posture, the body rotation, and the patient spine position. When assessing the patient shoulder distance, the patient shoulder corresponding to the same side of the imaged breast is at a distance from a support column (such as, Figure 1B and Figure 1CIn one embodiment, the patient's shoulder distance is a distance from a vertical edge of a support column 170 of the x-ray system. That is, if the right breast is imaged, the right shoulder distance from the corresponding vertical side edge of the support column may be determined, and if the left breast is imaged, the left shoulder distance from the corresponding vertical side edge of the support column may be determined. If the patient shoulder distance is greater than the threshold shoulder distance, it may be determined that the patient is not close enough to the x-ray system. If the patient shoulder distance is less than the threshold shoulder distance, it may be determined that the patient is close enough to the x-ray system to adequately include breast tissue, and therefore, the patient position does not need to be adjusted with respect to the shoulder distance.

[0106] In addition, based on the first camera image of the patient and the x-ray system, the level of shoulder droop can be assessed. In order to include a larger amount of breast tissue and reduce the tightening of the pectoralis muscle, the shoulder may droop, and therefore, the controller can assess the patient's shoulder to ear distance to ensure that the shoulder is not raised. Therefore, the controller can determine whether the shoulder to ear distance between the patient's shoulder and the ipsilateral ear is greater than a threshold value, which is based on the patient's morphology. Therefore, the distance between the patient's shoulder and the ipsilateral ear can be determined to assess the level of droop, wherein the greater the distance between the ear and the shoulder, the greater the level of droop.

[0107] After assessing the patient position, method 400 proceeds to 418. At 418, method 400 includes indicating to the user via the user interface that the pectoralis muscles are not being pulled into the patient position (based on the assessment at 416). Specifically, if the patient shoulder distance is greater than a threshold, the controller may indicate that the patient is further away from the x-ray system than desired. Thus, real-time feedback may be provided to the user to adjust the patient position and breast position to move the patient closer to the x-ray system and include more breast tissue for compression such that the pectoralis muscles are included.

[0108] If the patient position indicates that the patient's shoulder distance is within the threshold distance, then at the next step (i.e., at 418), method 400 includes indicating to the user via the user interface that the pectoralis muscles are not visible but the patient is within the threshold distance from the support structure. In addition, the real-time feedback may include an indication to pull additional breast tissue for compression without moving the patient further away from the vertical column.

[0109] In addition, if the shoulder-to-ear distance is greater than the threshold, the method 400 includes indicating (at 418) that the patient is in the desired droop position and no further shoulder adjustment is needed. Thus, the real-time feedback may include an indication to pull additional breast tissue with reduced patient body movement. However, if the shoulder-to-ear distance is less than the threshold, the method 400 includes indicating that the patient's shoulders are elevated, and may include providing feedback to adjust the patient's position so that the shoulders droop to the desired level and pull additional breast tissue for compression.

[0110] In this way, real-time feedback is provided to the user based on anatomical landmarks of the compressed breast identified from the compressed breast images from the second camera and the patient's position relative to the x-ray system and the patient's posture from the first camera images.

[0111] In one exemplary embodiment, if the shoulder distance and the shoulder to ear distance meet their respective criteria, one or more additional patient position parameters may be evaluated, including one or more of the patient's foot position, the patient's body rotation (slightly turned inward), and the patient's spine position (tilted from the waist toward the x-ray system), and feedback may be provided to adjust the patient's position accordingly. For example, it may be determined whether the patient's feet are pointed toward the x-ray system. If not, real-time feedback may be provided to adjust the patient's feet to point toward the x-ray system.

[0112] In one example, real-time feedback may include voice feedback combined with one or more instructions on the user interface. These instructions may include graphical instructions for the desired position and orientation superimposed on the camera image to help the user achieve the correct patient position.

[0113] While providing real-time feedback to the user, method 400 may continue to capture the first camera image and the second camera image, and may return to 414 to evaluate whether the pectoralis muscle is included after the adjustment. It should be understood that after each feedback and corresponding breast and / or patient position adjustment made by the user, the first visual sensor and the second visual sensor may capture new camera images, which are then input into the AI ​​model for further subsequent breast and patient position evaluations. Therefore, after the real-time feedback at steps 418, 424, 428, and 442, new camera images from the first visual sensor and the second visual sensor may be captured for subsequent analysis before evaluating the next breast position and patient position.

[0114] Returning to 414, if the pectoralis muscle is identified, the answer at 414 is yes, and the method 400 proceeds to 420. At 420, the method 400 includes evaluating whether the outer tissue and the inner tissue are included in the compressed breast tissue. Specifically, evaluating the inclusion of the inner tissue includes determining whether the inner boundary is clearly defined, which can be based on, for example, visualization of the cut. The evaluation of the inclusion of the outer tissue also includes determining whether the outer aspect including the outer boundary is clearly defined.

[0115] If the lateral tissue and the medial tissue are not included, the answer at 420 is no, and the method 400 proceeds to 422. At 422, the method 400 includes evaluating the patient position based on the input from the first camera. Specifically, the patient position can be evaluated for the included patient position parameters that can increase the medial tissue and the lateral tissue. The patient position parameters included with respect to the medial tissue and the lateral tissue include one or more of the patient's head position and the patient's shoulder position. In order to include enough medial tissue, the patient's head can be turned toward the opposite side. However, if the head is tilted, enough medial tissue may not be included. Therefore, the tilt level of the head toward the opposite side can be determined, and further, whether the head is turned toward the opposite side can be determined based on the head position and the shoulder position.

[0116] Additionally, in some examples, the position of the patient's contralateral breast may be assessed to increase the inclusion of medial tissue.

[0117] In addition, in some examples, the gantry position can be evaluated (e.g., using the first camera) and real-time feedback can be provided. For example, the gantry can be adjusted to include the correct tissue in the FOV. The gantry can be positioned in the patient's inframammary fold to obtain a certain position of the desired tissue.

[0118] After assessing the patient position to include the inner tissue and the outer tissue, the method 400 proceeds to 424. At 424, the method 400 includes indicating that all outer tissue and inner tissue are not included. In addition, the assessment of the patient position can be indicated, and feedback can be provided based on the patient position assessment. In one example, if the tilt level is less than the tilt threshold, and if the head is turned toward the contralateral side, the controller can indicate that the head position is in a desired position including the inner tissue and the outer tissue, and real-time feedback can be provided to pull more breast tissue to include the inner tissue and the outer tissue. If the patient position indicates that the tilt level is greater than the tilt threshold, the controller can indicate the head tilt, and can indicate to the user to reposition the head to turn the patient's head to the contralateral side (i.e., the side opposite to the side of the breast being compressed).

[0119] As described above, real-time feedback may include voice feedback combined with one or more instructions on the user interface. These instructions may include graphical instructions for the desired position and orientation superimposed on the camera image to help the user achieve the correct patient position.

[0120] Additionally, method 400 may include determining whether to include axillary tissue. The inclusion of axillary tissue may be based on the contour of the compressed breast contour and the contour of the end of the axillary cavity. The end of the axillary cavity should be located above the detector. The end of the axillary cavity may be detected by a deep learning method such as a Mask R-CNN method or by image processing (e.g., pattern matching).

[0121] In this way, breast position and patient position may be assessed for inclusion of medial and lateral tissue and axillary tissue after confirming desired inclusion of the pectoralis muscle, and real-time feedback based on camera images acquired by the first sensor and the second sensor may be provided to the user to improve breast positioning for a desired mammographic view.

[0122] After providing feedback for the outer tissue and the inner tissue inclusion, method 400 returns to 420 to continue evaluating the inner tissue and the outer tissue inclusion.

[0123] At 420, if the confirmation includes medial and lateral tissue, the answer at 420 is yes and the method proceeds to 426. At 426, method 400 includes determining whether the nipple has an outline for the CC view. Specifically, the nipple outline must be visible, and further, the nipple may be centered and should not point medially or laterally. The nipple position may be assessed using shape and color descriptors based on one or more camera images from the second visual sensor.

[0124] If the nipple does not have an outline or is not centered, the answer at 426 is no, and the method 400 proceeds to 428. At 428, the method 400 includes indicating to the user that the nipple does not have an outline or is not centered based on the nipple position assessment. In addition, one or more feedbacks may be provided to adjust the nipple position. The feedback includes annotating the camera image from the second visual sensor with an alignment line indicating a centerline along which the nipple should be positioned. In addition, the current misalignment may be indicated, for example, with an arrow.

[0125] In addition, at 426, method 400 may include determining whether the compressed breast is centered on the detector and whether the left portion of the breast and the right portion of the breast are symmetrical (also referred to herein as bilateral symmetry). This may include determining the position of the nipple opposite the edge line of the detector on the camera image, and determining that the left portion to the left of the nipple opposite the detector line and the right portion to the right of the nipple opposite the detector line display symmetry within threshold limits with respect to one or more of shape, area, and volume. In addition, real-time feedback may be provided to center the breast and achieve bilateral symmetry. This includes that an annotation of a central area (or rectangular box) in which the compressed breast exists when the central position is achieved may be shown on the camera image, and a direction to pull / adjust the breast to achieve acceptable bilateral symmetry may be indicated on the camera image.

[0126] If the nipple has an outline and is in a central position, the answer at 426 is yes, and the method 400 proceeds to 432. At 432, the method 400 includes confirming whether the interbreast groove is visible. The visibility of the interbreast groove can be determined based on the segmentation of one or more of the RGB image and the depth image. In another example, the visibility of the interbreast groove can be detected based on the camera image using a CNN algorithm (such as a Mask R-CNN deep learning algorithm). If the inframammary fold is not visible, the method 400 proceeds to 434. At 434, the method 400 includes indicating via a user interface that the interbreast groove is not visible, and providing feedback to adjust the breast tissue so that the interbreast groove is included in the compressed breast volume. The indication can be a graphical indication, such as an arrow pointing to a location where the interbreast groove is expected to be visible. In addition, the indication may include highlighting a specific area of ​​the breast to be adjusted on the camera image to include the interbreast groove. In addition, the direction of breast movement can be indicated for adjustment. For example, a graphical indication such as an arrow can be used to indicate to the user the direction in which the breast is pulled. In some embodiments, in addition to this, undesirable movements (such as rotation of the breast) can also be presented to the user using instructions to not perform the undesirable movement. In addition, one or more instructions can be provided regarding maintaining the current patient position. In some embodiments, the instructions can be displayed in an animated manner. In addition to the above-mentioned graphical instructions, voice feedback can also be provided to guide the user in positioning the breast. Then, method 400 returns to the inclusion of monitoring the interbreast cleavage.

[0127] If the interbreast cleavage is visible, the answer at 432 is yes, and the method 400 proceeds to 436. At 436, the method 400 includes confirming whether one or more wrinkles are detected. The one or more wrinkles may include one or more of skin wrinkles and skin folds on the breast under compression. Specifically, the method 400 may determine whether one or more skin folds are detected in one or more of the end region, the outer region, and the inner region of the compressed breast. During x-ray imaging, skin wrinkles and / or wrinkles may cause imaging artifacts or blur nearby anatomical structures. Therefore, if one or more wrinkles are detected, the answer at 436 is yes, and the method 400 proceeds to 442. At 442, the method 400 includes indicating to the user that one or more wrinkles are visible, and also includes, for example, annotating the camera image via a graphical indication to show one or more wrinkles. In addition, feedback on the desired adjustment may be provided to reduce the one or more wrinkles. For example, voice feedback and / or an indication on the user interface is provided to smooth the one or more wrinkles by pulling the skin toward the nipple. Furthermore, additionally, an indication may be utilized to indicate to the user the direction of the undesired movement (pulling the skin upward and posteriorly) to avoid the undesired movement. The method 400 then returns to 436 to monitor the one or more wrinkles.

[0128] At 436, if one or more skin folds / wrinkles are not detected, the method 400 proceeds to 438. At 438, the method 400 includes indicating to the user via the user interface that the breast position and the patient position are confirmed for the x-ray acquisition. Continuing to 440, the method 400 includes acquiring one or more x-ray projection images of the CC view of the compressed breast. The method 400 then ends.

[0129] Thus, one vision sensor can be used to assess breast position and provide real-time feedback to obtain the desired breast positioning for the CC view; and another vision sensor can be used to assess patient position and provide real-time feedback to adjust the patient position so that the desired breast position for the CC view can be achieved. Thus, the breast position and patient position can be adjusted based on input from the two vision sensors to obtain consistent and high-quality breast photos.

[0130] Although the above method describes assessing breast position and patient position, in some embodiments, a second camera may be utilized to assess breast position independent of patient position and this is within the scope of the present disclosure. For example, assessing breast position based on camera images from the second camera for the CC view may include a pectoralis assessment, an outer tissue and inner tissue assessment, a nipple position assessment, and one or more skin folds and wrinkles assessments. In addition, real-time feedback may be provided based on the breast position assessment. The real-time feedback may include one or more breast position adjustments and patient position adjustments.

[0131] In other embodiments, monitoring / assessing patient position may be assessed independently of breast position. In one example, breast position may be assessed, and upon confirmation of desired breast position, patient position may be assessed using one or more of the first camera and the second camera, and real-time feedback may be provided to obtain the desired patient position prior to initiating image acquisition. In another example, patient position may be assessed separately, and real-time feedback may be provided to obtain the desired patient position prior to initiating image acquisition.

[0132] Additionally, in some examples, a second camera for breast position monitoring may be used to assess patient position. For example, images from the second camera may be analyzed to check whether irrelevant structures, such as the contralateral breast, the patient's abdomen, etc., are not visible in the field of view of the second camera, and therefore, are not visible in the field of view of the x-ray source. Thus, irrelevant structures that may obscure visualization of compressed breast tissue may be reduced.

[0133] Furthermore, it will be appreciated that the order of assessment, wherein the pectoralis muscle is assessed first, followed by the medial and lateral tissues, nipple contour and IMF, and finally the crease, allows for efficient and faster assessment and feedback.

[0134] Additionally, one or more of breast position and patient position assessments may be performed, and real-time feedback may be provided for other views including, but not limited to, mediolateral view (ML view), lateral medial view (LM view), lateral medial oblique view (LMO view), mediolateral posterior view, stepped oblique view, point compression view, two-point compression view, axillary view, cut view, tangential view, reverse CC view, rolled CC view, bull's-eye CC view, elevated craniocaudal projection, caudocaudal projection, oblique projection, and inferomedial-superolateral oblique projection.

[0135] Returning to 410, if the desired view is an MLO view, method 400 proceeds to Figure 4B 446 in. At 446, method 400 includes determining whether the pectoralis muscle edge is clearly defined based on the camera image of the compressed breast acquired using the second visual sensor. In one example, evaluating the pectoralis muscle in the MLO view may include evaluating one or more of the axillary position, the breast boundary angle, and the curvature. In order to detect the breast shape, a classical camera image segmentation method may be utilized. In another example, a convolutional neural network (CNN) may be trained based on an X-ray image and a corresponding camera (RGB-D) image. For example, on an X-ray image, if one or more breast position assessment parameters (e.g., pectoralis muscle edge) are positioned as required, a corresponding camera image may be determined to show the expected compressed breast position for the one or more breast position assessment parameters. Therefore, a plurality of camera images showing the expected compressed position may be identified based on the corresponding X-ray image showing the expected breast position parameters, and a CNN may be trained based on the identified plurality of camera images. The trained network may then be used to perform inference of chest placement during breast position assessment. If the pectoralis muscle edge is not identified and / or the edge shape does not conform to an expected edge shape (eg, a convex shape or a straight line), the answer at 446 is no and method 400 proceeds to 448 .

[0136] In some embodiments, pectoralis muscle placement (i.e., pectoralis muscle edge) may be assessed after x-ray image acquisition. In one example, if the confidence level of the detection of pectoralis muscle edge clarity on the RGB image is less than a threshold confidence, the pectoralis muscle edge clarity may be assessed based on the x-ray image after x-ray image acquisition.

[0137] At 448, method 400 includes evaluating the patient position. The patient position may be evaluated based on input from the first visual sensor. Specifically, the patient position relative to the x-ray system may be evaluated. This includes the patient's ipsilateral arm position relative to the detector, the patient's hip and shoulder position relative to the x-ray system, and the compression paddle position relative to the patient. For example, the controller may evaluate whether the corner of the detector that contacts the patient's upper arm is positioned behind the ipsilateral axilla, whether the patient's hip is within the desired distance from the lower corner of the image receiver and is positioned forward, whether the patient's hip and shoulder are facing the x-ray unit, and whether the upper edge of the compression paddle is at a desired angle to the clavicle. Continuing to 450, method 400 includes providing feedback to the user to adjust one or more of the breast position and the patient position. For example, if one or more of the above-mentioned patient position parameters are not confirmed, inconsistencies may be indicated to the user via the user interface, and real-time feedback may be provided to adjust the inconsistent patient position. In addition, the real-time feedback may include instructions to pull the breast upward and outward before the breast rests on the detector surface to ensure that the chest muscle wall is included. Then method 400 returns to 446.

[0138] It should be understood that after each feedback and corresponding breast and / or patient position adjustment made by the user, the first visual sensor and the second visual sensor may capture new camera images, which are then input into the AI ​​model for further subsequent breast and patient position assessments. Thus, after the real-time feedback at steps 448, 454, 464, 470, 476, 480, and 484, new camera images from the first visual sensor and the second visual sensor may be captured for subsequent analysis before assessing the next breast position and patient position.

[0139] At 446 , if the pectoralis muscle edge is identified and the edge shape is consistent with the expected edge shape, the answer at 446 is yes and method 400 proceeds to 452 .

[0140] At 452, method 400 includes confirming whether the nipple has an outline. The nipple outline confirmation may be based on the position of the nipple in the camera image and the visibility of the nipple outline. In addition, at 452, method 400 may include determining whether the compressed breast is centered on the detector and whether the left portion of the breast and the right portion of the breast are symmetrical. The determination of the center position and left-right symmetry may be performed as described at 426. If the nipple does not have an outline, the compressed breast is not centered, and / or the left-right symmetry is not within a threshold limit (e.g., with respect to shape, area, and / or volume), the answer at 452 is no, and method 400 proceeds to 454. At 454, method 400 includes indicating to the user that the nipple does not have an outline, the compressed breast is not centered, and / or the left-right symmetry is not within a threshold limit, and also includes providing feedback to adjust the breast position so that the nipple has an outline, the nipple is centered, and acceptable left-right symmetry is obtained. The feedback includes, at 456, annotating the camera image from the second vision sensor for the MLO view with an alignment line indicating a center line along which the nipple should be positioned. In addition, the current misalignment may be indicated, for example, with an arrow. In addition, an annotation of a center area (or rectangular box) may be shown on the camera image where the compressed breast resides when the center position is achieved. In addition, the direction to pull / adjust the breast to obtain acceptable left-right symmetry may be indicated on the camera image.

[0141] Additionally, in the MLO view, the method 400 can assess the amount of sagging of the compressed breast. That is, based on the camera image, the method can determine that the compressed breast is bulging downward, and if so, the method 400 can provide additional feedback to reduce sagging. The method then returns to 452.

[0142] If the nipple has a contour, the answer at 452 is yes, and method 400 proceeds to 458. At 458, method 400 includes identifying the pectoral-nipple line (PNL). The PNL may be identified based on the location of the nipple and the pectoral margin.

[0143] Next, at 460, method 400 includes inferring whether the lower edge of the chest is at or below the PNL. If the answer at 460 is no, method 400 proceeds to 462. At 462, the method includes assessing the patient's position. The assessment of the patient's position may be similar to the position assessment at 448, and will not be repeated for the sake of brevity. Continuing to 448, method 400 includes indicating to the user through the user interface that the lower edge of the pectoralis is not at or below the PNL, and real-time feedback may be provided to adjust one or more of the breast position and the patient position to move the lower edge to or below the PNL. Then method 400 returns to 460.

[0144] If the lower edge is at or below the PNL, the answer at 460 is yes, and the method 400 proceeds to 466. At 466, the method 400 includes inferring whether there is a margin between the upper chest edge and the upper external breast edge. Thus, in one example, it may be determined whether the area of ​​the region between the upper chest edge and the upper external breast edge is greater than a threshold area. If the margin is not confirmed, the answer at 466 is no, and the method 400 proceeds to 468 to evaluate the patient position, as described above at 448. After evaluating the patient position, the method 400 proceeds to 470 to indicate that the desired margin between the upper chest edge and the upper external breast edge is not achieved. Additionally, real-time feedback may be provided to adjust one or more of the breast position and the patient position to obtain the desired margin. The method 400 then returns to 466.

[0145] Steps 460 and 466 are based on identifying the chest edge from the camera image. The detection of the chest edge may be determined as described above at 446. However, as also indicated above, if the confidence level of the detection of the chest muscle edge definition on the RGB image is less than a threshold confidence, the chest muscle edge definition may be evaluated after x-ray image acquisition. Thus, the evaluation of the lower edge of the chest muscle at 460 (i.e., the lower edge of the chest muscle is at or below the PNL) and the evaluation of the margin between the upper chest edge and the outer breast edge at 466 may be performed based on the x-ray image after x-ray image acquisition.

[0146] Furthermore, it should be appreciated that in some embodiments, the evaluations performed at steps 446, 460, and 466 may be performed once based on the x-ray image after the x-ray image is acquired, rather than once before the x-ray image is acquired.

[0147] If there is an expected wide margin, the answer at 466 is yes, and the method 400 proceeds to 472. At 472, the method 400 includes inferring whether all posterior tissue and axillary tissue and axillary end are included. In one example, the inclusion of the posterior tissue can be confirmed based on one or more of the patient position, the gantry position, and the breast size. For example, the method 400 can determine whether the user has pulled the breast tissue based on motion recognition from the image sequence obtained using the second visual sensor. In addition, from one or more images obtained using the second visual sensor, the distance of the nipple from the edge of the detector can be determined. The distance can be compared with a threshold distance, which is determined based on the breast size, where a larger breast has a larger threshold distance. For example, for a first smaller breast size, the first threshold distance between the nipple of the first breast and the edge of the detector can be a first shorter distance so as to include the posterior tissue; and for a second larger breast size, the second threshold distance between the nipple of the second breast and the edge of the detector can be a second larger distance so as to include the corresponding posterior breast tissue; and so on. In another example, a convolutional neural network (CNN) may be trained based on X-ray images and corresponding camera (RGB-D) images to include posterior tissue.

[0148] In addition, the inclusion of axillary tissue can be based on the contour of the compressed breast contour and the end of the axillary contour. The end of the axillary should be located above the detector. The end of the axillary can be detected by a deep learning method such as a mask RCNN method or by image processing (e.g., pattern matching). If one or more of the posterior tissue and the axillary tissue are not included, the answer at 472 is no, and the method 400 proceeds to 474 to evaluate the patient position, as described above at 448. Continuing to 476, the method 400 includes indicating to the user that one or more of the posterior tissue and the axillary tissue are not included. In addition, at 476, real-time feedback can be provided to adjust one or more of the patient position and the breast position to include the posterior tissue and the axillary tissue. The method 400 then returns to 472.

[0149] If all posterior tissue and axillary tissue are included, the answer at 472 is yes, and method 400 proceeds to 478. At 478, method 400 includes inferring whether an inframammary fold (IMF) is included. The inclusion of an IMF may be similar to the detection of visibility of an intermammary groove, and thus may be determined as described at 432, and will not be repeated here for the sake of brevity.

[0150] Continuing, step 480 for providing real-time feedback to adjust breast tissue for visibility of IMF when IMF is not detected, step 482 for inferring (after confirming IMF at 480) whether one or more skin folds are detected, steps 488 and 490 for providing real-time feedback to reduce skin folds when IMF is detected, step 484 for indicating confirmation of breast and patient position for MLO view when all breast position and patient position parameters for MLO view are met, and step 486 for acquiring one or more x-ray images for MLO view are similar to steps 434, 436, 442, 444, 438 and 440, respectively, and will not be repeated for the sake of brevity.

[0151] After confirming that the breast position and patient position parameters are met and one or more compressed breast images of the MLO view are acquired, the method 400 ends.

[0152] Although the methods described herein illustrate evaluating breast position and / or patient position using one or more camera images obtained using one or more of the first visual sensor and the second visual sensor prior to initiating x-ray image acquisition, it should be understood that, in addition to this, breast position monitoring may be performed based on the acquired views to evaluate whether one or more desired criteria for the corresponding views have been met. That is, after image acquisition, the acquired x-ray images (after breast position assessment) may also be evaluated to confirm imaging quality and to evaluate whether the desired criteria for the desired views are met. In addition, one or more of annotations and graphic overlays may be displayed on the acquired images to indicate whether the one or more desired criteria have been met, as well as the desired change in the current breast position.

[0153] Thus, in some embodiments, before initiating image acquisition and after initial positioning of the breast (compressed between the compression paddle and the detector), multiple breast positioning criteria may be evaluated to determine whether the breast is positioned for mammography. The multiple breast positioning criteria may be based on the view type (MLO or CC view). The multiple breast positioning criteria may include one or more overview criteria and / or one or more breast features. The one or more overview criteria for the CC view may include central breast position, left-right symmetry, the presence of the end of the axilla, and the presence of the interbreast groove. The one or more breast features for the CC view may include pectoralis muscle, lateral and medial tissue, axillary tissue, nipple features, and skin folds. Thus, the evaluation criteria for the one or more breast features for the CC view may include: the presence of pectoralis muscle, the inclusion of lateral and medial tissue and axillary tissue (e.g., the end of the axilla), the central position and nipple contour of the nipple, and the absence of skin folds in one or more of the tail region, the lateral region, and the medial region of the compressed breast. For the MLO view, the one or more overview criteria may include central breast position, left-right symmetry, and the amount of breast ptosis. The one or more breast features for the MLO view may include features based on the pectoralis muscle, posterior tissue, axillary tissue, inframammary fold (IMF), nipple features, and skin folds. Thus, the evaluation criteria for the one or more breast features for the MLO view may include: the amount of pectoralis muscle, the location of the pectoralis muscle edge relative to the nipple line, the inclusion of the posterior, axillary tissue (e.g., end of axillary), the center location of the nipple and the nipple contour, and the absence of skin folds in one or more of the IMF region, pectoral region, and axillary region of the compressed breast.

[0154] In one example, it may be desirable to sequentially evaluate the multiple breast positioning criteria. Based on the evaluation of the multiple breast positioning criteria, real-time feedback based on the evaluation of each criterion may be provided to help the user position the breast. In this way, correct positioning of the breast for each view for mammography may be achieved.

[0155] In another example, in addition to evaluating the multiple breast positioning criteria, at each step, the patient position may also be evaluated. Based on the evaluation of breast features and the patient evaluation, real-time feedback based on the evaluation of each feature may be provided to help the user position the breast.

[0156] In this way, correct positioning of the breast for mammography can be achieved for each view.In addition, by performing a step-by-step assessment of breast structure and patient position prior to x-ray image acquisition, image processing speed is increased and feedback speed is improved, which in turn leads to an overall improvement in breast positioning efficiency.

[0157] In addition, after adjusting one or more of the patient position and the breast position based on the evaluation of one or more of the patient position and the breast position, the x-ray system may continue to acquire one or more x-ray images. In addition, the one or more acquired x-ray images may be evaluated for one or more of the multiple breast positioning criteria. Therefore, the first evaluation may be performed using a camera image (e.g., an RGB depth image) acquired by one or more of the first visual sensor and the second visual sensor, and the second evaluation may be performed using one or more x-ray images acquired using the x-ray system. During the second evaluation using the x-ray image, one or more of the multiple breast position criteria may be evaluated simultaneously. For example, during the second evaluation, the acquired x-ray image may be evaluated for one or more overview criteria and / or one or more breast features (corresponding to the view) at the same time. In addition, feedback may be provided to the user via one or more of annotations and overlays on the acquired x-ray image.

[0158] In some other embodiments, during the first evaluation using the camera image, the one or more overview criteria for the CC view may include central breast location, left-right symmetry, and the presence of an interbreast groove. The one or more breast features for the CC view may include nipple features and skin folds. Thus, the evaluation criteria for the one or more breast features for the CC view during the first evaluation may include: the central location of the nipple and the nipple contour, and the absence of skin folds (first set of features) in one or more of the tail region, the lateral region, and the medial region of the compressed breast. During the second evaluation using the x-ray image, features not evaluated during the first evaluation, such as the presence of the end of the axilla, the presence of the pectoralis major muscle, the lateral and medial tissues, and the inclusion of axillary tissue (second set of features), may be evaluated. Similarly, the MLO view may be evaluated using the first set of features evaluated during the first evaluation and the second set of features different from the first set of features evaluated during the second evaluation. In some examples, the first set of features and the second set of features may be evaluated during the second evaluation.

[0159] In this way, correct positioning of the breast for mammography can be achieved for each view.In addition, by performing a step-by-step assessment of breast structure and patient position prior to x-ray image acquisition, image processing speed is increased and feedback speed is improved, which in turn leads to an overall improvement in breast positioning efficiency.

[0160] In some embodiments, one or more of breast positioning errors and patient positioning errors can be detected based on recognition of movement of one or more of the patient and the user (i.e., the technician performing the mammography). For example, the breast must be pulled in a specific way to position it. Therefore, the technician's posture can be analyzed to check whether the user has pulled the breast.

[0161] Go to Figure 5 , shows a high-level flow chart illustrating a method 500 for assessing breast position and breast morphology during digital breast tomosynthesis (DBT). The method 500 may be performed prior to initiating x-ray image acquisition in order to determine whether a desired breast position for DBT is achieved. The breast position assessment may be based on information from a second visual sensor (such as, Figure 1B The patient position assessment may be performed based on input from a second visual sensor and a first visual sensor (such as a camera 154 in FIG. 1 ), and the patient position assessment may be performed based on input from a second visual sensor and a first visual sensor (such as a camera 154 in FIG. 1 ). Figure 1A Method 500 may be performed by an image processing system (such as, Figure 1A The method may be implemented by a controller 44 in the image processing system, an edge device connected to the image processing system, a cloud communicating with the image processing system, or any suitable combination thereof. Figure 1A , Figure 1B and Figure 1C The method 500 is described with reference to the systems and components of FIG. 5 , but it should be understood that the method 500 may be implemented with other systems and components without departing from the scope of the present disclosure.

[0162] Method 500 begins at 502. Steps 502, 503, 504, 506, and 508 are similar to Figure 4A , and therefore will not be repeated for the sake of brevity. In short, the above steps of method 500 include monitoring an x-ray system for breast compression, and after compression, adjusting a first visual sensor and a second visual sensor at an initial angle for DBT acquisition. In one example, the initial angled position may be a first visual sensor position and a second visual sensor position for a CC view. In another example, the initial angled position may be a first visual sensor position and a second visual sensor position for an MLO view. It should be understood that the initial angle for DBT may be any selected angle of the x-ray source, and the first visual sensor and the second visual sensor may be adjusted accordingly. When adjusting the first visual sensor and the second visual sensor, one or more initial camera images from the first visual sensor and the second visual sensor may be obtained and evaluated for glare, and one or more image capture parameters and camera image processing parameters may be adjusted based on the detected glare.

[0163] Continuing to 510 , method 500 includes obtaining a first camera image from a first vision sensor and obtaining a second camera image from a second vision sensor, wherein an x-ray source is at an initial angle for DBT after adjusting for glare.

[0164] Next, at 512, method 500 includes evaluating the breast position and patient position at the initial angulation, and adjusting the breast position and patient position based on real-time feedback from the evaluation, such as Figure 4A and Figure 4B In short, the method 500 may assess the breast position based on the breast structure identified from the second camera image obtained using the second visual sensor and assess the patient position based on the first camera image obtained using the first visual sensor. The assessed breast structure and patient position are based on the initial angulation of the x-ray tube. For example, if the initial angulation is a zero-degree angulation of the x-ray source, the breast position and patient position assessment may include a step-by-step assessment of the pectoralis muscle inclusion, the lateral tissue and the medial tissue inclusion, the nipple contour confirmation, the IMF inclusion, and the absence of one or more skin and fat folds. In each assessment step, real-time feedback may be provided to adjust one or more of the breast position and the patient position to ensure that the breast and the patient are in the desired position. Similarly, if the initial angulation is an MLO view angulation (e.g., a 45-degree x-ray source angulation), the breast position and patient position assessment may include a step-by-step assessment of the pectoralis muscle edge shape and position, nipple contour confirmation, the margin between the upper pectoralis muscle edge and the upper external breast edge, the posterior and axillary tissue inclusion, the IMF inclusion, and the absence of one or more skin and fat folds. As described above, at each step, real-time feedback may be provided to the user to adjust one or more of the breast position and the patient position.

[0165] After adjusting the breast position and patient position at the initial x-ray tube angulation, the method 500 proceeds to 514. At 514, the method 500 includes evaluating the breast position and patient position at a selected angle from a plurality of angles within the range of angles for DBT acquisition, and providing real-time feedback to the user based on the breast position and patient position evaluation. Figure 6 Details of assessing breast position and patient positioning for DBT are discussed.

[0166] Go to Figure 6 , shows a high-level flow chart illustrating a method 600 for evaluating breast position and patient position at two or more selected angles within an angular range of an x-ray system during DBT imaging and providing real-time feedback for adjusting breast position for DBT acquisition. In addition, the method 600 includes evaluating patient position to detect occlusions (e.g., portions of the patient's body that are not imaged) when the gantry is moved at multiple angles for DBT acquisition. The method 600 may be performed prior to initiating an x-ray image acquisition to determine whether a desired breast position and patient body position for DBT is achieved. The breast position assessment may be based on information from a second visual sensor (e.g., Figure 1BThe patient position assessment may be performed based on input from a second visual sensor and a first visual sensor (such as a camera 154 in FIG. 1 ), and the patient position assessment may be performed based on input from a second visual sensor and a first visual sensor (such as a camera 154 in FIG. 1 ). Figure 1A Method 600 may be performed by an image processing system such as, Figure 1A The method may be implemented by a controller 44 in the image processing system, an edge device connected to the image processing system, a cloud communicating with the image processing system, or any suitable combination thereof. Figure 1A , Figure 1B and Figure 1C The method 600 is described with reference to the systems and components of the present invention, but it should be understood that the method 600 can be implemented with other systems and components without departing from the scope of the present disclosure. The method 600 will be described for a DBT acquisition in which the x-ray source is rotated around a compressed breast within an angular rotation range (+θ to -θ) about a medial position where the vertical axis of the x-ray system, such as Figure 1A The vertical axis 60 in is perpendicular to the detector. Other DBT acquisitions in which the x-ray source is rotated about an axis that is angled relative to the vertical axis (eg, 30 degrees, 45 degrees, 60 degrees, etc. from the medial position) are also within the scope of the present disclosure.

[0167] The method 600 starts at 602. At 602, the method 600 includes obtaining one or more camera images at each of two or more selected angles from each of a first visual sensor and a second visual sensor. The selected angles include various angles of an x-ray source at which projection images for DBT are obtained, and thus assessing breast position and patient position at the selected angles. Thus, in one example, each of a plurality of angles for performing DBT acquisition may be selected for assessing breast position and patient position. In another example, breast position and patient position assessment may be performed for two or more angles but less than the total number of angles for obtaining projection images for DBT. For example, the end angle and medial position in either direction (+θ or -θ) may be selected together. In some examples, in addition to the end angle and medial position, one or more angles therebetween may be selected for assessing breast position and patient position for subsequent DBT acquisition.

[0168] Next, at 604, method 600 includes identifying, from one or more camera images obtained using the second sensor, visible edges of each angled lower pectoralis muscle in the selected angle, lateral and medial tissues, and nipple locations. For example, the one or more camera images obtained using the second visual sensor may be input into an artificial intelligence-based neural network model for identifying anatomical landmarks of the compressed breast, including visible edges of the pectoralis muscle, lateral and medial tissues, and nipple locations. In one example, the output of the AI-based neural network model may include one or more segmented camera images with annotations for the above-mentioned anatomical landmarks of the compressed breast.

[0169] Continuing to 606, method 600 includes confirming visibility of the chest edge, lateral tissue and medial tissue included, confirmation of the nipple outline and central positioning of the nipple (i.e., not pointing medially or laterally), and visibility of the IMF at each selected angle. If visibility and positioning of all of the above anatomical structures at each of the two or more selected angles are confirmed, the answer at 606 is yes, and the method proceeds to 610. If none of the above anatomical landmarks of the breast are confirmed, the answer at 606 is no, and method 600 proceeds to 608. At 608, method 600 includes providing real-time feedback to adjust the breast position so that: the chest edge is visible, including lateral tissue and medial tissue, the nipple outline is visible, and the nipple is centrally positioned, and the IMF is visible. In addition, in some examples, the patient's body position in one or more camera images obtained using the first visual sensor at each of the two or more selected angles may be evaluated. Specifically, assessing the patient body position includes determining whether the patient's shoulder distance from the x-ray system is within a threshold distance, whether the patient's head position has a tilt level less than a threshold tilt, and whether the direction of the head rotation is toward the opposite side, whether the patient's foot position is directed toward the x-ray system, whether the body rotation is toward the medial position, and whether the patient's spine position is at an angle with the hip. If any of the above patient body parameters are not met, in addition to adjusting the breast position based on the breast anatomy, the real-time feedback may also include adjusting the patient position based on the patient body position assessment.

[0170] In one example, each of the breast anatomy structures including the pectoralis muscle, the lateral and medial tissues, the nipple, and the IMF may be evaluated together with the patient's body position in a step-by-step manner. For example, the pectoralis muscle is evaluated together with the patient's body position first, and when feedback is provided to adjust one or more of the breast position and the patient's body position for the visibility of the pectoralis muscle and then confirm the inclusion of the pectoralis muscle, the method continues to evaluate the lateral and medial tissues and the patient's body position together. When feedback is provided to adjust one or more of the breast position and the patient's body position to include the lateral and medial tissues and confirm the inclusion of the lateral and medial tissues, the method continues to evaluate the nipple contour and position. When feedback is provided to adjust the nipple contour and position and confirm the desired positioning of the nipple and the confirmation of the nipple contour, the method continues to evaluate whether the IMF is visible. After providing feedback to adjust one or more of the breast position and the patient's position to include the IMF and confirm the inclusion of the IMF, the method may proceed to the next step 610.

[0171] Continuing to 610, method 600 includes determining whether one or more wrinkles (such as skin wrinkles, fat wrinkles, etc.) are visible at each of the selected x-ray tube angles. If so, at 622, method 600 includes providing real-time feedback to adjust the breast position to reduce the occurrence of wrinkles. Then, method 600 returns to monitoring wrinkles. If one or more wrinkles are not visible, method 600 proceeds to 612.

[0172] At 612, method 600 includes evaluating a second patient position at each selected angle. The evaluation of the second patient position may be based on one or more camera images obtained from a second visual sensor after adjusting the breast position and the patient's body position to include anatomical landmarks and reduce wrinkles. Evaluating the second patient position includes evaluating the patient's ipsilateral shoulder position and contralateral shoulder position, the patient's head position, and the contralateral breast position. Specifically, it may be determined whether any of the patient's ipsilateral shoulder, the contralateral shoulder, the patient's head, and the contralateral breast are in the field of view of the x-ray source at each of the two or more selected angles, so that a portion of the field of view is occluded by the patient's body part. The second visual sensor captures the field of view of the x-ray source, and thus the second patient position evaluation may be performed using one or more camera images obtained using the second visual sensor. During the second patient position evaluation, the one or more camera images obtained using the second visual sensor at each of the selected angles may be input into an artificial intelligence-based algorithm that is trained to identify occluded portions of the patient's body in the one or more camera images. For example, if a portion of the contralateral breast is visible at one or more selected angles, the algorithm may identify the occluded portion and the angle at which the occlusion is identified, and indicate the identified occlusion on the one or more camera images.

[0173] Continuing to 614, method 600 includes confirming whether an occlusion is detected based on the evaluation of the second patient position (as described at 612). If the answer at 614 is yes, method 600 proceeds to 616 to provide real-time feedback to adjust the patient position to move the occluding body part away from the field of view without changing the position of the compressed breast. Then, method 600 returns to 614 to verify whether the occlusion is cleared after adjusting the patient position.

[0174] If no occlusion is detected, the answer at 614 is no, and the method proceeds to 618 to indicate, via the user interface, the breast position and patient position confirmed for DBT acquisition.

[0175] Continuing to 620 , method 600 includes performing DBT by acquiring a plurality of x-ray projection images at each of a plurality of angles.

[0176] In this manner, prior to DBT acquisition, breast position and patient position may be assessed based on camera images from a first visual sensor (for patient body position assessment) and a second visual sensor (for breast position assessment and occlusion identification) at two or more selected angles within the range of angles of x-ray source rotation for DBT. By assessing breast position and patient position at more than one angle of the x-ray source, breast positioning and patient positioning errors that may not be easily detected in only one angle may be identified, and real-time feedback may be provided to adjust one or more of the breast position and patient position. Thus, the quality and consistency of DBT may be improved. Furthermore, by identifying and correcting positioning errors prior to acquisition, the need for recalls is reduced, which in turn reduces radiation exposure to the patient and improves patient comfort and confidence in the diagnosis.

[0177] While the above method 600 describes performing a DBT scan without x-ray imaging and evaluating one or more of breast position and patient position at two or more x-ray source angulations prior to initiating a DBT acquisition with x-rays, in some embodiments, one or more of breast position and patient position at multiple DBT angulations may be evaluated without moving the x-ray tube, and real-time feedback may be provided based on the evaluation (prior to performing a DBT acquisition with x-rays). In one example, the position of the second visual sensor may be adjusted without moving the gantry to evaluate one or more of breast position and patient position at multiple DBT angulations. In another example, the evaluation of one or more of breast position and patient position for multiple angulations for DBT may be performed at a single camera position and a single gantry position.

[0178] In one embodiment, a set of breast features may be evaluated to determine whether the breast is positioned for mammography. The set of breast features may be based on the view type (MLO or CC view). In addition, the set of breast features may include tissue features, nipple features, breast folds, and skin folds. In one example, it may be desirable to evaluate the set of breast features sequentially, and at each step, the patient position may be evaluated. Based on the evaluation of the breast features and the patient evaluation, real-time feedback based on the evaluation of each feature may be provided to help the user position the breast. For example, the presence of the pectoralis muscle may first be identified and confirmed in the field of view of the camera. Upon confirming the presence of the pectoralis muscle, the controller may determine whether all lateral tissue and medial tissue are included. Next, the nipple position may be evaluated, and then the inframammary fold may be identified. Finally, upon ensuring that the breast is correctly positioned relative to the tissue, nipple position, and IMF, the controller may evaluate whether one or more skin folds are visible. At each step, if the corresponding feature is not detected or not correctly positioned, the controller may provide real-time feedback to adjust one or more of the breast and the patient. In this way, correct positioning of the breast for mammography for each view may be achieved. Furthermore, by performing a step-by-step assessment of breast structure and patient position, image processing speed is increased and feedback speed is improved, which in turn leads to an overall improvement in breast positioning efficiency.

[0179] Next, Figure 7 A high-level flow chart is shown that illustrates a method for reducing the use of a method coupled to a mammography system such as a Figure 1A Method 700 for detecting glare in a camera image obtained by a visual sensor of a mammography system 100 in a mammography system. The visual sensor may be Figure 1A The first visual sensor 101, Figure 1B The second visual sensor 154 in the mammography system or any camera coupled to the mammography system. The first visual sensor and the second visual sensor are used to obtain patient images and compressed breast images, which are then used to assess one or more of patient position and breast position based on patient morphology and breast anatomical landmarks. Therefore, before assessing breast position and patient position from the camera images, the camera images may be analyzed for glare and processed to reduce glare (if detected) to improve downstream analysis of the camera images. The method 700 may be performed by an image processing system (such as, Figure 1A The method 700 may be implemented by a controller 44 in the image processing system, an edge device connected to the image processing system, a cloud in communication with the image processing system, or any suitable combination thereof. Figures 1A to 1DThe method 700 will be described with respect to a camera image, and it should be understood that the method 700 for detecting and reducing glare can be applied to any camera image obtained using any visual sensor.

[0180] The method 700 starts at 702. At 702, the method 700 includes obtaining a camera image, and identifying a region of interest of the camera image. The region of interest may include an area on the camera image that is subject to further processing for position assessment. For example, if a camera image from a first visual sensor is obtained (for patient position assessment), the region of interest may include a portion of the camera image that includes the patient and the x-ray system. If a camera image from a second visual sensor is obtained (for identifying breast anatomical landmarks, and therefore for breast position assessment), the region of interest may include a portion of the camera image that includes a compressed breast.

[0181] Next, at 704, method 700 includes determining whether glare is detected within the region of interest. Glare may be detected based on photometric features including one or more of light intensity, color saturation, and brightness contrast. For example, a region of the camera image having a light intensity greater than a threshold intensity, a color saturation less than a threshold saturation level, and a brightness contrast less than a threshold contrast level may indicate a glare region. If glare is not detected within the region of interest or glare is detected outside the region of interest, method 700 proceeds to 708 to indicate that glare adjustment is not required, and method 700 ends. If glare is detected within the region of interest, method 700 proceeds to 706.

[0182] At 706, method 700 includes indicating an area within the region of interest that includes glare. Next, at 708, method 700 includes providing feedback to the user to reduce glare. The feedback may include: instructions for adjusting the camera position at 712; instructions for adjusting the room lighting to reposition the glare source (which is typically a light source) so that the glare area is not within the region of interest at 714; instructions for including or adjusting a polarizing filter on the camera lens at 716. In addition, in some examples, glare can be reduced by adjusting one or more camera image capture parameters (e.g., by commanding a camera controller to adjust the one or more camera image capture parameters) and camera image processing parameters, the one or more camera image capture parameters including camera position, room lighting, and polarizing filters, the camera image processing parameters based on the amount and position of each of the one or more glare areas to reduce glare.

[0183] After providing feedback, method 700 proceeds to 718. At 718, method 700 includes confirming whether the glare within the region of interest has been removed. If the answer at 718 is yes, method 700 proceeds to 708 to indicate that the glare adjustment is complete, and method 700 ends. However, if the glare has not moved away from the region of interest or decreased below a threshold amount, method 718 proceeds to 720.

[0184] At 720 , method 700 includes determining a number of images and camera settings (eg, reducing exposure) for the number of images to generate a composite image with reduced glare in the region of interest.

[0185] Continuing to 722, method 700 includes using the composite image for subsequent breast position assessment and / or patient position assessment.

[0186] Technical effects of the present disclosure include improved and real-time breast positioning assessment, guidance, and feedback prior to image acquisition, resulting in higher quality x-ray mammographic images without duplication and / or increased exposure to the patient. Another technical effect is improved diagnostic capability resulting from improved breast positioning and thus reduced need for recalls. Yet another technical effect is automated guidance for inexperienced users to obtain higher quality mammographic images.

[0187] In one embodiment, a method for an x-ray mammography system includes: using one or more cameras coupled to the x-ray mammography system to evaluate, via visual sensing, one or more of a patient's position, a local view of the patient, and the patient's breast anatomy; detecting, based on the evaluation, one or more of a patient positioning error and a breast positioning error; and providing real-time feedback to a user via a user interface of the mammography system based on the detection. In a first embodiment of the method, the method may additionally or alternatively include evaluating breast anatomy using a camera having a field of view that includes an imaging volume between a compression paddle and a detector of the x-ray mammography system. A second embodiment of the method optionally includes the first embodiment and further includes evaluating the patient position based on input from a first camera having a first field of view of the x-ray mammography system and the patient, and evaluating the breast anatomy based on input from a second camera having a second field of view that includes an imaging volume between a compression paddle and a detector of the x-ray mammography system. A third embodiment of the method optionally includes one or more of the first embodiment and the second embodiment and further includes evaluating the patient position including evaluating a patient shoulder distance from the x-ray system, a patient head position, a patient foot position, a patient posture, a patient body rotation, and a patient spine position. A fourth embodiment of the method optionally includes one or more of the first embodiment through the third embodiment and further includes detecting, by the x-ray mammography system, one or more of a patient positioning error and a breast positioning error prior to image acquisition. A fifth embodiment of the method optionally includes one or more of the first embodiment through the fourth embodiment and further includes the patient positioning error being based on one or more of a patient bone model and a relative position of the patient with respect to the x-ray mammography system. A sixth embodiment of the method optionally includes one or more of the first embodiment through the fifth embodiment and further includes the breast positioning error being based on a desired acquisition view and one or more breast anatomical features of the breast in the imaging volume. A seventh embodiment of the method optionally includes one or more of the first embodiment through the sixth embodiment and further includes detecting one or more of a breast positioning error and a patient positioning error based on recognition of movement of one or more of the patient and the user.An eighth embodiment of the method optionally includes one or more of the first to seventh embodiments, and further includes: when the desired acquisition view is a cranio-caudal view, assessing breast anatomy includes detecting one or more of: pectoralis muscle inclusion, breast centering, nipple position, interbreast groove inclusion, and absence of one or more skin and fat folds; and providing real-time feedback based on each detection; and when the desired acquisition view is a medio-lateral oblique view, assessing breast anatomy includes detecting one or more of: pectoralis muscle position, nipple position, axillary tissue inclusion, inframammary fold inclusion, and absence of one or more skin and fat folds; and providing real-time feedback based on each detection. The ninth embodiment of the method optionally includes one or more of the first to eighth embodiments, and further includes: during one or more of the quality check mode and the cleaning operation mode of the x-ray mammography system, using the one or more cameras to assess user morphology and user position via visual sensing; and adjusting one or more components of the x-ray mammography system based on the user morphology and the user position; and wherein the one or more components include the position of the compression paddle holder on the gantry rail of the x-ray mammography system, the x-ray system height, and the control station height of the workstation. The tenth embodiment of the method optionally includes one or more of the first to ninth embodiments, and further includes: before compressing the patient's breast, using the one or more cameras to assess the patient's patient morphology and breast morphology of the patient via visual sensing; and identifying a preferred compression paddle based on the patient morphology and the breast morphology; and indicating the preferred compression paddle on the user interface. The eleventh embodiment of the method optionally includes one or more of the first to tenth embodiments, and further includes: before compressing the patient's breast, using the one or more cameras to assess the patient's patient morphology and breast morphology of the patient via visual sensing; and adjusting the support table position on the x-ray mammography system based on the patient morphology and the breast morphology.

[0188] In another embodiment, a method for an x-ray mammography system includes: determining a current operating mode, assessing a breast position and a patient position of a breast in an imaging volume of the x-ray mammography system based on the current mode via a vision system including one or more vision sensors; detecting one or more of a patient positioning error and a breast positioning error based on the assessment; and providing real-time feedback including an indication of one or more of an expected breast position and an expected patient position via a user interface of the mammography system based on the detection; wherein the patient positioning error is detected based on a patient anatomy and a patient position relative to the x-ray mammography system; and wherein the breast positioning error is detected based on one or more anatomical structures of the breast. In a first example of the method, the method may additionally or alternatively include: when the current mode is a digital breast tomosynthesis mode, assessing one or more of a breast positioning error and a patient positioning error at one or more x-ray source angulations within an angular range of the x-ray mammography system before initiating a tomosynthesis acquisition. The second embodiment of the method optionally includes the first embodiment, and further includes initiating x-ray image acquisition for a current mode; detecting movement of the patient during the x-ray image acquisition; and indicating an error due to a change in patient position in response to the detection; wherein the current mode is one of a digital breast tomosynthesis mode and an image-guided biopsy mode. The third embodiment of the method optionally includes one or more of the first and second embodiments, and further includes wherein the patient position is assessed based on a first camera image from a first visual sensor of the visual system, the first visual sensor having a first field of view of the x-ray mammography system and the patient, and the breast position is assessed based on a second camera image from a second visual sensor of the visual system, the second visual sensor having a second field of view including an imaging volume between a compression paddle and a detector of the x-ray mammography system. The fourth embodiment of the method optionally includes one or more of the first to third embodiments, and further includes wherein the first visual sensor and the second visual sensor are each RGB depth-type visual sensors, and wherein indications for one or more of an expected breast position and an expected patient position are provided on the first camera image and the second camera image, respectively; and further includes, before assessing the patient position and the breast position, assessing the first field of view of the first sensor and the second field of view of the second visual sensor for one or more glare areas. A fifth embodiment of the method optionally includes one or more of the first to fourth embodiments, and further includes: acquiring an x-ray image of the breast; detecting one or more second breast positioning errors based on the acquired x-ray image; and providing second feedback including a second indication about the acquired x-ray image, the second indication including the second breast positioning error and one or more desired breast structures to be imaged; wherein the second feedback is provided via a user interface of the mammography system.

[0189] The above-mentioned system and method are also used in a medical imaging system, which includes: a gantry, the gantry including a radiation source for emitting radiation rays, a detector for receiving radiation rays, and a collimator for adjusting the field of view; a visual sensor system, the visual sensor system including at least a first visual sensor and a second visual sensor; and a workstation, the workstation including a first user interface, the first user interface including a first display portion, the workstation is communicatively connected to the gantry, the gantry is connected to a second user interface including a second display portion; and wherein the workstation includes a processor, the processor is configured with instructions in a non-volatile memory, which instructions, when executed, cause the processor to: use the first visual sensor to acquire a first camera image of a patient and the medical imaging system; use the second visual sensor to acquire a second camera image of a breast in an imaging volume of the medical imaging system; use the first camera image and the second camera image to acquire a second camera image. a first camera image, determining a patient position relative to the medical imaging system and determining a breast position relative to the patient's entire body position and the medical imaging system based on the patient's breast anatomy; initiating image acquisition in response to the patient position being consistent with the first model and the breast position being consistent with the second model; otherwise, providing real-time feedback via a first user interface and a second user interface of the medical imaging system, and preventing image acquisition using a radiation source until the patient position is consistent with the first model and the breast position is consistent with the second model; and providing an option via one or more of the first user interface and the second interface for a user to initiate image acquisition when one or more of the following conditions exist: when the patient position is inconsistent with the first model, and when the breast position is inconsistent with the second model; wherein the first model and the second model are based on artificial intelligence algorithms using the first camera image and the second camera image as input. In a first example of the system, the system may additionally or alternatively include: wherein the first visual sensor has a first field of view of the medical imaging system and the patient, and the second visual sensor has a second field of view of the imaging volume of the medical imaging system; and wherein the first model is based on one or more of a skeletal model of the patient and a relative position of the patient relative to the medical imaging system; and wherein the second model is based on a desired acquisition view, and one or more breast anatomical features of the breast in the imaging volume.

[0190] As used herein, the elements or steps listed in the singular and beginning with the word "one" or "a kind of" should be understood as not excluding a plurality of the elements or steps, unless such exclusion is explicitly stated. In addition, the reference to "an embodiment" of the present invention is not intended to be interpreted as excluding the existence of additional embodiments that also include the cited features. In addition, unless explicitly stated to the contrary, the embodiment of "including", "comprising" or "having" an element or multiple elements with a specific characteristic may include additional such elements without the characteristic. The terms "including" and "in..." are used as the concise language equivalents of the corresponding terms "including" and "wherein". In addition, the terms "first", "second" and "third" etc. are only used as marks, and are not intended to impose numerical requirements or specific positional order on their objects.

[0191] This written description uses examples to disclose the invention, including the best mode, and also to enable a person of ordinary skill in the relevant art to practice the invention, including making and using any devices or systems and performing any included methods. The patentable scope of the invention is defined by the claims, and may include other examples that occur to a person of ordinary skill in the art. Such other examples are intended to fall within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insignificant differences from the literal language of the claims.

Claims

1. A method for an x-ray mammography system, include: assessing one or more of a patient position of a patient, a local view of the patient, and a breast anatomy of the patient via visual sensing using one or more cameras coupled to the x-ray mammography system; detecting one or more of a patient positioning error and a breast positioning error based on the evaluating; providing real-time feedback to a user via a user interface of the x-ray mammography system based on the detection, and Before compressing the patient's breast, assessing patient morphology and breast morphology of the patient via the visual sensing using the one or more cameras; identifying a preferred compression paddle based on the patient morphology and the breast morphology; as well as The preferred compression paddle is indicated on the user interface.

2. The method of claim 1, wherein the breast anatomy is assessed using a camera having a field of view that includes an imaging volume between a compression paddle and a detector of the x-ray mammography system.

3. The method of claim 1 , wherein the patient position is assessed based on input from a first camera having a first field of view of the x-ray mammography system and the patient, and the breast anatomy is assessed based on input from a second camera having a second field of view including an imaging volume between compression paddles and a detector of the x-ray mammography system.

4. The method of claim 1, wherein assessing patient position comprises assessing patient shoulder distance from the x-ray mammography system, patient head position, patient foot position, patient posture, patient body rotation, and patient spine position.

5. The method of claim 1, wherein the patient positioning error is based on one or more of a patient skeletal model and a relative position of the patient with respect to the x-ray mammography system.

6. The method of claim 2, wherein the breast positioning error is based on a desired acquisition view, and one or more breast anatomical features of a breast in the imaging volume.

7. The method of claim 1, wherein one or more of the breast positioning error and the patient positioning error is detected based on identifying movement of one or more of the patient and a user.

8. The method of claim 6, wherein when the desired acquisition view is a cranio-caudal view, assessing the breast anatomy comprises detecting one or more of: pectoralis muscle inclusion, breast centering, nipple location, interbreast cleavage inclusion, and absence of one or more skin and fat folds; and providing real-time feedback based on each detection; and when the desired acquisition view is a medio-lateral oblique view, assessing the breast anatomy comprises detecting one or more of: pectoralis muscle location, nipple location, axillary tissue inclusion, inframammary fold inclusion, and absence of one or more skin and fat folds; and providing real-time feedback based on each detection.

9. The method according to claim 1, further comprising: include: During one or more of a quality inspection mode and a cleaning mode of operation of the x-ray mammography system, assessing user form and user position via the visual sensing using the one or more cameras; as well as adjusting one or more components of the x-ray mammography system based on the user morphology and the user position; and Wherein the one or more components include the position of the compression paddle holder on the gantry rail of the x-ray mammography system, the x-ray mammography system height, and the control station height of the workstation.

10. A method for use in an x-ray mammography system, include: Determine the current operating mode, assessing, via a vision system including one or more vision sensors, a breast position and a patient position of a breast in an imaging volume of the x-ray mammography system based on the current mode; detecting one or more of a patient positioning error and a breast positioning error based on the evaluating; providing real-time feedback including an indication of one or more of a desired breast position and a desired patient position via a user interface of the x-ray mammography system based on the detecting; wherein the patient positioning error is detected based on patient anatomy and a patient position relative to the x-ray mammography system; wherein the breast positioning error is detected based on one or more anatomical structures of the breast; as well as Before compressing the patient's breast, assessing patient morphology and breast morphology of the patient via the vision system using the one or more vision sensors; identifying a preferred compression paddle based on the patient morphology and the breast morphology; as well as The preferred compression paddle is indicated on the user interface.

11. The method according to claim 10, further comprising: include: When the current mode is a digital breast tomosynthesis mode, one or more of the breast positioning error and the patient positioning error are evaluated at one or more x-ray source angulations within an angular range of the x-ray mammography system prior to initiating a tomosynthesis acquisition.

12. The method of claim 10, wherein the patient position is assessed based on a first camera image from a first vision sensor of the vision system, the first vision sensor having a first field of view of the x-ray mammography system and the patient, and the breast position is assessed based on a second camera image from a second vision sensor of the vision system, the second vision sensor having a second field of view including an imaging volume between a compression paddle and a detector of the x-ray mammography system.

13. The method of claim 12, wherein the first vision sensor and the second vision sensor are each RGB depth-type vision sensors, and wherein an indication of one or more of an expected breast position and an expected patient position is provided on the first camera image and the second camera image, respectively; and further comprising, prior to evaluating the patient position and the breast position, evaluating the first field of view of the first vision sensor and the second field of view of the second vision sensor for one or more glare regions.

14. A medical imaging system, include: A frame, the frame comprising a radiation source for emitting radiation rays, a detector for receiving the radiation rays, and a collimator for adjusting the field of view; A vision sensor system, the vision sensor system comprising at least a first vision sensor and a second vision sensor; and a workstation including a first user interface including a first display portion, the workstation being communicatively coupled to the housing, the housing being coupled to a second user interface including a second display portion; and wherein the workstation comprises a processor configured with instructions in a non-transitory memory, the instructions when executed causing the processor to: capturing a first camera image of a patient and the medical imaging system using the first visual sensor; acquiring, using the second visual sensor, a second camera image of a breast in an imaging volume of the medical imaging system; determining a patient position relative to the medical imaging system and a breast position relative to the patient's entire body position and the medical imaging system based on the patient's breast anatomy using the first camera image and the second camera image; initiating image acquisition in response to the patient position consistent with the first model and the breast position consistent with the second model; otherwise, providing real-time feedback via the first user interface and the second user interface of the medical imaging system, and preventing image acquisition with the radiation source until the patient position is consistent with the first model and the breast position is consistent with the second model; as well as providing, via one or more of the first user interface and the second user interface, an option for a user to initiate image acquisition when one or more of the following conditions exist: when the patient position is inconsistent with the first model, and when the breast position is inconsistent with the second model; wherein the first model and the second model are based on artificial intelligence algorithms using the first camera image and the second camera image as input, and assessing the patient morphology and breast morphology of the patient via the vision sensor system using the first vision sensor and the second vision sensor prior to compressing the patient's breast; identifying a preferred compression paddle based on the patient morphology and the breast morphology; as well as The preferred compression paddle is indicated on the user interface.

Images