Automation of transvaginal ultrasound workflow

Abstract

The disclosure relates to automation of transvaginal ultrasound workflow. The present disclosure provides systems and methods for improving the visualization of an image volume of a uterus (801, 1006) and / or endometrium (1102, 700, 1002, 902, 802) of a subject acquired using a transvaginal ultrasound system (100) (TVUS). In one example, a method (400, 300) for the TVUS includes extracting a central axis (1004) of an endometrium (1102, 700, 1002, 902, 802) of a received two-dimensional (2D) ultrasound image (600, 500, 1100) of a uterus (801, 1006) of a subject; generating a uterine trace (804, 702) from the extracted central axis (1004); acquiring a three-dimensional (3D) image volume of the uterus (801, 1006) according to the uterus trace (804, 702); and displaying the 3D image volume on a display device (118, 234) of the TVUS.

Description

Technical Field

[0001] Embodiments of the subject matter disclosed herein relate to ultrasound imaging, and more particularly to systems and methods for automating ultrasound examination workflow. Background Art

[0002] Clinical ultrasound is an imaging modality that uses ultrasound waves to detect the internal structures of a patient's body and produce corresponding images. An ultrasound probe including multiple transducer elements transmits ultrasound pulses, which are reflected or transmitted back, refracted or absorbed by structures in the body. The ultrasound probe then receives the reflected echoes, which are processed into an image. For example, medical imaging devices such as ultrasound imaging devices can be used to obtain images of a patient's heart, uterus, liver, lungs, and various other anatomical parts. In some applications, such as in a transvaginal ultrasound system (TVUS), an ultrasound probe can be inserted into a cavity of the body.

[0003] Classifying uterine activity according to the guidelines of the European Society of Human Reproduction and Embryology (ESHRE) and the American Society of Reproductive Medicine (ASRM) is an important step in ensuring uterine health. Classifying uterine activity relies on performing a 3D acquisition of the endometrial region along the subject's uterus from the subject's uterine fundus to the cervix. However, in the image volume produced by a typical acquisition of the uterus and endometrium with a 3D TVUS probe, the endometrial structures may not be well aligned with any of the standard cross-sectional planes in the multi-planar views of the image volume. Summary of the invention

[0004] The present disclosure at least partially solves one or more of the above-identified problems by a method for TVUS, the TVUS comprising an image processing system comprising a processor and a non-volatile memory storing instructions that, when executed, cause the processor to display a three-dimensional (3D) ultrasound image volume of a uterus of a subject scanned using TVUS on a display device of the TVUS, the display aligning the endometrium of the uterus with a cross-sectional plane of the 3D ultrasound image volume. The cross-sectional plane may be a midsagittal plane of the 3D ultrasound image volume. In a first embodiment, a median axis of the endometrium is extracted from a two-dimensional (2D) ultrasound image of the uterus generated by a previous 2D scan of the uterus, and the median axis is used to generate a uterine trace. A 3D image volume of the uterus is then generated based on the uterine trace, wherein the endometrium is aligned with the cross-sectional plane of the 3D image volume. In a second embodiment, segmentation is performed on the endometrium of a 3D ultrasound image volume, and then the 3D ultrasound image volume is rotated using a rotation vector estimated from a set of eigenvectors of the segmented endometrium to align the long axis of the segmented endometrium with a cross-sectional plane of the 3D ultrasound image volume.

[0005] The above advantages and other advantages and features of the present specification will be apparent from the following detailed description when considered alone or in conjunction with the accompanying drawings. It should be understood that the above summary is provided to introduce a series of concepts further described in the detailed description in a simplified form. This is not meant to identify key features or essential features of the claimed subject matter, the scope of which is uniquely defined by the claims that follow the detailed description. In addition, the claimed subject matter is not limited to specific implementations that address any shortcomings mentioned above or in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Various aspects of the present disclosure may be better understood by reading the following detailed description and referring to the accompanying drawings, in which:

[0007] Figure 1 A block diagram of an exemplary embodiment of an ultrasound system is shown;

[0008] Figure 2 A block diagram illustrating an exemplary embodiment of an image processing system is shown;

[0009] Figure 3 is a flow chart illustrating an exemplary process for automating a first workflow for acquiring an image volume of the uterus using TVUS;

[0010] Figure 4 is a flow chart illustrating an exemplary process for a second workflow for automating acquisition of an image volume of the uterus;

[0011] Figure 5 An exemplary segmentation of the uterus in a 2D ultrasound image is shown;

[0012] Figure 6 An exemplary segmentation of the endometrium in a 2D ultrasound image is shown;

[0013] Figure 7 An exemplary central axis of the endometrium is shown;

[0014] Figure 8 An exemplary extrapolation of the mid-endometrial axis across the uterus is shown;

[0015] Fig. 9 is a first 2D ultrasound image showing the endometrium which is not aligned with a standard cross-sectional plane of the image volume as in the prior art;

[0016] Fig.10 is a second 2D ultrasound image showing the endometrium aligned with a standard cross-sectional plane of the image volume as in the prior art; and

[0017] Fig.11 is a third ultrasound image showing the major axes of the endometrium according to one or more embodiments of the present disclosure.

[0018] The accompanying drawings illustrate certain aspects of the described systems and methods for mapping one or more ultrasound images at a first resolution to one or more corresponding ultrasound images at a target resolution using a generative neural network. Together with the following description, the accompanying drawings illustrate and explain the structures, methods, and principles described herein. In the accompanying drawings, the sizes of components may be exaggerated or otherwise modified for clarity. Well-known structures, materials, or operations are not shown or described in detail to avoid obscuring aspects of the described components, systems, and methods. DETAILED DESCRIPTION

[0019] Methods and systems are described herein for improving visualization of an image volume of the uterus and / or endometrium of a subject acquired using a transvaginal ultrasound system (TVUS). One difficulty in visualizing such an image volume is that the endometrium may not be well aligned with any of the standard cross-sectional planes of the multi-planar views of the image volume on a display device of the TVUS. Therefore, alignment of the endometrium with a selected cross-sectional plane of the image volume is typically adjusted according to a manual workflow. The cross-sectional planes may be referred to as the A-plane, the B-plane, and the C-plane relative to the anatomy of the subject.

[0020] In a first example of a manual workflow, an initial scan may be performed using a two-dimensional (2D) ultrasound probe. A uterine trace, also referred to as a panoramic view line, may be manually drawn by the user of the TVUS on the 2D image reconstructed from the scan. A 3D acquisition may then be performed using this uterine trace as a guide to ensure that the endometrium is aligned with a cross-sectional plane (such as the mid-sagittal plane) of the resulting image volume. If the endometrium is not aligned with the cross-sectional plane, the size (e.g., volume, diameter, etc.) of the endometrium may not be measured.

[0021] However, this manual workflow has several disadvantages. First, performing the manual step of generating a 3D image volume in the proper orientation after performing the first 2D scan may increase the amount of time spent by the user of the ultrasound system and may increase the use of ultrasound system resources, resulting in a corresponding reduction in resources available for other tasks, all of which may increase the cost of operating the ultrasound system. In addition, the endometrium may be difficult to see because lesions or uterine abnormalities may distort the appearance of the endometrium and / or surrounding features. Therefore, the accuracy of alignment of the uterus / endometrium with the cross-sectional plane may depend on the user's experience level, and inexperienced users may attempt to redraw the uterine trace multiple times and perform various scans to achieve the desired alignment, further increasing resource usage and costs. Because the size and shape of the anatomical features of the uterus fluctuate during the menstrual cycle, the user's learning curve for accurately drawing the uterine trace may be slow.

[0022] In a second example of a manual workflow, a 3D TVUS acquisition may be performed without any pre-scan to generate a 3D image volume. Without a pre-scan, the endometrium may not be present in the mid-sagittal plane. To bring the endometrium into the mid-sagittal plane, a z-plane rotation technique may be manually performed by the user. However, rotating the image volume to achieve the desired alignment may be difficult and may rely on user experience for the reasons described above. Typically, the user may rotate the image volume back and forth multiple times to align the endometrium, which increases the amount of time the user spends viewing the ultrasound image and the use of ultrasound system resources (e.g., memory, processing power, etc.).

[0023] Therefore, in both the first exemplary workflow and the second exemplary workflow described above, one or more manual steps may be performed, which may be time-consuming and / or cumbersome, leading to increased usage of the TVUS system and reduced usability of the TVUS system. To address this issue, a method of automating these workflows is proposed, which enables a 3D image volume to be displayed on a display device of a TVUS system, wherein the endometrium is aligned with the mid-sagittal plane, without involving manual intervention.

[0024] In various embodiments, ultrasound imaging systems (such as Figure 1 The ultrasound imaging system may be a TVUS. The ultrasound imaging system may be communicatively coupled to an image processing system, such as Figure 2 The image processing system 202. Reference Figure 3 A first method for automating a first workflow for displaying an image volume is described and reference is made to Figure 4 A second method for automating a second workflow for displaying an image volume is described. Either or both of the first workflow and the second workflow may include performing segmentation of a uterus of a subject included in the image volume, such as Figure 5 As shown, and segmentation is performed on the endometrium of the subject included in the image volume, as Figure 6 As shown. Figure 7 As shown, a uterine trajectory for aligning the image volume on the display screen of TVUS can be generated by fitting a polynomial function to the extracted central axis of the endometrium. Figure 8 As shown, a uterine trajectory generated from the endometrium can be extrapolated to span the uterus. The image volume can then be rotated according to the uterine trajectory to display the endometrium aligned with the midsagittal plane of the TVOS, as shown in FIG. Fig. 9 shown.

[0025] See now Figure 1, which shows a schematic diagram of an ultrasound imaging system 100 according to an embodiment of the present disclosure. The ultrasound imaging system 100 includes a transmit beamformer 101 and a transmitter 102, which drives an element (e.g., a transducer element) 104 within a transducer array (referred to herein as a probe 106) to transmit a pulsed ultrasound signal (referred to herein as a transmit pulse) into a body (not shown). The probe 106 can be a one-dimensional transducer array probe, or the probe 106 can be a two-dimensional matrix transducer array probe. As further explained below, the transducer element 104 can be made of a piezoelectric material. When a voltage is applied to a piezoelectric crystal, the crystal physically expands and contracts, thereby emitting an ultrasonic spherical wave. In this way, the transducer element 104 can convert an electronic transmit signal into an acoustic transmit beam.

[0026] After the elements 104 of the probe 106 transmit the pulsed ultrasound signals into the body (of the patient), the pulsed ultrasound signals are backscattered from structures inside the body (such as blood cells or muscle tissue) to produce echoes that return to the elements 104. The echoes are converted into electrical signals or ultrasound data by the elements 104, and the electrical signals are received by the receiver 108. The electrical signals representing the received echoes pass through the receive beamformer 110, which outputs the ultrasound data. Additionally, the transducer elements 104 may generate one or more ultrasound pulses based on the received echoes to form one or more transmit beams.

[0027] According to some embodiments, the probe 106 may contain electronic circuits to perform all or part of the transmit beamforming and / or receive beamforming. For example, all or part of the transmit beamformer 101, the transmitter 102, the receiver 108, and the receive beamformer 110 may be located within the probe 106. In the present disclosure, the term "scan" may also be used to refer to the acquisition of data by the process of transmitting and receiving ultrasound signals. In the present disclosure, the term "data" may be used to refer to one or more data sets acquired with an ultrasound imaging system. In one embodiment, the data acquired via the ultrasound system 100 may be used to train a machine learning model. The user interface 115 may be used to control the operation of the ultrasound imaging system 100, including input for controlling patient data (e.g., patient medical history), for changing scanning or display parameters, for starting a probe repolarization sequence, etc. The user interface 115 may include one or more of the following: a rotating element, a mouse, a keyboard, a trackball, a hard key linked to a specific action, a soft key that can be configured to control different functions, and / or a graphical user interface displayed on a display device 118.

[0028] The ultrasound imaging system 100 also includes a processor 116 that controls the transmit beamformer 101, the transmitter 102, the receiver 108, and the receive beamformer 110. The processor 116 is in electronic communication (e.g., communicatively connected) with the probe 106. For purposes of this disclosure, the term "electronic communication" may be defined to include both wired communication and wireless communication.

[0029] The processor 116 can control the probe 106 to collect data according to instructions stored in the processor's memory and / or memory 120. The processor 116 can control which control elements in the element 104 are active and the shape of the beam emitted from the probe 106. The processor 116 also communicates electronically with the display device 118, and the processor 116 can process data (e.g., ultrasound data) into an image for display on the display device 118. According to an embodiment, the processor 116 may include a central processing unit (CPU). According to other embodiments, the processor 116 may include other electronic components capable of performing processing functions, such as a digital signal processor, a field programmable gate array (FPGA), or a graphics board. According to other embodiments, the processor 116 may include multiple electronic components capable of performing processing functions. For example, the processor 116 may include two or more electronic components selected from a list of electronic components, which include: a central processing unit, a digital signal processor, a field programmable gate array, and a graphics board. According to another embodiment, the processor 116 may also include a composite demodulator (not shown) that demodulates RF data and generates raw data. In another embodiment, demodulation may be performed earlier in the processing chain.

[0030] The processor 116 is adapted to perform one or more processing operations according to a plurality of optional ultrasound modalities on the data. In one example, the data may be processed in real time during a scanning session because the echo signals are received by the receiver 108 and transmitted to the processor 116. For the purposes of this disclosure, the term "real time" is defined as including a program executed without any intentional delay. For example, an embodiment may acquire images at a real-time frame rate of 7 frames / second to 20 frames / second. The ultrasound imaging system 100 may acquire 2D data of one or more planes at a significantly faster rate. However, it should be understood that the real-time frame rate may depend on the length of time it takes to acquire each frame of data for display. Therefore, when a relatively large amount of data is acquired, the real-time frame rate may be slower. Therefore, some embodiments may have a real-time frame rate significantly faster than 20 frames / second, while other embodiments may have a real-time frame rate lower than 7 frames / second.

[0031] The data may be temporarily stored in a buffer (not shown) during a scanning session and processed in less than real time in real time or off-line operation. In some embodiments, multiple processors (not shown) may be included to handle processing tasks handled by processor 116 according to the exemplary embodiments described above. For example, a first processor may be utilized to demodulate and extract the RF signal before displaying an image, while a second processor may be utilized to further process the data (e.g., by augmenting the data as further described herein). It should be appreciated that other embodiments may use different processor arrangements.

[0032] The ultrasound imaging system 100 can continuously acquire data at a frame rate of, for example, 10 Hz to 30 Hz (e.g., 10 to 30 frames per second). The image generated based on the data can be refreshed at a similar frame rate on the display device 118. Other embodiments can acquire and display data at different rates. For example, depending on the size of the frame and the intended application, some embodiments can acquire data at a frame rate less than 10 Hz or greater than 30 Hz. A memory 120 is included to store processed frames of the acquired data. In an exemplary embodiment, the memory 120 has sufficient capacity to store at least a few seconds of ultrasound data frames. The data frames are stored in a manner that is easy to retrieve according to their acquisition order or time. The memory 120 may include any known data storage medium.

[0033] In various embodiments of the present invention, the processor 116 may process data in different mode-related modules (e.g., B-mode, color Doppler, M-mode, color M-mode, spectral Doppler, elastic imaging, TVI, strain, strain rate, etc.) to form 2D or 3D data. For example, one or more modules may generate B-mode, color Doppler, M-mode, color M-mode, spectral Doppler, elastic imaging, TVI, strain, strain rate, and combinations thereof, etc. As an example, one or more modules may process color Doppler data, which may include traditional color blood flow Doppler, power Doppler, HD flow, etc. The image lines and / or frames are stored in the memory and may include timing information indicating the time when the image lines and / or frames are stored in the memory. These modules may include, for example, a scan conversion module that performs a scan conversion operation to convert the acquired image from beam space coordinates to display space coordinates. A video processor module may be provided that reads the acquired image from the memory and displays the image in real time while a procedure (e.g., ultrasound imaging) is performed on the patient. The video processor module may include a separate image memory, and the ultrasound image may be written to the image memory for reading and display by the display device 118 .

[0034] In various embodiments of the present disclosure, one or more components of the ultrasound imaging system 100 may be included in a portable handheld ultrasound imaging device. For example, the display device 118 and the user interface 115 may be integrated into the external surface of the handheld ultrasound imaging device, which may further contain a processor 116 and a memory 120. The probe 106 may include a handheld probe that electronically communicates with the handheld ultrasound imaging device to collect raw ultrasound data. The transmit beamformer 101, the transmitter 102, the receiver 108, and the receive beamformer 110 may be included in the same or different parts of the ultrasound imaging system 100. For example, the transmit beamformer 101, the transmitter 102, the receiver 108, and the receive beamformer 110 may be included in a handheld ultrasound imaging device, a probe, and a combination thereof.

[0035] After performing an ultrasound scan, a two-dimensional data block including scan lines and their samples is generated for each row of transducers included in the ultrasound probe (e.g., one data block for a 1D probe, or n data blocks for a 2D probe with n rows of transducers). After applying the back-end filter, a process called scan conversion is performed to transform the two-dimensional data block into a displayable bitmap image with additional scanning information (such as depth, angle of each scan line, etc.). During scan conversion, interpolation techniques are applied to fill in missing holes (e.g., pixels) in the resulting image. These missing pixels occur because each element of the two-dimensional block should generally cover many pixels in the resulting image. For example, in current ultrasound imaging systems, bicubic interpolation is applied, which utilizes adjacent elements of the two-dimensional block. Therefore, if the two-dimensional block is relatively small compared to the size of the bitmap image, the scan-converted image will include areas of poor or low resolution, especially for areas with greater depth.

[0036] The ultrasound images acquired by the ultrasound imaging system 100 may be further processed. In some embodiments, as described in more detail below, the ultrasound images generated by the ultrasound imaging system 100 may be transmitted to an image processing system where the ultrasound images may be processed by one or more machine learning (ML) models.

[0037] Although described herein as separate systems, it should be understood that in some embodiments, the ultrasound imaging system 100 includes an image processing system. In other embodiments, the ultrasound imaging system 100 and the image processing system may include separate devices. In some embodiments, the images generated by the ultrasound imaging system 100 may be used as a training data set for training one or more machine learning models, where the machine learning model may be used to perform one or more steps of ultrasound image processing as described below.

[0038] refer to Figure 2, block diagram 200 shows an image processing system 202 according to an embodiment. In some embodiments, the image processing system 202 is incorporated into the ultrasound imaging system 100. For example, the image processing system 202 may be provided in the ultrasound imaging system 100 as a processor 116 and a memory 120. In some embodiments, at least a portion of the image processing system 202 is disposed at a device (e.g., an edge device, a server, etc.) that is communicatively coupled to the ultrasound imaging system via a wired connection and / or a wireless connection. In some embodiments, at least a portion of the image processing system 202 is disposed at a separate device (e.g., a workstation) that can receive images from the ultrasound imaging system or from a storage device that stores images / data generated by the ultrasound imaging system. The image processing system 202 may be operably / communicatively coupled to a user input device 232 and a display device 234. At least in some examples, the user input device 232 may include a user interface 115 of the ultrasound imaging system 100, and the display device 234 may include a display device 118 of the ultrasound imaging system 100. The image processing system 202 may also be operably / communicatively coupled to an ultrasound probe 236 .

[0039] The image processing system 202 includes a processor 204 configured to execute machine-readable instructions stored in a non-transitory memory 206. The processor 204 may be a single-core or multi-core processor, and the program executed thereon may be configured for parallel processing or distributed processing. In some embodiments, the processor 204 may optionally include separate components distributed on two or more devices, which may be located at a distance and / or configured for coordinated processing. In some embodiments, one or more aspects of the processor 204 may be virtualized and performed by a remotely accessible networked computing device configured in a cloud computing configuration.

[0040] The non-transitory memory 206 may store an artificial intelligence (AI) module 208, a segmentation module 210, and an image database 214. The AI ​​module 208 may include one or more rule-based systems and / or machine learning models (e.g., neural networks), and instructions for implementing these systems or ML models to calculate the desired rotation and / or alignment of the 3D image volume on the display device 234 of the ultrasound system, as described in more detail below. The AI ​​module 208 may include trained and / or untrained neural networks, and may also include various data or metadata related to one or more neural networks stored therein. In some embodiments, the AI ​​module 208 may include instructions for implementing one or more gradient descent algorithms, applying one or more loss functions and / or training routines for adjusting parameters of one or more neural networks of the AI ​​module 208. The AI ​​module 208 may include a training data set for one or more neural networks of the AI ​​module 208. In addition, the AI ​​module 208 may include other types of models, such as statistical models, probabilistic models, and / or mathematical models.

[0041] Specifically, the AI ​​module 208 may store one or more ML segmentation models for performing segmentation on anatomical features of the ultrasound image. The one or more ML segmentation models may include a 2D segmentation model and a 3D segmentation model. For example, a first segmentation model may be used to segment the endometrium and / or uterus of a subject in a 2D ultrasound image, as described below with reference to Figure 5 and Figure 6 The second 3D segmentation model can be used to segment the endometrium in the 3D image volume, as described below with reference to Figure 4 described.

[0042] The image database 214 may include ultrasound images acquired via an ultrasound probe, including 2D images and 3D images.

[0043] In some embodiments, non-transitory memory 206 may include components disposed on two or more devices that may be remotely located and / or configured for coordinated processing. In some embodiments, one or more aspects of non-transitory memory 206 may include a remotely accessible networked storage device configured in a cloud computing configuration.

[0044] The display device 234 may include one or more display devices utilizing nearly any type of technology. In some embodiments, the display device 234 may include a computer monitor and may display ultrasound images. The display device 234 may be combined with the processor 204, the non-transitory memory 206, and / or the user input device 232 in a common housing, or may be a peripheral display device and may include a monitor, a touch screen, a projector, or other display devices known in the art that may enable a user to view ultrasound images generated by the ultrasound imaging system and / or interact with various data stored in the non-transitory memory 206.

[0045] The user input device 232 may include one or more of a touch screen, a keyboard, a mouse, a trackpad, a motion sensing camera, or other devices configured to enable a user to interact with and manipulate data within the image processing system 202. In one example, the user input device 232 may enable a user to draw a uterine trace on a 2D ultrasound image of the uterus. The uterine trace may be used to align the endometrium of the uterus with a cross-sectional plane of a 3D image volume on the display device 234 that includes the uterus.

[0046] It should be understood that Figure 2 The image processing system 202 shown is for illustration and not for limitation. Another suitable image processing system may include more, fewer, or different components.

[0047] See now Figure 3 , which illustrates an exemplary method 300 for automating a first workflow followed by a user of an ultrasound system to align an image volume on a display device of the ultrasound system so that anatomical features of the patient are displayed in a standard cross-sectional plane of the image volume. For example, the cross-sectional plane may be a midsagittal plane of the image volume. The ultrasound system may be a TVUS, and the anatomical features may include the patient's uterus and / or endometrium. The method 300 may be performed by an image processing system (such as a Figure 2 A processor (eg, processor 204) of image processing system 202 executes instructions stored in a memory (eg, memory 206) of the image processing system.

[0048] Method 300 begins at 302, where the method includes receiving a 2D ultrasound image acquired from a 2D pre-scan of a patient using TVUS. The 2D ultrasound image may be acquired via an ultrasound probe of an ultrasound system, where the ultrasound probe is configured in a 2D mode. The ultrasound image may include a uterus of the patient. The pre-scan may be used to determine an alignment of the patient's endometrium so that a second 3D ultrasound scan may be performed using a configuration of the ultrasound probe in a 3D mode that will generate an image volume in which the endometrium is aligned with a standard cross-sectional plane of the image volume.

[0049] For example, Fig. 9 A first 2D ultrasound image 900 including endometrium 902 is shown, wherein the endometrium 902 is not aligned with a standard cross-sectional plane of the image volume. In the first 2D ultrasound image 900, the endometrium 902 is displayed as a small oval, wherein the extent of the endometrium 902 is not visible and various features and / or characteristics of the endometrium cannot be seen. In contrast, Fig.10 A second 2D ultrasound image 1000 is shown that includes the endometrium 1002, wherein the endometrium 1002 is aligned with the mid-sagittal plane of the image volume. In the second 2D ultrasound image 1000, the endometrium 1002 is shown in a side perspective, wherein the extent of the endometrium 1002 can be seen as well as features and / or characteristics of the endometrium 1002 visible from the side perspective, such as a mid-axis 1004 of the endometrium 1002. Additionally, the shape of the uterus 1006 and the alignment of the endometrium 1002 with the uterus 1006 can be seen.

[0050] Returning to method 300, at 304, method 300 includes performing segmentation on the uterus. In various embodiments, a segmentation model of an image processing system (e.g., Figure 2 The AI ​​module 208) performs segmentation on the uterus.

[0051] Figure 5 An exemplary ultrasound image 500 including a segmented uterus 502 is shown. In various embodiments, the uterus can be segmented using a convolutional neural network (CNN) trained on uterus and endometrial markers. The CNN can have an architecture such as UNet, MaskRCNN, etc.

[0052] At 306 , method 300 includes performing segmentation on the endometrium of the uterus using the segmentation model. Figure 6 An exemplary ultrasound image 600 including a segmented uterus 602 is shown.

[0053] At 308, method 300 includes extracting a median axis of the segmented endometrium. Extracting the median axis may rely on a morphological thinning method that successively erodes pixels from the boundary of the segmented endometrium while preserving the endpoints of the line segments of the boundary until no further thinning is possible, at which point the remaining portion approximates the median axis. The median axis may include a set of points extending through the middle portion of the segmented endometrium.

[0054] At 310, method 300 includes generating a uterine trace by fitting a polynomial function to the extracted medial axis. Figure 7 , an exemplary representation of the endometrium 700 is shown, wherein an exemplary uterine trace 702 is generated by fitting a polynomial function to a set of points forming a medial axis and plotted on the endometrium 700 .

[0055] At 312, method 300 includes extrapolating a uterine trace to span the uterus from a uterine fundus end of the uterus to a cervical end of the uterus. In other words, the uterine trace defines a long axis of the uterus, and it is desired to display the uterus with the long axis aligned with a cross-sectional plane of a 3D image volume of the uterus in a multi-planar view on a display device. Because the endometrium is positioned along the long axis of the uterus, the endometrium can be used as a guide to generate the uterine trace. However, the endometrium may not span the entire length of the uterus along the long axis. Therefore, in order to generate a complete uterine trace, the (initial) uterine trace generated from the extracted median axis of the endometrium can be extrapolated according to a polynomial fit to extend from the patient's uterine fundus at a first end of the uterus to the patient's cervix at a second end of the uterus.

[0056] Figure 8 An exemplary extrapolated uterine trace is shown in . Figure 8 , a 2D ultrasound image 800 including a uterus 801 is shown. The view of the 2D ultrasound image 800 corresponds to the midsagittal plane of the 3D image volume including the 2D ultrasound image 800. Figure 8 801 is not aligned with the mid-sagittal plane. A first end 806 of the uterus 801 is shown, but a second end 808 of the uterus 801 is not visible. Because the second end 808 is not visible and the uterus 801 is not centered, reliable measurements (e.g., volume, etc.) of the uterus 801 may not be possible. In order to view the entire uterus 801, the image volume may be rotated in a subsequent 3D acquisition, which may be performed based on the uterine trace 804 of the 2D ultrasound image 800. The uterine trace 804 may be generated by fitting a polynomial function to the mid-axis of the endometrium 802, as described above in Figure 3 The method 300 is described in detail.

[0057] However, in order to accurately position the endometrium 802 and the uterus 801 in the center of the midsagittal plane, the uterine trace 804 is extrapolated to span the uterus 801 from a first end 806 (e.g., the fundus end of the uterus 801) to a second end 808 (e.g., the cervical end of the uterus 801). To extrapolate the uterine trace 804, a first extension 814 of the uterine trace 804 to the first end 806 (e.g., the cervix) and a second extension 810 of the uterine trace 804 to the second end 808 (e.g., the fundus) are generated using a polynomial function for fitting the medial axis. The extension 814 has a length 816 from the first end of the endometrium 802 to the first end 806 of the uterus 801, and the extension 810 has a length 812 from the second end of the endometrium 802 to the second end 808 of the uterus 801.

[0058] Returning to method 300, at 314, method 300 includes acquiring a 3D image volume based on the extrapolated uterine trace. Once the trace is drawn, the 3D acquisition can be performed automatically. The user acquires the endometrial plane while scanning, and draws a uterine trace that passes through the endometrium centered from the cervix to the fundus. The uterine trace ensures that the cervix, fundus, and endometrium are clearly visible. Once the trace is drawn, the 3D acquisition is triggered to ensure that the image on which the uterine trace is drawn is in the mid-sagittal plane. For example, the mechanical 3D probe can scan from about -45° to +45°, where the 0th angle corresponds to the plane in which the uterine trace is drawn.

[0059] At 316, method 300 includes displaying the 3D image volume on a display device of the TVUS, and method 300 ends. When the 3D image volume is displayed on the display device, the patient's endometrium and uterus (e.g., Figure 8 The endometrium 802 and uterus 801) can be centered in the midsagittal plane (or other cross-sectional plane) of the 3D image volume.

[0060] See now Figure 4 , which illustrates an exemplary method 400 for automating a second workflow followed by a user of an ultrasound system to align an image volume on a display device of the ultrasound system so that anatomical features of the patient are displayed in a standard cross-sectional plane (e.g., a midsagittal plane) of an image viewing application of the ultrasound system used to view the image volume. The ultrasound system may be a TVUS, and the anatomical features may include a uterus and / or endometrium of the patient. The method 400 may be performed by an image processing system (such as a Figure 2 A processor (eg, processor 204) of image processing system 202 executes instructions stored in a memory (eg, memory 206) of the image processing system.

[0061] Method 400 begins at 402, where the method includes receiving a 3D ultrasound image acquired from a 3D scan of a patient using TVUS. The ultrasound image may include a uterus of the patient.

[0062] At 404, method 400 includes using the Figure 3 The segmentation model described in method 300 performs segmentation on the endometrium of the uterus. The segmentation model may be stored in a non-transitory memory of the image processing system (e.g., stored in Figure 2The segmentation may then be used to generate a mask of the endometrium, wherein the mask may be used to generate a masked version of the 3D ultrasound image. The masked version may include image data of the 3D ultrasound image inside the endometrium and exclude image data outside the endometrium. The mask may include a 3D matrix of weight values ​​that may be multiplied by the image data value of each voxel of the 3D ultrasound image. For example, the weight values ​​of the mask assigned to the voxels of the 3D ultrasound image outside the endometrium may be 0.0, and the weight values ​​of the mask assigned to the voxels of the 3D ultrasound image inside the endometrium may be 1.0.

[0063] At 406, method 400 includes estimating a set of eigenvectors of the masked endometrium. In a first step, a covariance matrix of voxel coordinates in 3D space (x, y, z) may be calculated, the covariance matrix corresponding to the masked endometrium. Eigenvalues ​​and eigenvectors of the covariance matrix may then be calculated. The eigenvectors may determine the principal axes of rotation of the endometrium. For example, the largest eigenvector may be aligned with the longest dimension of the endometrium, which corresponds to the long axis of the endometrium. The second eigenvector and the third eigenvector may be aligned with other short axes of the endometrium.

[0064] Brief reference Fig.11 , an ultrasound image 1100 of the endometrium 1102 is shown, wherein a set of principal axes 1105 of the endometrium 1102 are shown superimposed on the ultrasound image 1100. The major axis 1104 may correspond to the largest first eigenvector; the first minor axis 1106 may correspond to the smaller second eigenvector; and the second minor axis 1108 may correspond to the even smaller third eigenvector. Distinguishing the second eigenvector from the third eigenvector may be complicated due to changes in the size and shape of the endometrium during the patient's menstrual cycle. The principal axes may represent axes about which the endometrium 1102 can be rotated to align the endometrium 1102 with a desired cross-sectional plane in order to view the full extent of the endometrium 1102.

[0065] By calculating the covariance matrix and eigenvectors for the masked endometrium rather than the 3D ultrasound image volume, the amount of processing performed during the calculation of the eigenvalues ​​and eigenvectors of the covariance matrix can be reduced. Thus, the computational resources of the ultrasound system (e.g., Figure 1 The invention can reduce the loss of processor 116 and memory 120), thereby improving performance and improving the operation of the ultrasound system.

[0066] At 408, method 400 includes estimating a rotation vector based on the estimated set of eigenvectors. The rotation vector may align a long axis of the endometrium characterized by a maximum eigenvector with a desired cross-sectional plane of the 3D ultrasound image volume.

[0067] At 410, the method 400 includes performing a 3D rotation on the 3D ultrasound image volume according to the rotation vector. The volume may be rotated so that the plane described by the maximum eigenvector and the minimum eigenvector is aligned with the A-plane. The above assumption is valid mainly in stage 2 or stage 3 endometrium. When the above assumption is not true, outliers (stage 0 and stage 1) may exist and these outliers may be eliminated based on the detected endometrial volume falling below a certain threshold.

[0068] At 412, method 400 includes displaying the rotated 3D ultrasound image volume on a display device of the TVUS, and method 400 ends. When the 3D ultrasound image volume is displayed on the display device, the patient's endometrium and uterus can be centered in the midsagittal plane (or other cross-sectional plane) of the 3D ultrasound image volume. Fig.10 An example of a 3D ultrasound image volume with the endometrium and uterus centered in the mid-sagittal plane is shown in .

[0069] Therefore, two methods are proposed for automatically acquiring a 3D ultrasound image volume of a patient's uterus, wherein the uterus has a desired alignment with a cross-sectional plane of the image volume. In the first method, a uterine trace is automatically generated on a 2D ultrasound image based on segmenting the endometrium of the uterus, extracting the medial axis of the segmented endometrium, and fitting a polynomial function to the extracted medial axis. A 3D image volume is then acquired based on the automatically generated uterine trace. In the second method, a 3D segmentation is performed on the endometrium, and a set of eigenvectors of the segmented endometrium is estimated based on the covariance matrix of the pixel coordinates in the 3D space corresponding to the endometrial mask. The calculations performed to estimate the eigenvectors are advantageously performed on the segmented, masked endometrium, rather than on the 3D image volume, in order to reduce the computational load on the ultrasound system during dry work, thereby freeing up resources and improving the efficiency of the ultrasound system. A rotation vector is then estimated based on the set of eigenvectors, and the 3D image volume is automatically rotated based on the rotation vector so that the uterus of the displayed 3D image volume is aligned with the cross-sectional plane. By automatically aligning the uterus (and / or endometrium) with the cross-sectional plane before display using either of the two methods, the amount of time a radiologist spends reading an ultrasound image volume can be reduced. Without using either of the two methods, the radiologist would have to orient the 3D image volume before examining the structures included in the region of interest of the uterus and / or endometrium. Orienting the 3D image volume may be difficult, and the radiologist may have to adjust the orientation of the 3D image volume multiple times before achieving the desired orientation. By automatically acquiring and / or displaying a 3D image volume in a desired orientation, the time spent using the ultrasound system and associated resources can be reduced, improving the functionality of the ultrasound system as a whole and increasing the usability of the ultrasound system for other patients and / or used by other radiologists. In addition, the method described herein can achieve a more accurate orientation of the 3D image volume relative to the cross-sectional plane than is manually achieved by the radiologist, thereby enabling faster and more effective diagnosis of patients.

[0070] The technical effect of automatically acquiring and / or orienting the 3D ultrasound image volume so that the uterus and / or endometrium included in the 3D ultrasound image volume is displayed aligned with the cross-sectional plane of the 3D ultrasound image volume is that the time spent by the radiologist in reading the 3D ultrasound image volume can be reduced, the efficiency of the radiologist's workflow can be improved, and the overall use of the ultrasound imaging system can be reduced.

[0071] The present disclosure also provides support for a transvaginal ultrasound system (TVUS), the transvaginal ultrasound system comprising: an image processing system, the image processing system comprising a processor and a non-volatile memory, the non-volatile memory storing instructions, the instructions when executed causing the processor to: acquire a three-dimensional (3D) ultrasound image volume of a uterus of a subject scanned using the TVUS, wherein the endometrium of the uterus is aligned with a cross-sectional plane of the 3D ultrasound image volume, and display the 3D ultrasound image volume on a display device of the TVUS. In a first example of the system, additional instructions are stored in the non-volatile memory, the additional instructions when executed causing the processor to align the endometrium with the cross-sectional plane by: receiving a two-dimensional (2D) ultrasound image of the uterus generated from a 2D scan of the subject's uterus, extracting a median axis of the endometrium in the 2D ultrasound image, generating a uterine trace of the uterus according to the extracted median axis, and acquiring a 3D ultrasound image volume of the uterus with the endometrium aligned with the cross-sectional plane of the 3D ultrasound image volume according to the uterine trace. In a second example of the system that optionally includes the first example, additional instructions are stored in a non-transitory memory, which when executed cause the processor to perform segmentation on the endometrium using the segmentation model and extract the median axis from the segmented endometrium. In a third example of the system that optionally includes one or both of the first and second examples, additional instructions are stored in a non-transitory memory, which when executed cause the processor to generate a uterine trace according to the extracted median axis by fitting a polynomial function to the extracted median axis. In a fourth example of the system that optionally includes one or more or each of the first to third examples, additional instructions are stored in a non-transitory memory, which when executed cause the processor to perform segmentation on the uterus using the segmentation model and extrapolate the uterine trace from the uterine fundus end of the uterus to the cervical end of the uterus according to the polynomial function and the segmented uterus. In a fifth example of the system, which optionally includes one or more or each of the first to fourth examples, additional instructions are stored in the non-transitory memory, which, when executed, cause the processor to: perform segmentation of the endometrium in the 3D ultrasound image volume, and perform a 3D rotation on the 3D ultrasound image volume to align the long axis of the segmented endometrium with the cross-sectional plane of the 3D ultrasound image volume. In a sixth example of the system, which optionally includes one or more or each of the first to fifth examples, additional instructions are stored in the non-transitory memory, which, when executed, cause the processor to: estimate a set of eigenvectors of the segmented endometrium, estimate a rotation vector based on the estimated set of eigenvectors, and perform a 3D rotation on the 3D ultrasound image volume based on the rotation vector.In a seventh example of the system, which optionally includes one or more or each of the first to sixth examples, additional instructions are stored in a non-transitory memory, which, when executed, cause the processor to calculate a covariance matrix corresponding to pixel coordinates in a 3D space of an endometrial mask, and estimate eigenvectors according to the covariance matrix. In an eighth example of the system, which optionally includes one or more or each of the first to seventh examples, before performing a 3D rotation on the 3D ultrasound image volume, the endometrium is not aligned with a cross-sectional plane of the 3D ultrasound image volume. In a ninth example of the system, which optionally includes one or more or each of the first to eighth examples, the cross-sectional plane is a midsagittal plane of the 3D ultrasound image volume. In a tenth example of the system, which optionally includes one or more or each of the first to ninth examples, additional instructions are stored in a non-transitory memory, which, when executed, cause the processor to display a 3D ultrasound image volume on a display device of the TVUS, the 3D ultrasound image volume being oriented so that its cross-sectional plane is visible to a user of the TVUS.

[0072] The present disclosure also provides support for a method for a transvaginal ultrasound system (TVUS), the method comprising: receiving a two-dimensional (2D) ultrasound image generated from a scan of a subject's uterus performed using TVUS, extracting a central axis of the endometrium of the 2D ultrasound image, generating a uterine trace according to the extracted central axis, acquiring a three-dimensional (3D) image volume of the uterus according to the uterine trace, wherein the endometrium is aligned with a cross-sectional plane of the 3D image volume, and displaying the 3D image volume on a display device of the TVUS. In a first example of the method, generating the uterine trace according to the extracted central axis also includes performing segmentation on the endometrium by a segmentation model, and extracting the central axis from the segmented endometrium. In a second example of the method that optionally includes the first example, the method also includes: generating the uterine trace by fitting a polynomial function to the extracted central axis. In a third example of the method that optionally includes one or both of the first and second examples, generating the uterine trace according to the extracted central axis also includes performing segmentation on the uterus by a segmentation model, and extrapolating the uterine trace according to the polynomial function and the segmented uterus. In a fourth example of the method optionally including one or more or each of the first to third examples, extrapolating the uterine trajectory according to the polynomial function and the segmented uterus further includes extrapolating the median axis from the uterine fundus end of the uterus to the cervical end of the uterus. In a fifth example of the method optionally including one or more or each of the first to fourth examples, acquiring a 3D image volume of the uterus according to the uterine trajectory with the endometrium aligned with a cross-sectional plane of the 3D image volume further includes configuring the mechanical 3D probe to scan from approximately -45° to +45°, wherein the 0th angle corresponds to the plane in which the uterine trajectory is drawn.

[0073] The present disclosure also provides support for a method for a transvaginal ultrasound system (TVUS), the method comprising: receiving a three-dimensional (3D) image volume generated from a scan of a subject's uterus performed using TVUS, performing a rotation on the 3D image volume to align the endometrium of the uterus with the mid-sagittal plane of the TVUS, and displaying the 3D image volume with the endometrium aligned with the mid-sagittal plane on a display device of the TVUS. In a first example of the method, performing a rotation on the 3D image volume to align the endometrium with the mid-sagittal plane of the TVUS also includes: performing segmentation on the endometrium, estimating a set of eigenvectors of the segmented endometrium, determining a long axis of the endometrium based on the eigenvectors, estimating a rotation vector from the set of eigenvectors, and rotating the 3D image volume based on the rotation vector. In a second example of the method that optionally includes the first example, estimating the set of eigenvectors of the segmented endometrium also includes calculating a covariance matrix corresponding to pixel coordinates in a 3D space of an endometrial mask, and estimating eigenvalues ​​and eigenvectors of the covariance matrix.

[0074] When introducing the elements of various embodiments of the present disclosure, the articles "one", "a kind of" and "the" are intended to mean that there are one or more such elements. The terms "first", "second", etc. do not represent any order, amount or importance, but are used to distinguish one element from another element. The terms "include", "comprises", and "have" are intended to be inclusive, and mean that additional elements may also exist in addition to the listed elements. As used herein, the terms "connected to", "coupled to", etc., an object (e.g., material, element, structure, member, etc.) may be connected to or coupled to another object, regardless of whether the one object is directly connected or coupled to another object, or whether there are one or more intervening objects between the one object and another object. In addition, it should be understood that reference to "an embodiment" or "embodiment" of the present disclosure is not intended to be interpreted as excluding the existence of additional embodiments that are also combined with the cited features.

[0075] In addition to any previously indicated modifications, those skilled in the art may devise numerous other variations and alternative arrangements without departing from the spirit and scope of the present specification, and the appended claims are intended to cover such modifications and arrangements. Therefore, although the information has been described in detail and in detail as described above in conjunction with what are currently considered to be the most practical and preferred aspects, it will be apparent to those of ordinary skill in the art that many modifications, including but not limited to form, function, mode of operation, and use, may be made without departing from the principles and concepts set forth herein. Likewise, as used herein, the examples and embodiments are meant to be illustrative in all respects and should not be construed as limiting in any way.

Claims

1. A transvaginal ultrasound system (TVUS) (100), comprising: An image processing system (202) comprising a processor (204) and a non-transitory memory (206), the non-transitory memory storing instructions that, when executed, cause the processor (204) to: acquiring a three-dimensional (3D) ultrasound image volume of a uterus (801, 1006) of a subject scanned using the TVUS (100), wherein an endometrium (1102, 700, 1002, 902, 802) of the uterus (801, 1006) is aligned with a cross-sectional plane of the 3D ultrasound image volume; and The 3D ultrasound image volume is displayed on a display device (118, 234) of the TVUS (100).

2. The TVUS (100) according to claim 1, wherein further instructions are stored in the non-transitory memory (206), which when executed cause the processor (204) to align the endometrium (1102, 700, 1002, 902, 802) with the cross-sectional plane by: receiving a two-dimensional (2D) ultrasound image of the uterus (801, 1006) generated from a 2D scan of the uterus (801, 1006) of the subject; Extracting a central axis (1004) of the endometrium (602) in the 2D ultrasound image (600); generating a uterine trace (804, 702) of the uterus (801, 1006) according to the extracted central axis (1004); and The 3D ultrasound image volume of the uterus (801, 1006) is acquired according to the uterine trajectory (804, 702), wherein the endometrium (1102, 700, 1002, 902, 802) is aligned with the cross-sectional plane of the 3D ultrasound image volume.

3. The TVUS (100) according to claim 2, wherein additional instructions are stored in the non-volatile memory (206), and when the additional instructions are executed, the processor (204) causes the processor (204) to perform segmentation on the endometrium (1102, 700, 1002, 902, 802) using a segmentation model and extract the central axis (1004) from the segmented endometrium (1102, 700, 1002, 902, 802).

4. The TVUS (100) according to claim 3, wherein further instructions are stored in the non-volatile memory (206), which when executed cause the processor (204) to generate the uterine trace (804, 702) according to the extracted medial axis (1004) by fitting a polynomial function to the extracted medial axis (1004).

5. The TVUS (100) according to claim 4, wherein additional instructions are stored in the non-volatile memory (206), which, when executed, cause the processor (204) to perform segmentation on the uterus (801, 1006) using the segmentation model, and to extrapolate the uterine trajectory (804, 702) from the uterine fundus end of the uterus (801, 1006) to the cervical end of the uterus (801, 1006) based on the polynomial function and the segmented uterus (801, 1006, 502, 602).

6. The TVUS (100) of claim 1, wherein further instructions are stored in the non-transitory memory (206), the further instructions when executed causing the processor (204) to: performing segmentation on the endometrium (1102, 700, 1002, 902, 802) in the 3D ultrasound image volume; as well as A 3D rotation is performed on the 3D ultrasound image volume to align a major axis (1104) of the segmented endometrium (1102, 700, 1002, 902, 802) with the cross-sectional plane of the 3D ultrasound image volume.

7. The TVUS (100) of claim 6, wherein further instructions are stored in the non-transitory memory (206), the further instructions when executed causing the processor (204) to: estimating a set of eigenvectors of the segmented endometrium (1102, 700, 1002, 902, 802); estimating a rotation vector based on the estimated set of eigenvectors; and The 3D rotation of the 3D ultrasound image volume is performed according to the rotation vector.

8. The TVUS (100) of claim 7, wherein further instructions are stored in the non-transitory memory (206), the further instructions when executed causing the processor (204) to: generating a mask of the endometrium (1102, 700, 1002, 902, 802) based on the segmented endometrium (1102, 700, 1002, 902, 802), and applying the mask to the 3D ultrasound image volume to generate a masked version of the 3D ultrasound image volume, the masked version including image data of the 3D ultrasound image volume inside the segmented endometrium (1102, 700, 1002, 902, 802) and excluding image data outside the segmented endometrium (1102, 700, 1002, 902, 802); A covariance matrix of pixel coordinates in 3D space corresponding to the segmented endometrium (1102, 700, 1002, 902, 802) in the masked version of the 3D ultrasound image volume is calculated, and the eigenvectors are estimated based on the covariance matrix.

9. The TVUS (100) according to claim 6, wherein before performing the 3D rotation on the 3D ultrasound image volume, the endometrium (1102, 700, 1002, 902, 802) is not aligned with the cross-sectional plane of the 3D ultrasound image volume.

10. The TVUS (100) according to claim 6, wherein the cross-sectional plane is a mid-sagittal plane of the 3D ultrasound image volume.

11. The TVUS (100) according to claim 1, wherein further instructions are stored in the non-volatile memory (206), which when executed cause the processor (204) to display the 3D ultrasound image volume on the display device (118, 234) of the TVUS, the 3D ultrasound image volume being oriented so that its cross-sectional plane is visible to a user of the TVUS.

12. A method (300) for use with a transvaginal ultrasound system (TVUS), the method comprising: receiving a two-dimensional (2D) ultrasound image (302) generated from a scan of a subject's uterus performed using the TVUS; Extracting the central axis of the endometrium of the 2D ultrasound image (308); generating a uterine trace according to the extracted central axis (310); acquiring a three-dimensional (3D) image volume (314) of the uterus according to the uterine trajectory, wherein the endometrium is aligned with a cross-sectional plane of the 3D image volume; as well as The 3D image volume is displayed on a display device of the TVUS (316).

13. The method of claim 12, wherein generating the uterine trace according to the extracted median axis further comprises performing segmentation (306) on the endometrium by a segmentation model, and extracting the median axis (308) from the segmented endometrium.

14. The method of claim 13, further comprising generating the uterine trace (310) by fitting a polynomial function to the extracted medial axis.

15. The method of claim 14, wherein generating the uterine trace according to the extracted medial axis further comprises performing segmentation on the uterus by the segmentation model, and extrapolating the uterine trace according to the polynomial function and the segmented uterus.

Images