Ultrasound image feature segmentation
Ultrasound image data is trained through self-supervised learning framework and convolutional neural network, and the poor contact areas of the liver and probe are automatically segmented, solving the problem of time-consuming and costly data labeling in NAFLD diagnosis, and achieving efficient and accurate measurement of liver hardness diagnosis.
Patent Information
- Application Number
- CN202410690051.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-06-05
- Filing Date
- 2024-05-30
- Publication Date
- 2024-12-06
AI Technical Summary
The prior art is difficult to effectively and economically diagnose non-alcoholic lipohepatitis (NAFLD), especially due to the invasiveness and cost of liver biopsy, and the time-consuming and error of data labeling for liver segmentation and hardness measurements in ultrasound imaging. question.
The self-supervised learning framework combined with convolutional neural network is adopted to train and segment ultrasonic image data through SimCLR and ENet frameworks, and automatically locate the poor contact areas of the liver and probe, reduce the dependence on annotated data, and improve model training efficiency and accuracy.
Automatic segmentation and shear wave elastic imaging of poor contact areas of the liver and probe are realized, reducing diagnostic costs and time, and improving the accuracy and efficiency of NAFLD diagnosis.
Smart Images

Figure CN119091131A_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] [not applicable] Background Art
[0003] In general, the present application relates to ultrasound imaging and shear wave elastography. Non-alcoholic fatty liver disease (NAFLD) is a cause of chronic liver disease, which can be characterized by the accumulation of excess fat in the liver or caused by the accumulation of excess fat in the liver, leading to damage and inflammation. Currently, the incidence of NAFLD in the United States is on the rise, resulting in considerable medical costs. Liver biopsy can be used to diagnose NAFLD, but it is an invasive and relatively expensive test and may be affected by sampling errors and interpretation fluctuations. Due to these limitations, non-invasive alternatives have been developed, including ultrasound. As NAFLD progresses, liver stiffness increases, making it a biomarker. Shear wave elastography (SWE) is an ultrasound method that can measure or estimate the stiffness of liver tissue. Summary of the invention
[0004] According to an embodiment, a method for training a model to segment a test image includes: receiving a first plurality of training images at a self-supervised learning framework, wherein the first plurality of training images include ultrasound data corresponding to a patient's liver, wherein the test image includes ultrasound image data; processing the first plurality of training images using a learning algorithm of the self-supervised learning framework, and responsively adjusting the trained model; and receiving the trained model and a second plurality of training images at the supervised learning framework, and responsively adjusting the trained model, wherein the second plurality of training images include ultrasound data corresponding to the patient's liver and annotations of the liver. The self-supervised learning framework may include a contrastive learning framework. The self-supervised learning framework may use a convolutional neural network as an encoder. The self-supervised learning framework may employ a projection head. The self-supervised learning framework may include a SimCLR framework. The supervised learning framework may include an ENet framework. The self-supervised learning framework may include a SimCLR framework, and the supervised learning framework includes an ENet framework. The supervised learning framework may include an encoder and a decoder, the encoder including a plurality of stages and the decoder including a plurality of stages, wherein each stage includes a plurality of bottleneck modules, and the plurality of bottleneck modules are configured to manage dimensionality. The supervised learning framework may include a max pooling layer, wherein the supervised framework further includes a decoder including a max unpooling layer and a spatial convolution algorithm. The supervised learning framework may be configured to avoid bias terms in the projection.
[0005] According to an embodiment, a method for segmenting a structure in ultrasound image data includes: obtaining ultrasound image data of a patient including a liver using ultrasound energy; receiving the ultrasound image data at a processor; executing inference instructions by the processor to segment the liver in real time in the ultrasound image data to form a segmented liver; and presenting the ultrasound image data on a display. The method may also include: determining a region of interest within the segmented liver; and performing shear wave elastic imaging on data obtained from the region of interest. The region of interest may be automatically determined. The method may also include executing inference instructions by the processor to segment a probe poor contact region in the ultrasound image data. The method may also include presenting information corresponding to the probe poor contact region on a display together with the ultrasound image data.
[0006] According to an embodiment, a system includes: an ultrasound probe and a receiver, the receiver being configured to obtain ultrasound image data of a patient including a liver; a processor, the processor being configured to receive the ultrasound image data and execute inference instructions to segment the liver in the ultrasound image data in real time to form a segmented liver; and a display, the display being configured to present the ultrasound image data and information associated with the segmented liver. The processor may also be configured to determine a region of interest within the segmented liver, and to cause a shear wave elastic imaging process to be performed to obtain shear wave elastic imaging data from the region of interest. The region of interest may be determined automatically. The processor may also be configured to execute inference instructions to segment a probe poor contact region in the ultrasound image data. The display may also be configured to present information corresponding to the probe poor contact region together with the ultrasound image data. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] Figure 1 An ultrasound system according to an embodiment is shown.
[0008] Figure 2 An ultrasound system for performing shear wave elastography is shown in accordance with an embodiment.
[0009] Figure 3 A process for training a model and using the model to segment anatomical image data according to an embodiment is shown.
[0010] The foregoing summary of the invention and the following detailed description of certain technologies of the present application will be better understood when read in conjunction with the accompanying drawings. For illustrative purposes, certain technologies are shown in the accompanying drawings. However, it should be understood that the claims are not limited to the arrangements and tools shown in the accompanying drawings. DETAILED DESCRIPTION
[0011] Disclosed herein is an embodiment of using machine learning to train a model and using the model to segment ultrasound image data of a patient. One example of segmentation is for the liver capsule and / or liver (hereinafter referred to as "liver" unless otherwise specified). Another example of segmentation is for image data obtained based on poor probe contact. The segmented organ (e.g., liver) can be used to locate one or more regions of interest (hereinafter referred to as "region of interest" or "ROI" unless otherwise specified). The ROI can be the location from which shear wave elastography data is obtained. Shear wave elastography measurements can indicate the hardness of the patient's liver and can be used to diagnose NAFLD. The region of interest can be automatically located, or the location can be facilitated by an automatic process. An embodiment of such a process is disclosed in U.S. Serial No. 18 / 205,809, filed on June 5, 2023, the entire text of which is incorporated herein by reference.
[0012] Disclosed herein are embodiments of an aggregated machine learning process that reduces the need for relatively large annotated datasets for training. Instead, relatively small annotated datasets can be used. For example, to train a model to segment the liver in ultrasound image data, annotated images can be provided to a learning framework, where the annotations include the shape of the liver. Creating these annotations may require manual steps, such as a person manually or electronically drawing the liver in the corresponding image. Such an annotation process can be relatively time consuming and costly. Therefore, if fewer annotations are required to train a model, this can reduce the time and cost of model training.
[0013] An embodiment of a two-stage learning model is disclosed that reduces the size of annotated training data, which is often used to improve the segmentation accuracy of deep learning models. The aggregation includes a first framework and a second framework. The first framework is a self-supervised contrastive learning framework in which the model is trained using unannotated ultrasound image data. An example of such a framework is SimCLR. The second framework is a supervised framework in which the model is further trained using annotated ultrasound image data. An example of such a framework is ENet. The second framework fine-tunes the model initially trained by the first framework. The resulting model can be implemented on an ultrasound instrument together with the second framework or a variant thereof. Ultrasound image data can be processed using inference instructions in the second framework to segment structures such as the liver. Multiple regions such as the liver and probe poor contact areas can be segmented simultaneously. The segmented segments can be used to determine a region of interest, which can indicate a location in the liver from which the propagation of shear waves is tracked as part of a shear wave elastography process. An implementation scheme of such an aggregate learning model is described in the paper “Liver Segmentation in Ultrasound Images Using Self-Supervised Learning with Physics-inspired Augmentation and Global-Local Refinement” presented by Ali et al. at the 36th Canadian Conference on Artificial Intelligence on June 5, 2023, the entire text of which is incorporated herein by reference.
[0014] When reading in conjunction with the accompanying drawings, the following specific embodiments of the aforementioned invention content and certain embodiments will be better understood. In terms of the scope of the figures of the functional blocks of each embodiment shown in the accompanying drawings, these functional blocks do not necessarily represent the division between the hardware circuits. Therefore, for example, one or more functional blocks (e.g., processors or memory) can be implemented in a single piece of hardware (e.g., a general signal processor or random access memory block, a hard disk, etc.) or multiple hardware components. Similarly, a program can be an independent program, can be incorporated into an operating system as a subroutine, can be a function in an installed software package, etc. It should be understood that each embodiment is not necessarily limited to the arrangement and tools shown in the accompanying drawings. It should also be understood that the embodiments can be combined as should be understood, or other embodiments can be utilized, and structural, logical and electrical changes can be made without departing from the scope of each embodiment. Therefore, the following detailed description should not be considered as a restrictive meaning, and the scope of the present invention is limited by the attached claims and their equivalents.
[0015] As used herein, elements or steps listed in the singular and beginning with the word "a" or "an" should be understood not to exclude a plurality of said elements or steps, unless such exclusion is explicitly stated. Furthermore, reference to "one embodiment" or "an embodiment" is not intended to be interpreted as excluding the existence of other embodiments that also include the described features. Furthermore, unless explicitly stated otherwise, an embodiment that "comprises" or "has" an element or elements having a particular property may include additional elements that do not have that property.
[0016] In addition, as used herein, the term "image" or "image data" refers in a broad sense to both a visible image and data representing a visible image. Some embodiments generate (or are configured to generate) at least one visible image. In addition, as used herein, the term "image" or "image data" is used to refer to an ultrasound mode, such as B-mode, CF mode and / or a sub-mode of B-mode and / or CF, such as shear wave elastic imaging (SWEI), TVI, Angio, B-flow, BMI, BMI_Angio, and in some cases also includes MM, CM, PW, TVD, CW, where "image" and / or "plane" includes a single beam or multiple beams.
[0017] Furthermore, as used herein, the term processor or processing unit refers to any type of processing unit that can perform the computations of the various embodiments, such as single-core or multi-core: CPU, graphics board, DSP, FPGA, ASIC, or combinations thereof.
[0018] Various embodiments of generating or forming an image described herein may include a process for forming an image that includes beamforming in some embodiments and does not include beamforming in other embodiments. For example, an image may be formed without beamforming, such as by multiplying a matrix of demodulated data by a matrix of coefficients such that the product is an image and wherein the process does not form any "beams." Additionally, the formation of an image may be performed using a combination of channels that may originate from more than one transmit event (e.g., synthetic aperture techniques).
[0019] In various embodiments, ultrasound processing to form images, including ultrasound beamforming, such as receive beamforming, is performed, for example, in software, firmware, hardware, or a combination thereof. One specific implementation of an ultrasound system having a software beamformer architecture formed according to various embodiments disclosed herein.
[0020] Furthermore, as used herein, the term processor or processing unit refers to any type of processing unit, whether single-core or multi-core: CPU, accelerated processing unit (APU), graphics board, DSP, FPGA, ASIC, or a combination thereof, that can perform the required computations required by various implementations.
[0021] Other embodiments may provide a computer-readable device and / or a non-transitory computer-readable medium, and / or a machine-readable device and / or a non-transitory machine-readable medium, on which the above devices and / or media store machine codes executable by the machine and / or computer and / or a computer program having at least one code segment, so that the machine and / or computer performs the steps for segmenting the liver in ultrasound image data as described herein.
[0022] Therefore, the present disclosure may be implemented in hardware, software, or a combination of hardware and software. The present disclosure may be implemented in a centralized manner in at least one computer system, or in a distributed manner, where different elements are distributed across several interconnected computer systems. Any type of computer system or other device suitable for executing the methods described herein is suitable.
[0023] The various embodiments may also be embedded in a computer program product, which comprises all the features enabling the implementation of the methods described herein and which, when loaded into a computer system, is capable of carrying out these methods. A computer program herein refers to any expression of a set of instructions in any language, code or notation, which is intended to cause a system with information processing capabilities to perform specific functions directly or after either or both of the following: a) conversion into another language, code or notation; b) reproduction in a different material form.
[0024] Although the present disclosure has been described with reference to certain embodiments, it will be appreciated by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the scope of the present disclosure. In addition, many modifications may be made to adapt specific circumstances or materials to the teachings of the present disclosure without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the specific embodiments disclosed, but the present disclosure will include all embodiments falling within the scope of the appended claims.
[0025] Figure 1 is a block diagram of an exemplary ultrasound system 100 operable to segment ultrasound image data and perform further processing on the segmented data, according to various embodiments. Figure 1 , an ultrasound system 100 and a training system 190 are shown. The ultrasound system 100 includes a transmitter 102, an ultrasound probe 104, a transmit beamformer 110, a receiver 118, a receive beamformer 120, an A / D converter 122, an RF processor 124, an RF / IQ buffer 126, a user input device 130, a signal processor 132, an image buffer 136, a display system 134, and an archive 138.
[0026] The transmitter 102 may include suitable logic, circuitry, interfaces and / or code operable to drive the ultrasound probe 104. The ultrasound probe 104 may include a two-dimensional (2D) array of piezoelectric elements. The ultrasound probe 104 may include a set of transmit transducer elements 106 and a set of receive transducer elements 108 that are generally constructed of the same elements. In certain embodiments, the ultrasound probe 104 may be operable to acquire ultrasound image data covering at least a majority of an anatomical structure such as the liver or any suitable anatomical structure, or a feature of interest such as an area where the ultrasound probe 104 has poor contact with the patient.
[0027] The transmit beamformer 110 may include suitable logic, circuitry, interfaces, and / or code operable to control the transmitter 102 to drive the set of transmit transducer elements 106 via the transmit sub-aperture beamformer 114 to transmit ultrasound transmit signals into a region of interest (e.g., a person, an animal, an underground cavity, a physical structure, etc.). The transmitted ultrasound signals may be backscattered from structures in the object of interest (such as blood cells or tissue) to generate echoes. The echoes are received by the receive transducer elements 108.
[0028] The set of receive transducer elements 108 in the ultrasound probe 104 may be operable to convert received echoes into analog signals, sub-aperture beamformed by the receive sub-aperture beamformer 116, and then transmitted to the receiver 118. The receiver 118 may include suitable logic, circuitry, interfaces, and / or code that may be operable to receive the signals from the receive sub-aperture beamformer 116. The analog signals may be transmitted to one or more of the plurality of A / D converters 122.
[0029] The plurality of A / D converters 122 may include suitable logic components, circuits, interfaces, and / or codes that are operable to convert analog signals from the receiver 118 into corresponding digital signals. The plurality of A / D converters 122 are disposed between the receiver 118 and the RF processor 124. Nevertheless, the present disclosure is not limited in this regard. Therefore, in some embodiments, the plurality of A / D converters 122 may be integrated within the receiver 118.
[0030] The RF processor 124 may include suitable logic components, circuits, interfaces and / or codes that are operable to demodulate the digital signals output by the plurality of A / D converters 122. According to an embodiment, the RF processor 124 may include a complex demodulator (not shown) that is operable to demodulate the digital signals to form I / Q data pairs representing corresponding echo signals. The RF or I / Q signal data may then be transferred to the RF / IQ buffer 126. The RF / IQ buffer 126 may include suitable logic components, circuits, interfaces and / or codes that are operable to provide temporary storage of the RF or I / Q signal data generated by the RF processor 124.
[0031] The receive beamformer 120 may comprise suitable logic, circuitry, interfaces, and / or code operable to perform digital beamforming processing, for example, to sum delayed channel signals received from the RF processor 124 via the RF / IQ buffer 126 and output a beam summed signal. The resulting processed information may be a beam summed signal output from the receive beamformer 120 and communicated to the signal processor 132. According to some embodiments, the receiver 118, the plurality of A / D converters 122, the RF processor 124, and the beamformer 120 may be integrated into a single beamformer, which may be a digital beamformer. In various embodiments, the ultrasound system 100 comprises a plurality of receive beamformers 120.
[0032] The user input device 130 may be used to enter patient data, scan parameters, settings, select a protocol and / or template, select an exam type, select a desired ultrasound image view, select an active sample identification algorithm, reposition an automatically placed region of interest, etc. In an exemplary embodiment, the user input device 130 is operable to configure, manage, and / or control the operation of one or more components and / or modules in the ultrasound system 100. In this regard, the user input device 130 is operable to configure, manage, and / or control the operation of the transmitter 102, ultrasound probe 104, transmit beamformer 110, receiver 118, receive beamformer 120, RF processor 124, RF / IQ buffer 126, user input device 130, signal processor 132, image buffer 136, display system 134, and / or archive 138. The user input device 130 may include a button, a rotary encoder, a touch screen, a touch pad, a trackball, motion tracking, voice recognition, a mouse device, a keyboard, a camera, and / or any other device capable of receiving user instructions. In some embodiments, for example, one or more of the user input devices 130 may be integrated into other components such as the display system 134. For example, the user input device 130 may include a touch screen display.
[0033] The signal processor 132 may include suitable logic components, circuits, interfaces and / or codes that are operable to process ultrasound scan data (i.e., summed IQ signals) to generate ultrasound images for presentation on a display system 134. The signal processor 132 is operable to perform one or more processing operations based on a plurality of selectable ultrasound modalities on the acquired ultrasound scan data. In an exemplary embodiment, the signal processor 132 may be used to perform display processing and / or control processing, etc. The acquired ultrasound image data may be processed in real time during a scanning session as echo signals are received. Additionally or alternatively, the ultrasound scan data may be temporarily stored in the RF / IQ buffer 126 during a scanning session and processed in a less real-time manner in an online operation or an offline operation. In various embodiments, the processed image data may be presented at the display system 134 and / or may be stored at an archive 138. The archive 138 may be a local archive, a picture archiving and communication system (PACS), an enterprise archive (EA), a vendor-independent archive (VNA), or any suitable device for storing images and related information.
[0034] The signal processor 132 may be one or more central processing units, microprocessors, microcontrollers, etc. For example, the signal processor 132 may be an integrated component, or may be distributed in various locations. In an exemplary embodiment, the signal processor 132 may include an image analysis processor 140, a segmentation processor 150, a sample identification processor 160, and a region of interest (ROI) positioning processor 170. The signal processor 132 may be capable of receiving input information from the user input device 130 and / or the archive 138, receiving image data, generating output that can be displayed by the display system 134, and manipulating the output in response to the input information from the user input device 130, etc. The signal processor 132 including the image analysis processor 140, the segmentation processor 150, the sample identification processor 160, and the region of interest (ROI) positioning processor 170 can, for example, execute any of the methods and / or instruction sets discussed herein according to various embodiments.
[0035] The ultrasound system 100 is operable to continuously acquire ultrasound scan data at a frame rate suitable for the imaging situation under consideration. Typical frame rates are in the range of 20 to 120, but may be lower or higher. The acquired ultrasound scan data may be displayed on a display system 134 at a display rate that is the same as the frame rate, or at a display rate that is slower or faster than the frame rate. An image buffer 136 is included to store processed frames of acquired ultrasound scan data that are not scheduled for immediate display. Preferably, the image buffer 136 has sufficient capacity to store ultrasound scan data frames equivalent to at least several minutes. The frames of ultrasound scan data are stored in a manner that is easy to retrieve from them according to their acquisition order or time. The image buffer 136 may be embodied as any known data storage medium.
[0036] The signal processor 132 may include an image analysis processor 140 that includes suitable logic, circuitry, interfaces, and / or code that is operable to analyze acquired ultrasound image data to determine whether a desired ultrasound image view has been obtained. For example, the image analysis processor 140 may analyze ultrasound image data acquired by the ultrasound probe 104 to determine whether a desired view, such as any suitable ultrasound image view of the liver, has been obtained. Once the desired image view is obtained, the image analysis processor 140 may instruct the signal processor 132 to freeze the view presented at the display system 134. The view may be stored in the archive 138 and / or at any suitable data storage medium. The image analysis processor 140 may include, for example, an artificial intelligence image analysis algorithm, one or more deep neural networks (e.g., a convolutional neural network, such as ENet), and / or may utilize any suitable image analysis technology or machine learning processing functionality configured to determine whether a desired view has been obtained. Additionally and / or alternatively, artificial intelligence image analysis techniques or machine learning processing functionality configured to provide image analysis techniques may be provided by different processors or distributed across multiple processors at the ultrasound system 100 and / or distributed across a remote processor communicatively coupled to the ultrasound system 100. In various embodiments, the image analysis processor 140 may include suitable logic, circuitry, interfaces, and / or code that is operable to provide quality metrics associated with an acquired view. For example, the image analysis processor 140 may analyze an acquired ultrasound image view (as a whole), a region of an acquired ultrasound image view, an acquired ultrasound image view segmented by a segmentation processor 150, and the like to provide quality metrics associated with the acquired view. The image analysis processor 140 may be configured to cause the display system 134 to compare the quality metrics with an acquired ultrasound image view. Figure 1For example, the quality metric may be a score (e.g., 1, 2, 3, 4, 5), a grade (e.g., A, B, C, D, F), a rating (e.g., excellent, good, fair, poor), a color coding (e.g., green, yellow, red), etc. for the acquired ultrasound image view (as a whole) and / or each region of the acquired ultrasound image view. The quality metric may help the user determine whether to continue acquiring the view or to obtain additional ultrasound image data. The image analysis processor 140 may store the quality metric in an archive and / or any suitable data storage medium.
[0037] The signal processor 132 may include a segmentation processor 150 comprising suitable logic, circuitry, interfaces, and / or code operable to segment flow image frames and B-mode frames. The segmentation processor 150 may be used to identify structures or features (e.g., areas of liver or poor probe contact) in the acquired ultrasound image data. In this regard, the segmentation processor 150 may include, for example, an artificial intelligence image analysis algorithm, one or more deep neural networks (e.g., convolutional neural networks such as ENet), and / or may utilize any suitable form of artificial intelligence image analysis technology or machine learning processing functionality configured to provide automated segmentation functionality. Examples of such image analysis techniques are disclosed at Figure 3 For example, the segmentation processor may implement a framework 370 associated with the trained model 360.
[0038] Additionally and / or alternatively, an artificial intelligence image analysis technique or machine learning processing function configured to provide automated segmentation may be provided by a different processor or distributed on multiple processors at the ultrasound system 100 and / or distributed on a remote processor communicatively coupled to the ultrasound system 100. For example, the image segmentation function may be provided as a deep neural network, which may be composed of, for example, an input layer, an output layer, and one or more hidden layers between the input layer and the output layer. Each layer may be composed of multiple processing nodes, which may be referred to as neurons. For example, the image segmentation function may include an input layer having neurons for each sample or a group of samples from ultrasound image views of the liver and kidneys obtained. The output layer may have neurons corresponding to a variety of predefined anatomical structures, such as the liver, the renal cortex, or any suitable anatomical structure. Each neuron in each layer may perform a processing function and pass the processed ultrasound image information to one of the multiple neurons in the downstream layer for further processing. For example, neurons in the first layer may learn to identify structural edges in the obtained ultrasound image. Neurons in the second layer may learn to recognize shapes based on detected edges from the first layer. Neurons in the third layer may learn the position of the recognized shapes relative to landmarks in the obtained ultrasound image. The processing performed by the deep neural network can identify anatomical structures or other features and the locations of the structures or features in the acquired ultrasound images with a high probability.
[0039] In an exemplary embodiment, the segmentation processor 150 may be configured to store the image segmentation information in the archive 138 and / or any suitable storage medium. The segmentation processor 150 may be configured to cause the display system 134 to present the image segmentation information together with the acquired ultrasound image. The image segmentation information may be provided to the image analysis processor 140 for providing a quality metric associated with the acquired ultrasound image view, as discussed above.
[0040] Still reference Figure 1, the training system 190 may include a training engine 192 and a training database 194. The training engine 192 may include suitable logic components, circuits, interfaces and / or codes that are operable to train neurons of a deep neural network (e.g., an artificial intelligence model) inferred (i.e., deployed) by the image analysis processor 140, the segmentation processor 150, the sample identification processor 160 and / or the ROI positioning processor 170. For example, the artificial intelligence model inferred by the image analysis processor 140 may be trained to automatically identify ultrasound image views (e.g., liver capsule or poor probe contact area). The artificial intelligence model inferred by the segmentation processor 150 may be trained to automatically segment the obtained ultrasound image views to identify anatomical structures or other ultrasound image data (e.g., liver or poor probe contact area). For example, the training engine 192 may use the database to train the deep neural network deployed by the image analysis processor 140 and / or the segmentation processor 150. The ultrasound image may include an ultrasound image of a specific anatomical feature or related structure such as the liver or (one or more) areas where the probe has poor contact with the patient's skin, or any suitable ultrasound image and feature.
[0041] In certain embodiments, the training system 190 and the training database 194 may include a database such as disclosed in Figure 3 and those parts in the corresponding text. For example, the training engine 192 may include an aggregation framework that includes a self-supervised learning framework 310 and a supervised learning framework 340. The training database 194 may include data sets, such as a data set 320 for training a model using the self-supervised learning framework 310, and a data set 350 for training a model using the supervised learning framework 340.
[0042] In certain embodiments, the training engine 192 and / or the training database 194 may be a remote system communicatively coupled to the ultrasound system 100 via a wired or wireless connection, such as Figure 1 Additionally and / or alternatively, components or all of the training system 190 may be integrated with the ultrasound system 100 in various forms. In certain embodiments, the training system 190 may be separate from the ultrasound system 100, and a model may be trained that is then included in the ultrasound system 100. For example, the model may be trained to segment the liver capsule or the liver and / or poor probe contact areas, and the trained model may then be included in the ultrasound system 100, for example as part of the signal processor 132 or associated memory.
[0043] Figure 2A block diagram of an exemplary ultrasound system 200 is shown that is operable to perform a shear wave elastography procedure, for example, to obtain information about liver tissue in a region of interest. The ultrasound system 200 may be similar to and / or share components with the ultrasound system 100. For example, the ultrasound system 200 and the ultrasound system 100 may be integrated with each other. For example, there may be only one system that performs conventional ultrasound imaging and shear wave elastography. Figure 2 An ultrasound system 200 and a vibrating device or transducer 20 are shown. The transducer 20 may be located external to the ultrasound system 200. The transducer 20 may be configured to provide an external "push" force 22 to generate shear waves 5 in tissue 1 of a patient. As will be discussed further, the tissue 1 may be liver tissue or tissue or structure proximate to liver tissue. Additionally and / or alternatively, the ultrasound system 200 itself may provide a push force, such as a relatively high intensity ultrasound push pulse 205, for generating shear waves 5 in the tissue 1. The ultrasound system 200 includes a transmitter 202 (e.g., similar or identical to the transmitter 102 ), an ultrasound probe (or “probe”) 204 (e.g., similar or identical to the probe 104 ), a transmit beamformer 210 (e.g., similar or identical to the transmit beamformer 110 ), a receiver 218 (e.g., similar or identical to the receiver 118 ), a receive beamformer 220 (e.g., similar or identical to the receive beamformer 120 ), an RF processor 224 (e.g., similar or identical to the RF processor 124 ), an RF / IQ buffer 226 (e.g., similar or identical to the RF / IQ buffer 126 ), a user input module 230 (e.g., similar or identical to the user input module 130 ), a signal processor 132 (e.g., similar or identical to the signal processor 132 ), an image buffer 236 (e.g., similar or identical to the image buffer 136 ), and a display system 234 (e.g., similar or identical to the display system 134 ).
[0044] The transmitter 202 may include suitable logic, circuitry, interfaces and / or code operable to drive a transducer ultrasound probe 204. The ultrasound probe 204 may include a one-dimensional (1D, 1,25D, 1,5D or 1,75D) array or a two-dimensional (2D) array of transducers such as piezoelectric elements. The ultrasound probe 204 may include a transmitting transducer element 106 and a receiving transducer element 208 that may be fully coextensive, partially coextensive or separated.
[0045] The transmit beamformer 210 may comprise suitable logic, circuitry, interfaces and / or code operable to control the transmitter 202 to drive the transmit transducer elements 206 via the transmit sub-aperture beamformer 214 to transmit relatively high intensity ultrasound push pulses 205 into the interference points of the tissue 1 and to transmit ultrasound transmit signals 207 into the region of interest of the tissue 1. As used herein, the term “high intensity ultrasound push pulse” refers to an ultrasound transmit signal 207 having a relatively high intensity and a relatively high intensity. 2 Up to 700mW / cm 2 The derated spatial peak time average intensity (ISPTA.3) between the interference points. The transmitted high-intensity ultrasonic push pulse 205 can displace the tissue 1 to produce a shear wave 5 that propagates laterally from the interference point. The transmitted ultrasonic signal 207 can be reflected from structures such as the tissue 1 deformed by the shear wave 5 to produce an echo 209. The echo 209 is received by the receiving transducer element 208. The group of receiving transducer elements 208 in the ultrasonic probe 204 is operable to convert the received echo 209 into an analog signal, perform sub-aperture beamforming through the receiving sub-aperture beamformer 216, and then transmit it to the receiver 218.
[0046] The receiver 218 may include suitable logic, circuitry, interfaces, and / or code operable to receive and demodulate signals from the receive subaperture beamformer 216. The demodulated analog signals may be communicated to one or more of a plurality of A / D converters 222 (e.g., similar to the A / D converter 122). The plurality of A / D converters 222 may include suitable logic, circuitry, interfaces, and / or code operable to convert the demodulated analog signals from the receiver 218 into corresponding digital signals. The plurality of A / D converters 222 are disposed between the receiver 218 and the receive beamformer 220. However, the invention is not limited in this respect. Thus, in some embodiments, the plurality of A / D converters 222 may be integrated within the receiver 218.
[0047] The receive beamformer 220 may comprise suitable logic, circuitry, interfaces, and / or code operable to perform digital beamforming processing on signals received from the plurality of A / D converters 222. The resulting processed information may be converted back into corresponding RF signals. The corresponding output RF signals output from the receive beamformer 220 may be communicated to the RF processor 224. According to some embodiments, the receiver 218, the plurality of A / D converters 222, and the beamformer 220 may be integrated into a single beamformer, which may be a digital beamformer.
[0048] The RF processor 224 may include suitable logic components, circuits, interfaces and / or codes that can operate to demodulate the RF signal. According to an embodiment, the RF processor 224 may include a complex demodulator (not shown) that can operate to demodulate the RF signal to form an I / Q data pair representing the corresponding echo signal. The RF or I / Q signal data can then be transmitted to the RF / IQ buffer 226.
[0049] The RF / IQ buffer 226 may comprise suitable logic, circuitry, interfaces and / or code that may be operable to provide temporary storage of RF or I / Q signal data generated by the RF processor 224 .
[0050] The user input module 230 may be used to initiate shear wave elastography, change scan modes, enter patient data, surgical instrument data, scan parameters, set up, configure parameters, etc. In an exemplary embodiment, the user input module 230 is operable to configure, manage, and / or control the operation of one or more components and / or modules in the ultrasound system 200. In this regard, the user input module 230 is operable to configure, manage, and / or control the operation of the transmitter 202, the ultrasound probe 204, the transmit beamformer 210, the receiver 218, the receive beamformer 220, the RF processor 224, the RF / IQ buffer 226, the user input module 230, the signal processor 232, the image buffer 236, and / or the display system 234. The user input module 230 may be located at various locations on and / or around the ultrasound system 200, such as on the probe 204, at a control panel, and / or at any suitable location.
[0051] The signal processor 232 may include suitable logic components, circuits, interfaces and / or codes that are operable to process ultrasound information (i.e., RF signal data or IQ data pairs) for presentation on a display system 234. The signal processor 232 is operable to perform one or more processing operations based on a plurality of selectable ultrasound modalities on the acquired ultrasound information. The acquired ultrasound information may be processed in real time during a scanning session as echo signals are received. Additionally or alternatively, the ultrasound information may be temporarily stored in an RF / IQ buffer 226 during a scanning session and processed in a less real-time manner when operating online or offline. In an exemplary embodiment, the signal processor 232 may include a shear wave elastography processing module 240.
[0052] The ultrasound system 200 is operable to continuously acquire ultrasound information at a frame rate suitable for the imaging situation under consideration. Typical frame rates are in the range of 20fps to 70fps, but can be lower or higher. For example, the frame rate associated with the high pulse repetition frequency used to image the shear waves 5 in the tissue 1 for shear wave elastography can be higher. In various embodiments, the pulse repetition frequency in the shear wave elastography mode is at least 300 pulses / second, and preferably greater than or equal to 1000 pulses / second. The acquired ultrasound information can be displayed on the display system 234 at a display rate that is the same as the frame rate or at a slower or faster display rate than the frame rate. An image buffer 236 is included for storing processed frames of acquired ultrasound information that are not scheduled for immediate display. Preferably, the image buffer 236 has sufficient capacity to store ultrasound information frames equivalent to at least a few seconds. The ultrasound information frames are stored in a manner that facilitates retrieval according to their acquisition order or time. The image buffer 236 can be embodied as any known data storage medium.
[0053] The shear wave elastography processing module 240 may include suitable logic components, circuits, interfaces and / or codes that are operable to handle processing of shear wave ultrasound data to provide information about a region of interest, such as a region of interest in the liver. As used herein, the term "shear wave ultrasound data" refers to ultrasound information received at the signal processor 232 corresponding to a received echo 209 generated by backscattering the transmitted ultrasound signal 207 from structures (e.g., tissue 1) and surgical instruments in the object of interest that are deformed by the shear wave 5. In this regard, the shear wave elastography processing module 240 may include suitable logic components, circuits, interfaces and / or codes that are operable to handle processing of shear wave ultrasound data to determine the local distribution of shear wave velocity in the tissue 1. The shear wave velocity may be calculated by direct inversion of the Helmholtz equation, time of flight measurement, or any suitable calculation method. For example, shear wave ultrasound data can be acquired after a push disturbance is induced in the tissue 1 by the force of the focused ultrasound beam 205 or by an external driving force 22. The push disturbances 205, 22 generate shear waves 5 that propagate laterally from the disturbance point. The ultrasound system 200 acquires shear wave ultrasound data using a high pulse repetition frequency. As used herein, the term "high pulse repetition frequency" refers to a pulse repetition frequency of at least 300 pulses / second. In a preferred embodiment, the pulse repetition frequency used to acquire shear wave ultrasound data is greater than or equal to 1000 pulses / second.
[0054] The shear wave elastic imaging processing module 240 may include suitable logic components, circuits, interfaces and / or codes that are operable to convert the local distribution of shear wave velocity in the tissue 1 into a map, such as a velocity distribution map, an elasticity map, a spatial gradient map, or any suitable map representing the contrast between the needle 10 and the surrounding tissue 1. For example, the local distribution can be plotted based on the shear wave velocity to generate a velocity distribution map. For another example, the local distribution can be converted into an elasticity map by calculating the hardness based on Young's modulus, a similar shear modulus, or any suitable conversion calculation method. In addition, in various embodiments, a spatial gradient filter can be applied to the velocity distribution map and / or the elasticity map to generate a spatial gradient map that provides enhanced visualization of the tissue of interest.
[0055] The graph represents the velocity of the shear wave passing through the tissue at the lateral position from the interference point in the shear wave ultrasound data. The shear wave propagation velocity corresponds to the hardness of the tissue at the lateral position. Specifically, a higher shear wave velocity corresponds to a higher hardness, while a lower shear wave velocity corresponds to a lower hardness. Based on differences in velocity and / or elasticity of tissue (e.g., a region of interest in the liver), the hardness of the liver can be assessed. For example, these graphs can be color-coded graphs or grayscale graphs with a color or grayscale range corresponding to the shear wave velocity and / or elasticity. Specifically, the elasticity graph can have a dark blue or dark gray / black corresponding to soft elasticity to a red or light gray / white corresponding to hard elasticity, and so on. The graph with elasticity information can be superimposed on an ultrasound image such as a B-mode image or any suitable ultrasound image.
[0056] Additionally and / or alternatively, the shear wave elastography processing module 240 may include suitable logic, circuitry, interfaces and / or code operable to perform image segmentation. In various embodiments, the shear wave elastography processing module 240 may automatically or semi-automatically perform image segmentation.
[0057] Figure 3 A system 300 for training and using a deep learning model for segmenting ultrasound image data is shown. The training portion of the system 300 is shown on the left side of the dashed line, while the instrument portion of the system 300 is shown on the right side. An example of an instrument is shown in Figure 1 and Figure 2Training can be optionally performed on the instrument or in conjunction with the instrument. For example, in order to obtain data for training, ultrasound image data must be obtained, and in some cases, the ultrasound image data is annotated. Training can be performed by one or more processors, hardware, and / or memories such as those discussed herein. One or more machine learning frameworks disclosed herein can be implemented using software such as PyTorch or C++ for learning frameworks. In an embodiment, the same software is used to implement both the self-supervised learning framework and the supervised learning framework. According to such embodiments, the same model structure can be used for both frameworks.
[0058] On the training side of system 300, a self-supervised learning framework 310 and a supervised learning framework 340 are provided. Both frameworks 310 and 340 include instructions stored in one or more computer-readable media, and these instructions can be executed by one or more processors. Instructions can be written using source code such as PyTorch or C++, and then compiled into executable instructions. Framework 310 trains model 330 stored in one or more memories. Model 330 is further improved by framework 340 to obtain model 360. Model 360 may be similar to model 330. For example, model 330 and model 360 may have the same structure, but the nodes may have different weights. Framework 340 uses annotated data set 350 to improve model 330 to form model 360, which is a model for segmentation on an instrument.
[0059] The framework 310 may receive and process an unannotated dataset 320 including ultrasound image data (e.g., B-mode images). Such images may be referred to as training images. The unannotated dataset 320 may include ultrasound image data of the abdomen of different patients, including image data of the patient's liver or areas where the probe has poor contact with the patient. Although the framework 310 may learn to represent different patterns, appearances, and conditions that appear in ultrasound images of the liver, it may not inherently understand the real-world meaning of those patterns. In order to use the learned representations for specific tasks (e.g., segmenting the liver or areas where the probe has poor contact), a framework 340 (discussed below) may be used. The framework 310 may learn high-dimensional representations of images that capture significant visual features but may not be clearly explained in human terms. For example, for ultrasound images of the liver, the framework 310 may learn representations of various textures, patterns, shapes, structures, and contrasts present in the data. The framework 310 may process (e.g., continuously process) images in the unannotated dataset 320 using a learning algorithm until a model 330 is finally determined. Framework 310 includes training instructions that are executed by a processor to train model 330 using unannotated dataset 320. As framework 310 continues to process training images in unannotated dataset 320, it can adjust model 330 accordingly in a continuous manner.
[0060] The framework 310 may be a contrastive learning framework for learning visual representations (e.g., a patient's liver or a probe contact area in image data). One such framework is SimCLR. The framework 310 may provide a contrastive loss function, wherein a neural network is trained to distinguish different views of the same image data, rather than, for example, training a model to classify images into a set of predefined categories. The framework 310 may be considered a self-supervised framework rather than a supervised framework. The framework 310 does not require an annotated dataset to train the model 330. The framework 310 may include various components. One such component of the framework 310 may be random data augmentation, i.e., random cropping, followed by resizing back to the original size, random color distortion, and random Gaussian blur augmentation methods may be used to create two related views (positively) of the same example (e.g., in sequence). This allows the model 330 to learn to distinguish different views of the same image. Another component of the framework 310 may be a feature extractor, such as a convolutional neural network (CNN). Such a CNN may be based on a ResNet-50 architecture. The CNN may map an input image to a feature space and generate a feature representation of the image. The model 330 is then fine-tuned on the target dataset using a contrastive loss function, which is another possible component of the framework 310. The contrastive loss function can compare the feature representations of different views of the same image and attempt to minimize the distance between the representations of the same image while maximizing the distance between the representations of different images. Another component of the framework 310 can be a projection head, such as a multi-layer perceptron (MLR) with one hidden layer. The MLR can be used to project the feature representation of the image into a lower dimensional space, which can be used as the representation of the image provided to the contrastive loss function. This can be achieved by performing a linear transformation on the feature representation followed by applying a nonlinear activation function (such as ReLU).
[0061] In an embodiment where the model 330 is trained using the framework 310, during training, N "examples" of a mini-batch are randomly selected and a comparative prediction task is performed on pairs of augmented examples, resulting in 2N data points. In SimCLR, the term "example" refers to an image that has been altered in a particular way for training purposes. This alteration process, known as data augmentation, can include steps such as cropping a portion of an image, changing its color, or blurring it. These altered versions of the same image are then used by the system to learn how to distinguish between different views of the same image and completely different images. Instead of explicitly sampling negative examples, all other 2(N-1) augmented examples in the mini-batch are treated as negative examples for a given positive pair. The final loss is calculated for all positive pairs in the mini-batch and is defined as:
[0062]
[0063] in, is an indicator function for determining whether a sample is positive or negative relative to the anchor image (i.e., it is 1 if k is not equal to i). The temperature parameter τ controls the degree of similarity between representations and can be used to adjust the difficulty of the contrast task (increasing the temperature produces a softer and less discriminative similarity function, while decreasing the temperature produces a harder and more discriminative similarity function).
[0064] The framework 310 may use relatively large batch sizes (eg, 256 to 8192 images) and / or may use a LARS optimizer. Aggregating the mean and variance of batch normalization across all devices during training may be performed.
[0065] After the model 330 is trained by the framework 310, it is further trained by the framework 340, which may be a supervised learning framework. The framework 340 further trains the model 330 using the dataset 350 to form a model 360. The dataset 350 includes training images 354, validation images 356. Both the training images 354 and the validation images 356 are annotated with annotations 352. The annotations 352 may be annotations of the liver and / or annotations showing areas where the probe has poor contact with the patient. The annotations 352 may be drawn manually by a clinician using an electronic tool. Different annotations 352 may be associated with each image in the dataset 350. The framework 340 may first process the training images 354 (with annotations 352) and then process the validation images 356 (with annotations 352) to train the model 360. The validation images 356 may indicate the effectiveness of the model 360 while further tuning the performance. As framework 340 processes training images 354 and / or validation images 356 , framework 340 continues to responsively adjust model 360 .
[0066] An example of framework 340 is the ENet (Efficient Neural Network) framework. Framework 340 can be a deep neural network architecture for real-time semantic segmentation and can achieve high accuracy when segmenting images while being computationally efficient. Such frameworks 340 enable real-time performance on resource-constrained devices such as medical devices. In the example of ENet, the framework 340 architecture is based on ResNet and is divided into several levels. For example, the first three levels form the encoder portion of the network, and the last two levels form the decoder portion. Each level may include several bottleneck modules, wherein each module consists of a convolutional layer (e.g., three convolutional layers). These layers may include a 1x1 projection layer, a main convolutional layer, and a 1x1 expansion layer for reducing dimensionality. Batch normalization and PReLU activation may be used between all convolutional layers. In the case of downsampling, a maximum pooling layer may be added to the main branch. Framework 340 may use spatial dropout as a regularization technique, wherein different dropout rates are used in different levels. In the decoder, maximum anti-pooling can be used instead of maximum pooling, and unbiased spatial convolution can be used instead of padding. The last layer of framework 340 can be a full convolution. Framework 340 can be optimized to improve performance, such as not using or avoiding bias terms in projection to reduce memory usage. Batch normalization can be used between one or more convolutional layers. In addition, nonlinearity can be used to improve accuracy. The term "nonlinearity" in this text refers to the activation function used in the neural network architecture. In a neural network, an activation function can be used to introduce nonlinearity into the output of a neuron. This allows the network to learn from the errors it generates and adjust its weights accordingly, enabling it to model the complex nonlinear relationship between its input and output. In the case of ENet, nonlinearity can be introduced by using the PReLU activation function.
[0067] After the model 360 has been trained, it can be used with instruments such as those disclosed in the system 100, the system 200, and as described above. The framework 370 can be implemented by a processor such as the segmentation processor 150 or the signal processor 232. The framework 370 can acquire an ultrasound image 380 generated by an instrument, for example, ultrasound image data generated by the system 100 or the system 200. The ultrasound image 380 can be referred to as a test image. The framework 370 includes inference instructions for processing the ultrasound image 380 using the trained model 360. Thus, the framework 370 can determine, for example, a segmentation 390 of the liver and / or a probe poor contact region in the ultrasound image 380.
[0068] Once the segmentations 390 are determined, they can be used for further processing, such as by instruments such as those disclosed in system 100, system 200. Examples of further processing are disclosed in embodiments of U.S. Ser. No. 18 / 205,809, filed on June 5, 2023, which is incorporated herein by reference in its entirety. For example, the segmented liver can be used to automatically determine the location or possible location of a region of interest from which data (e.g., data generated by a shear wave elastography process, such as in Figure 2 For another example, the segmentation of the probe poor contact area can be used to generate an indication of the location where the probe is poorly in contact with the patient's skin and display the indication to the instrument operator. Examples of such displays include display system 134, display system 234.
[0069] Those skilled in the art will appreciate that various changes may be made and equivalents may be substituted without departing from the scope of the new technology disclosed in this application. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the new technology without departing from the scope of the invention. Therefore, the new technology is not intended to be limited to the specific technology disclosed, but the new technology will include all technologies that fall within the scope of the appended claims.
Claims
1. A method (300) for training a model (360) to segment a test image (380), wherein the test image (380) comprises ultrasound image data, the method comprising: Receiving a first plurality of training images (320) at a self-supervised learning framework (310), wherein the first plurality of training images (320) includes ultrasound data corresponding to a liver of a patient; Processing the first plurality of training images (320) using a learning algorithm of the self-supervised learning framework (310), and responsively adjusting a trained model (330); as well as The trained model (330) and a second plurality of training images (350) are received at a supervised learning framework (340), and the trained model (360) is responsively adjusted, wherein the second plurality of training images (350) includes ultrasound data corresponding to a patient's liver (354, 356) and annotations (352) of the liver.
2. The method of claim 1, wherein the self-supervised learning framework (310) comprises a contrastive learning framework.
3. The method of claim 2, wherein the self-supervised learning framework (310) uses a convolutional neural network as an encoder.
4. The method of claim 2, wherein the self-supervised learning framework (310) employs a projection head.
5. The method of claim 1, wherein the supervised learning framework (340) comprises an encoder and a decoder, the encoder comprising a plurality of stages and the decoder comprising a plurality of stages, wherein each stage comprises a plurality of bottleneck modules, the plurality of bottleneck modules being configured to manage dimensionality.
6. A method (300) for segmenting a structure (390) in ultrasound image data (380), the method (300) comprising: Acquiring ultrasound image data of the patient including the liver using ultrasound energy (380); Receiving the ultrasound image data (380) at a processor (132, 232); executing, by the processor (132, 232), inference instructions to segment the liver in real time in the ultrasound image data (380) to form a segmented liver (390); and The ultrasound image data (380) is presented on a display (134, 234).
7. The method (300) of claim 6, further comprising: determining a region of interest within the segmented liver (390); as well as Shear wave elastography (200) is performed on data obtained from the region of interest. The method according to claim 7 , wherein the region of interest is identified automatically.
9. The method of claim 6, further comprising executing, by the processor (134, 234), inference instructions to segment a probe poor contact region in the ultrasound image data (380).
10. The method of claim 9, further comprising presenting information corresponding to the probe poor contact area on the display (134, 234) along with the ultrasound image data (380).
11. A system (100, 200), comprising: An ultrasound probe (104, 204) and a receiver configured to obtain ultrasound image data (380) of a patient including a liver; a processor (132, 232) configured to receive the ultrasound image data (380) and execute inference instructions (370) to segment the liver in the ultrasound image data (380) in real time to form a segmented liver (390); and A display (134, 234) configured to present the ultrasound image data (380) and information associated with the segmented liver (390).
12. The system of claim 11, wherein the processor (132, 232) is further configured to determine a region of interest within the segmented liver (390) and to cause a shear wave elastography process (200) to be performed to obtain shear wave elastography data from the region of interest. The system of claim 11 , wherein the region of interest is determined automatically.
14. The system of claim 11, wherein the processor is further configured to execute inference instructions (370) to segment probe poor contact areas in the ultrasound image data (380).
15. The system of claim 14, wherein the display (134, 234) is further configured to present information corresponding to the probe poor contact area together with the ultrasound image data.
Citation Information
Patent Citations
Ultrasound imaging techniques for shear-wave elastography
US20240398382A1
