Biopsy prediction and guidance using ultrasound imaging and associated devices, systems and methods

Abstract

An ultrasound imaging system includes: a communication interface that communicates with an ultrasound imaging device and is configured to receive an ultrasound image of an anatomical structure of an object; and a processor that communicates with the communication interface and is configured to: apply a prediction network to the ultrasound image to identify multiple potential malignant locations in the anatomical structure of the object and determine multiple malignant possibilities at the multiple potential malignant locations; and output the multiple potential malignant locations and the multiple malignant possibilities for the multiple potential malignant locations to a display that communicates with the processor to guide a biopsy determination for the anatomical structure of the object.

Description

Technical Field

[0001] The present disclosure relates generally to ultrasound imaging, and in particular to providing automated systems and methods for guided biopsies. Background Art

[0002] Ultrasound plays an important role in screening and diagnosing many conditions in different anatomical structures. Ultrasound is also often used to guide the biopsy procedure from the perspective of biopsy needle guidance and from the perspective of target recognition. In the thyroid gland, for example, brightness mode (B-mode) ultrasound features (e.g., size, echogenicity, shape, calcifications, margins, and the presence or absence of cystic components) are used to classify nodules as benign or malignant. In addition to B-mode ultrasound features, elasticity features can also be used.

[0003] A thyroid suspected of malignancy typically includes multiple candidate biopsy locations (e.g., multiple nodules or multiple locations within a large nodule). The need for a biopsy and the biopsy location are typically identified by a clinician using the clinician's expertise to determine the level of suspicion of cancer in a given nodule and / or using an algorithm based on nodule size to recommend whether the patient should be instructed to undergo a biopsy. For example, for highly suspected nodules, a biopsy is indicated for nodules as small as about 1 centimeter (cm), while for nodules with very low suspicion, a biopsy is indicated for nodules greater than 2 cm in diameter. Once the biopsy location is identified, the needle can be advanced toward the target under ultrasound guidance.

[0004] Biopsy samples can be obtained using one of two techniques - capillary technique or aspiration technique. Capillary technique and aspiration technique have similar workflows, but differ in the way the samples are collected. Typically, a hollow needle with a beveled tip and a 10 cubic centimeter (cc) Luer Lock syringe are used. The length of the needle can be between about 1.25 inches and about 1.5 inches. The thickness of the needle can be between about 23G (gauge pressure) and about 27G. Thicker needles (e.g., about 20G or 22G) can cause patient trauma and / or blood contamination. Therefore, it is recommended to use a needle of at least about 24G to about 25G. In some cases, for deeper insertion and / or for patients with thicker necks, a solid core needle shorter than the needle can be inserted into the needle (e.g., a hollow needle) to provide support.

[0005] Because the suspected malignant thyroid can include multiple biopsy candidate locations, it is necessary for the clinician to select the preferred location. In addition, it is clinically shown that no more than n times or n attempts (wherein, n can be generally about 5) are allowed in a biopsy to minimize the trauma of the patient. The selection of the biopsy location can also be based on the proximity of the nodule and / or needle path to one of the many key structures around the thyroid (e.g., jugular vein, carotid artery, trachea and esophagus). For this reason, biopsy is usually unable to diagnose or may be difficult to make a clear diagnosis based on the biopsy sample. Another problem is false negative biopsy results, wherein, thyroid malignancy may obtain benign results. The lack of ultrasound experience by some users can also lead to long-term intervention and / or difficult intervention, thereby causing patient trauma and frustration to the clinician. Summary of the invention

[0006] Although existing ultrasound imaging has been proven to be useful for clinical guidance and diagnosis, there is still a clinical need for improved systems and techniques for providing automated biopsy guidance tools. Embodiments of the present disclosure provide a deep learning framework to assist and guide users in determining and performing biopsies. The disclosed embodiments can identify one or more suspected cancer locations in a live ultrasound image and one or more locations that are excluded due to expected non-diagnostic / benign results. The disclosed embodiments can display a cancer suspicion map with a malignant prediction probability for each pixel superimposed on the live ultrasound image and a corresponding confidence map for the prediction. The deep learning prediction network is also capable of identifying organs at risk (OARs) or critical structures in the live ultrasound image. The disclosed embodiments can determine one or more optimal biopsy paths in the live ultrasound image, which can maximize yield and avoid critical structures. The disclosed embodiments can display the optimal biopsy path via a contextual user interface, such as providing a display of the needle path relative to the position of the ultrasound imaging probe in use and / or relative to the target biopsy location. The disclosed embodiments can provide a user interface to guide the user to manipulate the ultrasound imaging probe from a suboptimal position to a position that can provide an optimal image for guiding a biopsy sample. The disclosed embodiments may provide a warning to the user when the user attempts to insert the needle into a trajectory that may compromise the OAR or may not reach the target.

[0007] In one embodiment, an ultrasound imaging system includes: a communication interface that communicates with an ultrasound imaging device and is configured to receive an ultrasound image of an anatomical structure of a subject; and a processor that communicates with the communication interface and is configured to: apply a prediction network to the ultrasound image to: identify a plurality of potential malignant locations in the anatomical structure of the subject, determine a plurality of malignancy probabilities at the plurality of potential malignant locations, identify critical areas of the anatomical structure of the subject to avoid biopsy, and determine a plurality of confidence levels for the plurality of malignancy probabilities at the plurality of potential malignant locations; and output the plurality of potential malignant locations and the plurality of malignancy probabilities for the plurality of potential malignant locations to a display that communicates with the processor to guide a biopsy determination for the anatomical structure of the subject.

[0008] In some embodiments, wherein the ultrasound image includes at least one of the following: brightness mode (B-mode) information, strain information, elasticity information, or tissue Doppler information. In some embodiments, wherein the processor is further configured to output an overlay of indications of at least one of the multiple potential malignant locations, the multiple malignant possibilities, the multiple confidence levels, and the key areas on the ultrasound image to the display. In some embodiments, the system further includes a display, the display being configured to display a first graph and a second graph, the first graph including an overlay of indications of the multiple potential malignant locations, the multiple malignant possibilities, and the key areas on a first instance of the ultrasound image, and the second graph including an overlay of indications of the multiple confidence levels at the multiple potential malignant locations on a second instance of the ultrasound image. In some embodiments, wherein the processor is further configured to determine a biopsy needle path for biopsy of the anatomical structure of the object based on at least one of the multiple potential malignant locations, the key areas, or a target biopsy location in the anatomical structure of the object. In some embodiments, wherein the processor is further configured to determine a target biopsy location for the anatomical structure of the object based on at least one of the multiple malignancy probabilities or the multiple confidence levels for the multiple malignancy probabilities at the multiple potential malignancy locations, and wherein the biopsy needle path is also determined based on the determined target biopsy location. In some embodiments, the system further comprises a user interface in communication with the processor and configured to receive a selection of the target biopsy location. In some embodiments, wherein the processor is further configured to output the determined biopsy needle path to the display as an overlay on the ultrasound image. In some embodiments, wherein the processor is further configured to apply the prediction network to the ultrasound image to determine the biopsy needle path. In some embodiments, wherein the processor is further configured to: determine that the critical area is along a trajectory path of a biopsy needle directed to the anatomical structure of the object; and output an indication of a potential collision between the biopsy needle and the critical area to the display. In some embodiments, the ultrasound image is received when the ultrasound imaging device is positioned at a first imaging position relative to the anatomical structure of the object, and wherein the processor is further configured to: apply the prediction network to the ultrasound image to determine a motion control configuration, the motion control configuration being used to reposition the ultrasound imaging device from the first imaging position to a second imaging position for performing the biopsy; and output a visual indicator to the display for repositioning the ultrasound imaging device based on the motion control configuration.In some embodiments, the prediction network is trained by providing a plurality of test ultrasound images of an anatomical structure of a test subject, a biopsy location of a biopsy performed on the anatomical structure of the test subject, and a pathological result of the biopsy performed at the biopsy location; and assigning scores to pixels of the plurality of test ultrasound images based on the biopsy location and the pathological result. In some embodiments, the processor is further configured to update the prediction network based on at least one of the following: a target biopsy location for a biopsy determined based on the plurality of potential malignant locations and the plurality of malignant possibilities; or a pathological result of the biopsy. In some embodiments, the anatomical structure of the subject includes at least a portion of a thyroid gland.

[0009] In one embodiment, a method of ultrasound imaging includes: receiving an ultrasound image of an anatomical structure of an object from an ultrasound imaging device; applying a prediction network to the ultrasound image to perform the following operations: identifying a plurality of potential malignant locations in the anatomical structure of the object, determining a plurality of malignant possibilities at the plurality of potential malignant locations, identifying critical areas of the anatomical structure of the object to avoid biopsy, and determining a plurality of confidence levels for the plurality of malignant possibilities at the plurality of potential malignant locations; and displaying the plurality of potential malignant locations and the plurality of malignant possibilities for the plurality of potential malignant locations through a display to guide a biopsy determination of the anatomical structure of the object.

[0010] In some embodiments, wherein the displaying comprises: displaying a first map including an overlay of the multiple potential malignant locations, the multiple malignant probabilities, and an indication of the critical region on a first instance of the ultrasound image; and displaying a second map including an overlay of the multiple confidence levels corresponding to the multiple potential malignant locations on a second instance of the ultrasound image. In some embodiments, the method further comprises: determining a target biopsy location for biopsy of an anatomical structure of the subject based on at least one of the multiple malignant probabilities at the multiple potential malignant locations or the multiple confidence levels for the multiple malignant probabilities; determining a biopsy needle path for performing the biopsy based on at least one of the multiple potential malignant locations, the critical region, or the target biopsy location of the anatomical structure of the subject; and displaying the determined biopsy needle path as an overlay on the first map by the display. In some embodiments, the method further comprises: determining that the critical region is along a trajectory path of a biopsy needle directed toward the anatomical structure of the subject; and displaying an indication of a potential collision between the biopsy needle and the critical region by the display. In some embodiments, the ultrasound image is received when the ultrasound imaging device is positioned at a first imaging position relative to the anatomical structure of the object, and wherein the method further comprises: applying the prediction network to the ultrasound image to determine a motion control configuration, the motion control configuration being used to reposition the ultrasound imaging device from the first imaging position to a second imaging position for performing the biopsy; and displaying, by the display, a visual indicator for repositioning the ultrasound imaging device based on the motion control configuration.

[0011] Other aspects, features, and advantages of the present disclosure will become apparent from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Illustrative embodiments of the present disclosure will be described with reference to the accompanying drawings, in which:

[0013] Figure 1 is a schematic diagram of an ultrasound imaging system according to some aspects of the present disclosure.

[0014] Figure 2 is an ultrasound image of a patient's thyroid gland and surrounding areas according to some aspects of the present disclosure.

[0015] Figure 3 is a schematic diagram illustrating an automated pixel-level ultrasound image annotation and needle path prediction scheme according to some aspects of the present disclosure.

[0016] Figure 4is a graphical display of a malignancy potential map for an ultrasound image according to some aspects of the present disclosure.

[0017] Figure 5 is a graphical display of a malignancy likelihood confidence map for an ultrasound image according to some aspects of the present disclosure.

[0018] Figure 6 is a graphical display of a malignancy potential map with visual indicators for a biopsy needle path according to aspects of the present disclosure.

[0019] Figure 7 is a graphical display of a contextual display including a suggested biopsy needle path in relation to an ultrasound probe and a needle entry point according to aspects of the present disclosure.

[0020] Figure 8 is a schematic diagram illustrating an automated ultrasound probe position guidance scheme according to some aspects of the present disclosure.

[0021] Fig. 9 is a graphical display of a user interface for guiding probe positioning according to some aspects of the present disclosure.

[0022] Fig.10 is a schematic diagram illustrating the configuration of a deep learning network according to some aspects of the present disclosure.

[0023] Fig.11 is a schematic diagram illustrating a deep learning network training scheme according to some aspects of the present disclosure.

[0024] Fig.12 is a schematic diagram illustrating a deep learning network fine-tuning scheme according to some aspects of the present disclosure.

[0025] Fig.13 is a flow chart of a method of providing a user alert in a biopsy examination according to aspects of the present disclosure.

[0026] Fig.14 is a flow chart of a method of guiding a biopsy according to aspects of the present disclosure. DETAILED DESCRIPTION

[0027] In order to promote the understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the accompanying drawings, and specific language will be used to describe these embodiments. However, it should be understood that it is not intended to limit the scope of the present disclosure. As would be generally expected by those skilled in the art to which the present disclosure relates, any changes and further modifications to the described equipment, systems and methods and any further application of the principles of the present disclosure are fully anticipated and included in the present disclosure. In particular, it is fully anticipated that the features, components and / or steps described with respect to an embodiment may be combined with the features, components and / or steps described with respect to other embodiments of the present disclosure. However, for the sake of brevity, the numerous iterative forms of these combinations will not be described separately.

[0028] Figure 1 1 is a schematic diagram of an ultrasound imaging system 100 according to some aspects of the present disclosure. The system 100 is used to scan a region or volume of a patient's body. The system 100 includes an ultrasound imaging probe 110 that communicates with a host 130 via a communication interface or link 120. The probe 110 includes a transducer array 112, a beamformer 114, a processing component 116, and a communication interface 118. The host 130 includes a display 132, a processing component 134, and a communication interface 136.

[0029] In an exemplary embodiment, the probe 110 is an external ultrasound imaging device that includes a housing configured to be handheld and operated by a user. The transducer array 112 can be configured to obtain ultrasound data when the user grasps the housing of the probe 110 so that the transducer array 112 is positioned adjacent to and / or in contact with the patient's skin. The probe 110 is configured to obtain ultrasound data of anatomical structures within the patient's body when the probe 110 is positioned outside the patient's body.

[0030] The transducer array 112 transmits an ultrasonic signal toward the anatomical target 105 and receives an echo signal reflected from the target 105 and transmitted back to the transducer array 112. The ultrasonic transducer array 112 can include any suitable number of acoustic elements, including one or more acoustic elements and / or multiple acoustic elements. In some cases, the transducer array 112 includes a single acoustic element. In some cases, the transducer array 112 can include an array of acoustic elements having any number of acoustic elements in any appropriate configuration. For example, the transducer array 112 can include 1 acoustic element to 1000 acoustic elements, including acoustic elements of a number such as 2 acoustic elements, 4 acoustic elements, 36 acoustic elements, 64 acoustic elements, 128 acoustic elements, 500 acoustic elements, 812 acoustic elements and / or other larger and smaller values. In some cases, the transducer array 112 may include an array of acoustic elements having any number of acoustic elements in any appropriate configuration, for example, a linear array, a planar array, a curved array, a curvilinear array, a circular array, an annular array, a phased array, a matrix array, a one-dimensional (1D) array, a 1.x-dimensional array (e.g., a 1.5D array), or a two-dimensional (2D) array. An array of acoustic elements (e.g., one or more rows, one or more columns, and / or one or more orientations) can be uniformly or independently controlled and activated. The transducer array 112 can be configured to obtain a one-dimensional, two-dimensional, and / or three-dimensional image of the patient's anatomical structure. In some embodiments, the transducer array 112 may include a piezoelectric micromachined ultrasonic transducer (PMUT), a capacitive micromachined ultrasonic transducer (CMUT), a single crystal, lead zirconate titanate (PZT), a PZT composite material, other suitable transducer types, and / or combinations thereof.

[0031] The beamformer 114 is coupled to the transducer array 112. For example, the beamformer 114 controls the transducer array 112 for transmission of ultrasound signals and reception of ultrasound echo signals. The beamformer 114 provides an image signal to the processing component 116 based on the response or the received ultrasound echo signal. The beamformer 114 may include multiple stages of beamforming. Beamforming can reduce the number of signal lines for coupling to the processing component 116. In some embodiments, the transducer array 112 combined with the beamformer 114 may be referred to as an ultrasound imaging component.

[0032] In some embodiments, target 105 may include at least a portion of a patient's thyroid gland that is biopsied. In other embodiments, target 105 may include any anatomical structure (e.g., lungs, blood vessels, heart, kidneys, and / or liver) of a patient that is suitable for ultrasound imaging.

[0033] The processing component 116 is coupled to the beamformer 114. The processing component 116 may include a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a controller, a field programmable gate array (FPGA) device, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processing component 134 may also be implemented as a combination of computing devices (e.g., a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors combined with a DSP core, or any other such configuration). The processing component 116 is configured to process the beamformed image signal. For example, the processing component 116 may perform filtering and / or orthogonal demodulation to regulate the image signal. The processing component 116 and / or 134 can be configured to control the array 112 to obtain ultrasound data associated with the target 105.

[0034] The communication interface 118 is coupled to the processing component 116. The communication interface 118 may include one or more transmitters, one or more receivers, one or more transceivers, and / or circuits for transmitting and / or receiving communication signals. The communication interface 118 can include hardware components and / or software components that implement a specific communication protocol suitable for transmitting signals to the host 130 via the communication link 120. The communication interface 118 can be referred to as a communication device or a communication interface module.

[0035] The communication link 120 may be any suitable communication link. For example, the communication link 120 may be a wired link, such as a Universal Serial Bus (USB) link or an Ethernet link. Alternatively, the communication link 120 may be a wireless link, such as an Ultra Wideband (UWB) link, an Institute of Electrical and Electronics Engineers (IEEE) 802.11 WiFi link, or a Bluetooth link.

[0036] At host 130, communication interface 136 may receive the image signal. Communication interface 136 may be substantially similar to communication interface 118. Host 130 may be any suitable computing and display device, such as a workstation, personal computer (PC), laptop, tablet, or mobile phone.

[0037] The processing component 134 is coupled to the communication interface 136. The processing component 134 can be implemented as a combination of software components and hardware components. The processing component 134 may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a controller, an FPGA device, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. The processing component 134 can also be implemented as a combination of computing devices (e.g., a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors combined with a DSP core, or any other such configuration). The processing component 134 can be configured to generate image data based on the image signal received from the probe 110. The processing component 134 can apply advanced signal processing and / or image processing techniques to the image signal. In some embodiments, the processing component 134 can form a three-dimensional (3D) volume image based on the image data. In some embodiments, the processing component 134 can perform real-time processing on the image data to provide a streaming video of an ultrasound image of the target 105.

[0038] In some embodiments, the processing component 134 can perform image analysis on the image data or image frame for biopsy determination. For example, the processing component 134 can receive an ultrasound image of a target 105 including the patient's tissue. The processing component 134 can apply a deep learning-based technique to determine the location of a potential malignancy or abnormality at the patient's tissue, determine whether a biopsy is needed, identify the target biopsy location and / or suggest a potential needle path that avoids a critical organ or tissue area and reaches the target location. The processing component 134 can apply a deep learning-based technique to determine whether the imaging plane can provide an optimal view of the target biopsy location and how to guide the user to the optimal view for guiding a biopsy when manipulating the probe 110. The processing component 134 can monitor the insertion point or entry point of the biopsy needle selected by the clinician, determine the trajectory of the biopsy needle that will collide and reach the critical area of ​​the tissue, and / or generate a warning to the clinician when a collision occurs between the trajectory of the biopsy needle and the critical area. This article describes the mechanism of predicting and guiding biopsy in more detail.

[0039] The display 132 is coupled to the processing component 134. The display 132 can be a monitor or any suitable display. The display 132 is configured to display ultrasound images, image videos, and / or information associated with potential malignant tissue locations, target biopsy locations, suggested biopsy needle paths, instructions for moving the probe 110 to an optimal imaging view for biopsy, and / or warnings regarding biopsy generated by the processing component 134 as described in more detail herein.

[0040] The system 100 can be configured for various stages of the biopsy prediction and guidance process based on ultrasound imaging. In one embodiment, the system 100 can be used to collect ultrasound images to form a training data set for deep learning network training. For example, the host 130 may include a memory 138, and the memory 138 can be any suitable storage device, such as a cache memory (e.g., a cache memory of the processing component 134), a random access memory (RAM), a magnetoresistive RAM (MRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a solid-state storage device, a hard disk drive, a solid-state drive, other forms of volatile and non-volatile memory, or a combination of different types of memory. The memory 138 can be configured to store an image data set 140 for deep learning-based training.

[0041] In some embodiments, the system 100 can be used to train a deep learning network for biopsy prediction and guidance. For example, a deep learning network can be trained using an ultrasound image of a tissue that includes a potential malignant area, a target location to which a tissue biopsy is to be applied, and / or corresponding pathological results obtained from the biopsy. A deep learning network can be trained to identify potential malignant tissue locations, identify critical areas to avoid biopsy (e.g., organs at risk (OARs)), and / or determine the probability of malignancy and the associated confidence level at the identified location. A deep learning network can be trained to determine a biopsy target location based on the identified potential malignant tissue location and / or determine one or more biopsy needle paths that avoid critical areas and reach the target. A deep learning network can be trained to determine movements for manipulating the probe 110 to an imaging plane suitable for guiding a biopsy.

[0042] In some embodiments, the system 100 can be used in a live biopsy procedure in a clinical setting, where a trained deep learning network can be applied to determine whether a biopsy is needed, identify potential malignant tissue locations, determine target biopsy locations, and / or suggest one or more needle paths that avoid critical areas for biopsy. The mechanism for automatically and systematically predicting and guiding biopsy procedures based on deep learning techniques is described in more detail herein.

[0043] Figure 22 is an ultrasound image 200 of a patient's thyroid gland 210 and surrounding areas according to some aspects of the present disclosure. A system similar to system 100 can be used to capture image 200. Image 200 shows a nodule 212 in a patient's thyroid gland and structures and / or tissues near the patient's thyroid gland. Nodule 212 can be malignant or benign. It may be necessary to biopsy nodule 212 to determine whether nodule 212 is malignant or benign. Some structures and / or tissues are critical structures or organs at risk (OARs) that need to avoid the biopsy needle path. As shown, internal jugular vein 220, carotid artery 230 and trachea 240 are close to thyroid gland 210. Internal jugular vein 220, carotid artery 230 and trachea 240 are critical structures. Therefore, when biopsy is performed on nodule 212, internal jugular vein 220, carotid artery 230 and trachea 240 cannot be along the biopsy needle path.

[0044] Figure 3-12 Together, we demonstrate the use of deep learning-based techniques in the biopsy workflow to automatically and systematically predict biopsy location, biopsy needle path, and guide the biopsy process.

[0045] Figure 3 300 is a schematic diagram illustrating an automated pixel-level ultrasound image annotation and needle path prediction scheme 300 according to some aspects of the present disclosure. The scheme 300 is implemented by the system 100. The scheme 300 can provide biopsy guidance to a clinician who is preparing to perform a biopsy examination (e.g., before inserting a needle). The clinician can capture one or more images 302 of a patient's tissue (e.g., target 105) under a potential biopsy examination using the probe 110. In one embodiment, the tissue can include at least a portion of the patient's thyroid gland (e.g., thyroid gland 210), and the biopsy can be a thyroid biopsy to determine whether the thyroid gland includes abnormal cells (e.g., a malignant tumor). The image 302 can be captured using B-mode and optionally in combination with various ultrasound imaging modalities. The image 302 can provide various imaging information, including B-mode imaging information, strain imaging information, modulus imaging information, Doppler imaging information (e.g., power Doppler, color blood flow, and spectral Doppler) and tissue Doppler imaging (TDI) information and / or any suitable imaging information that can facilitate imaging data analysis to locate potential malignancies and predict and guide biopsies. For simplicity of illustration and discussion, Figure 3 Four images 302 are illustrated. However, a clinician may capture any suitable number (e.g., 5, 6, or more) of images 302 in any suitable number (e.g., approximately 5, 6, or more) of ultrasound imaging modalities to perform a biopsy procedure. As an example, the images 302 may include image A captured using B-mode, image B captured using a strain imaging mode, image C 302 captured using a shear / Young's modulus imaging mode, and image D 302 in TDI mode.

[0046] In one embodiment, images B, C, and D 302 of various modalities (e.g., strain, modulus, Doppler, and TDI) can be registered to a base B-mode image A 302. The system 100 can be configured to capture images 302 in different modalities by interleaving one or more of strain, modulus, Doppler, and TDI modalities with B-mode imaging. The system 100 can display any of the images 302 on the display 132. In some embodiments, a clinician can select a B-mode image A 302 to be displayed on the display 132. The B-mode image A 302 can be similar to the image 200.

[0047] The image 302 is sent to the processing component 134 at the host 130. The processing component 134 receives the image 302 and applies the deep learning network 310 to the received image 302. As described in more detail herein, the deep learning network 310 is trained to predict the probability of malignancy for each pixel on the input image (e.g., image 302). The deep learning network 310 can identify suspected malignant tissue locations from the patient tissue captured by the image 302. The deep learning network 310 can identify key areas or OARs (e.g., the internal jugular vein 220, the carotid artery 230, and the trachea 240) in the patient tissue that the biopsy needle path needs to avoid. The deep learning network 310 can determine the malignancy possibility at the identified potential malignant tissue location. The deep learning network 310 can determine a confidence level for the prediction or determination of the malignancy possibility at the identified potential malignant tissue location. In some embodiments, the deep learning network 310 is capable of identifying non-diagnostic and / or benign tissue locations from the patient tissue captured by the image 302. Non-diagnostic and / or benign tissue locations can be excluded from target biopsy location selection. The architecture of the deep learning network 310 and the training of the deep learning network 310 will be described in more detail herein. In some embodiments, instead of using the deep learning network 310, any suitable OAR segmentation method such as thresholding, active contours, and / or machine learning-based segmentation methods can be used to calculate the critical regions or OARs.

[0048] The deep learning network 310 outputs a malignancy potential map 320 and a confidence map 330 based on the identified potential malignant tissue locations, the identified critical regions, the determined malignancy likelihoods, and the corresponding confidence levels. The malignancy potential map 320 can be referred to as a suspicion map. Figure 4 and Figure 5 An example of a malignancy potential map 320 and a confidence map 330 is shown in . The processing component 134 can output the malignancy potential map 320 and the confidence map 330 to a user interface 360 ​​(eg, the display 132 ).

[0049] Figure 4is a graphical display of a malignancy potential map 400 for a base ultrasound image 440 according to some aspects of the present disclosure. The malignancy potential map 400 may correspond to Figure 3 The image 440 may correspond to Figure 3 B-mode image A 302 of . A malignancy potential map 400 can be displayed on the display 132 or the user interface 360. The malignancy potential map 400 includes an OAR 410 and a potential malignant tissue location 420 on an image 440 and an overlay of the corresponding malignancy probability. The potential malignant tissue location 420 is shown as a closed surface shape. The OAR 410 is shown as a closed surface shape and an open surface shape. As shown, the malignancy map 400 can include text annotations for the OAR 410. The gap between the OAR 410 and the potential malignant tissue location 420 indicates that no specific information related to malignancy and / or biopsy is obtained from the prediction. The potential malignant tissue location 420 can be color-coded according to the malignancy probability. For example, the color scheme 430 can use a first color (e.g., green) to represent a low malignancy probability and gradually transition to a second color (e.g., red) to represent a high malignancy probability.

[0050] Figure 5 FIG. 5 is a graphical display of a malignancy likelihood confidence map 500 for a base ultrasound image 440 according to some aspects of the present disclosure. The confidence map 500 may correspond to Figure 3 Confidence map 500 can be displayed on display 132 or user interface 360. Confidence map 500 includes the image 440 and the Figure 4 4. An overlay of confidence levels 510 (shown as closed surface shapes) corresponding to the likelihood of malignancy at the potential malignant locations 420 of FIG. 400 can be color coded for confidence levels 510. For example, color scheme 520 can use a first color (e.g., light blue) to represent low confidence levels and gradually transition to a second color (e.g., dark blue) and red to represent high confidence levels.

[0051] Confidence map 500 may have a one-to-one correspondence with malignancy potential map 400 (eg, at the pixel level). Confidence map 500 and malignancy potential map 400 may be displayed simultaneously on display 132. Figure 4 and Figure 5 The malignancy potential map 400 and the confidence map 500 are illustrated superimposed on a B-mode image 440 , but in some other embodiments, the malignancy potential map 400 and the confidence map 500 can be superimposed on an ultrasound image of another ultrasound imaging modality (eg, TDI, strain, or modulus).

[0052] from Figure 4 and Figure 5It can be observed that areas with low malignancy possibility (e.g., low suspicion for cancer) can be predicted with a high confidence level, areas with high malignancy possibility (e.g., high suspicion for cancer) can be predicted with a low confidence level, and areas with medium malignancy possibility (e.g., medium suspicion for cancer) can be predicted with a reasonable confidence level.

[0053] return Figure 3 After determining the malignancy potential map 320 and the corresponding confidence map 330 , a target biopsy location 340 and one or more biopsy needle paths 350 can be identified from the malignancy potential map 320 and the corresponding confidence map 330 using various mechanisms.

[0054] exist Figure 3 In the first example marked by the circle labeled 1 in , the clinician can select a target biopsy location 340 based on the malignancy potential map 320 and the corresponding confidence map 330. For example, the clinician can select a target biopsy location 340a from a tissue region with an intermediate possibility of malignancy because the region with an intermediate possibility of malignancy is most likely to provide sufficient sample volume for pathology testing. The clinician can input the selection of the target biopsy location 340a via the user interface 360. The user-selected target biopsy location 340a can be input into the deep learning network 310. The deep learning network 310 can predict one or more biopsy needle paths 350a based on the user-selected target biopsy location 340a and the identified critical areas (e.g., avoiding OAR 410). (1) As described in more detail herein, the deep learning network 310 is trained to identify an optimal biopsy needle path to reach a biopsy target.

[0055] exist Figure 3 In the second example, marked by the circle labeled 2 in FIG. 1 , a user-selected target biopsy location 340a can be input into a biopsy path determination unit 370. The biopsy path determination unit 370 can calculate one or more biopsy needle paths 350b that can safely reach the identified target biopsy location 340 without intersecting any critical area (e.g., OAR 410). The biopsy path determination unit 370 can determine the trajectory for the needle path 350b based on certain constraints (e.g., using image analysis algorithms and numerical optimization algorithms).

[0056] exist Figure 3 In the third example marked by the circle labeled 3 in FIG. 3 , the biopsy path determination unit 370 can determine the target biopsy location based on a weighted combination of the malignancy likelihood in the malignancy potential map 320 and the corresponding confidence level in the confidence map 330. The biopsy path determination unit 370 can determine one or more biopsy needle paths 350c to reach the determined biopsy location based on trajectory calculation.

[0057] exist Figure 3 In the fourth example marked by the circle labeled 4 in FIG. 3 , the deep learning network 310 can be trained based on the identified potential malignant tissue location 420, the identified critical region or OAR 410, the determined likelihood of malignancy, and the corresponding confidence level to predict a target biopsy location and one or more biopsy needle paths 350 d to reach the predicted biopsy target location. For example, the deep learning network 310 can be trained to select a biopsy needle path 350 d that maximizes gain and avoids intersections with any critical structures (e.g., OAR 410). In other words, the deep learning network 310 can automatically suggest one or more biopsy needle paths 350 d based on one or more of the images 302.

[0058] In general, the malignancy potential map 320 and the confidence map 330 can assist the clinician in selecting a target biopsy location 340 and / or a biopsy needle path 350. The clinician may be provided with the option of using any suitable combination of user-defined selections, deep learning-based predictions (e.g., by the deep learning network 310), and numerical calculations (e.g., by the biopsy path determination unit 370).

[0059] As described below Figure 6 As shown, processing component 134 can output the determined or suggested biopsy needle path 350 to display 132 .

[0060] Figure 6 is a graphical display of a malignancy potential map 600 with a visual indicator for a biopsy needle path 610 in accordance with aspects of the present disclosure. Malignancy potential map 600 may correspond to malignancy potential map 400. Biopsy needle path 610 is shown reaching a biopsy target 620 (as indicated by a dashed ellipse). Biopsy target 620 and biopsy needle path 610 may correspond to biopsy needle path 350, and biopsy target 620 may correspond to target biopsy location 340 and biopsy needle path 350. Although Figure 6 Shown as having two needle paths 610 a and 610 b , however in some embodiments any suitable number (eg, approximately 1, 3, or more) of needle paths 610 can be superimposed on the malignancy potential map 600 .

[0061] In some embodiments, the protocol 300 may also display the suggested needle path 350 relative to the needle entry point with respect to the patient's skin and / or the probe 110 during use to provide the user with a more contextual visualization of the needle path 350, as described below. Figure 7 as shown in .

[0062] Figure 77 is a graphical display 700 of a contextual display according to aspects of the present disclosure, including a suggested biopsy needle path 610 relative to an ultrasound probe 710 and a needle entry point 720. Display 700 may correspond to a display view on display 132. Probe 710 may correspond to probe 110. Biopsy needle path 610 is superimposed on B-mode image 440 along with potential malignant tissue location 420 and extends beyond image 440. Needle entry points 720a and 720b may correspond to skin entry points into needle paths 610a and 610b, respectively. For example, needle path 610a may require the needle to be inserted at an angle of about 30 degrees to the patient's skin at needle entry point 720a, while needle path 610b may require the needle to be inserted at an angle of about 15 degrees to the patient's skin at needle entry point 720b.

[0063] In some embodiments, protocol 300 may consider other factors when determining biopsy needle path 350. Other factors may include whether the clinician is left-handed or right-handed, the position of the clinician relative to the patient under examination, and / or the position of the nodule being biopsied (e.g., nodule 212) relative to probe 110 (e.g., whether probe 110 is facing to the left or right of the nodule).

[0064] Figure 8 8 is a schematic diagram illustrating an automated ultrasound probe position guidance protocol 800 according to some aspects of the present disclosure. Protocol 800 is implemented by system 100. Protocol 800 is similar to protocol 300, but can additionally provide ultrasound probe position guidance to a clinician preparing to perform a biopsy (e.g., before inserting a needle). In protocol 800, the clinician can select an initial probe position 810 for a probe (e.g., probe 110) to capture image 302. (1) The processing component 134 may determine the probe position 810 (1) A suboptimal image plane for guiding a biopsy may be provided (e.g., because the biopsy target (e.g., nodule 212) does not adequately appear in the image plane and / or the location of one or more OARs (e.g., internal jugular vein 220, carotid artery 230, trachea 240, and OAR 410) may not allow the needle to easily and / or safely reach the target without damaging the OAR).

[0065] When a suboptimal image plane is detected, the processing component 134 can apply the deep learning network 310 to determine a motion control configuration 820 to steer the probe to the optimal image plane for performing the biopsy. The motion control configuration 820 may include a method for moving the probe relative to the patient from the initial position 810 to the optimal image plane. (1) Translate and / or rotate to next position 810 (2)The deep learning network 310 can learn the position of the OARs relative to one or more thyroid lobes during the training of the deep learning network 310. The optimal criteria (e.g., reaching the optimal imaging plane) can include the ability to reach the target biopsy location and maintain a certain distance from all OARs. The processing component 134 can output instructions with visual indicators to the display 132 to provide feedback and / or suggestions to the clinician on how to manipulate the probe position in real time, as described below. Fig. 9 as shown in .

[0066] In some cases, the process of providing movement instructions (e.g., motion control configuration 820) and manipulating the probe can include multiple iterations. For example, the clinician can move the probe from a first position to a second position according to the instructions provided by the deep learning network 310. After the clinician moves the probe to the second position, the clinician can capture another set of images 302, and the deep learning network 310 may attempt to locate the biopsy target and needle path from the newly captured images 302. When the newly captured image 302 fails to provide an optimal view for guiding a biopsy (e.g., based on the malignant potential map 320 and confidence map 330 output by the deep learning network 310), the process of determining the motion control configuration 820 can be repeated, as shown by arrow 802.

[0067] Fig. 9 900 for guiding probe positioning according to some aspects of the present disclosure. The user interface 900 may be displayed on the display 132 or the user interface 360. The user interface 900 includes a user interface 900 for guiding probe positioning when the probe (e.g., the probe 110) is positioned in the initial imaging position (e.g., the probe position 810). (1) ). The first image view 910 may be suboptimal due to the proximity of an OAR 912 (e.g., OAR 410) to a potential malignant tissue location 914 (e.g., potential malignant tissue location 420). The potential malignant tissue location 914 is shown as a closed surface shape. The OAR 912 is shown as a closed surface shape and an open surface shape.

[0068] The user interface 900 may include a suggestion to rotate the probe about the probe's primary handle axis as indicated by visual indicator 902. The clinician may follow the visual indicator 902 (e.g., from the probe position 810 (1) To probe position 810 (2) ) Rotate the probe. After the clinician rotates the probe, one or more images (e.g., image 302) can be captured. The deep learning network 310 can be applied to generate another malignancy potential map, as shown in the second image view 920.

[0069] In some embodiments, the user interface 900 can indicate suboptimality by including a red dot 930 next to the first optimal image view 910 and a green dot 932 next to the second optimal image view 920. In some other cases, the user interface 900 can use visual indicators of any suitable shape and / or color to indicate suboptimality and / or optimality of an image view. After obtaining the second optimal image view 920, the user interface 900 can show a suggested needle path 940 (e.g., biopsy needle paths 350, 610, and 940) in the second optimal image view 920.

[0070] Fig.10 1 is a schematic diagram illustrating a configuration 1000 of a deep learning network 310 according to some aspects of the present disclosure. The deep learning network 310 may include one or more convolutional neural networks (CNNs) 1010. The CNNs 1010 may operate on one or more ultrasound images 1002 similar to the image 302. For example, the image 1002 may include a combination of one or more of image A captured using B-mode and image B captured using a strain imaging mode, image C 302 captured using a shear / Young's modulus imaging mode, and image D 302 in TDI mode.

[0071] CNN 1010 may include a set of N convolutional layers 1020 followed by a set of K fully connected layers 1030, where N and K may be any positive integers. The values ​​N and K may vary depending on the embodiment. In some cases, the values ​​N and K may depend on the size of the training and / or validation datasets (e.g., image dataset 140). Each convolutional layer 1020 may include a set of filters 1022 configured to extract features from an input image. Although in Fig.10 1010 . Not shown, but in some embodiments, the convolutional layers 1020 can be interleaved with spatial pooling layers, each of which includes a set of downsampling operations that can reduce the dimensionality of the extracted imaging features. In addition, each convolutional layer 512 can include a nonlinear function configured to extract a rectified feature map (e.g., including a rectified nonlinear (ReLU) operation). In addition, each convolutional layer 1020 can include a batch normalization function to accelerate the training of the CNN 1010. The fully connected layers 1030 can be nonlinear and can gradually shrink the last convolutional layer 1020. (N) to produce output 1050.

[0072] In one embodiment, CNN 1010 is trained to predict the probability of malignancy for each pixel on an input image (e.g., images 302 and 1002). In one embodiment, CNN 1010 is trained to predict a biopsy target (e.g., target biopsy location 340 and biopsy target 620) based on an input image and / or a biopsy needle path (e.g., biopsy needle path 350, 610, and 940) from the input image to the biopsy target. In one embodiment, CNN 1010 is trained to predict a motion control configuration (e.g., motion control configuration 820) for moving a probe (e.g., probe 110) from a current position to an optimal imaging position to guide a biopsy given an input image captured by the probe when the probe is at the current position. Training of CNN 1010 is described in more detail herein.

[0073] Although CNN 1010 is illustrated as a fully convolutional neural network (FCN), in some embodiments, CNN 1010 can be configured as an encoder-decoder type network (e.g., a U-net architecture) or any other suitable learning-based prediction network to achieve similar functionality.

[0074] Fig.11 1 is a schematic diagram illustrating a deep learning network training scheme 1100 according to some aspects of the present disclosure. The scheme 1100 is implemented by the system 100. The scheme 1100 illustrates the training of the deep learning network 310 to predict the probability of malignancy for each pixel on the input image 302. In order to train the deep learning network 310, an annotated or labeled ultrasound image 1110 of a tissue (e.g., including a thyroid gland) can be input to the deep learning network 310. The annotated ultrasound image 1110 can include one or more biopsy locations 1120 where a biopsy is performed. Each biopsy location 1120 is shown as a closed surface shape. A malignant certainty can be assigned to each biopsy location 1120 based on a corresponding pathology result 1150 obtained at the biopsy location 1120. The pathology result 1150 can indicate whether the tissue sample collected at the biopsy location 1120 is malignant, benign, or other (e.g., uncertain). In other words, the annotated ultrasound image 1110 includes a spatial malignancy certainty map for the underlying tissue captured by the image 1110.

[0075] The biopsy location 1120 is a discrete tissue location, and the pathology result 1150 is a discrete result for the corresponding biopsy location 1120. In one embodiment, a 2D Gaussian function 1140 can be applied to the pathology result 1150 at each biopsy location 1120 to provide malignant possibility information with a higher spatial dimension. The peak 1142 of the Gaussian function 1140 can be mapped to the biopsy location 1120 where the biopsy sample was obtained. For example, the pathology result 1150 for the biopsy location 1120a is malignant. After the Gaussian function 1140 is applied to the biopsy location 1120a, there is the highest malignant certainty (e.g., the peak 1142 of the Gaussian function 1140) at the biopsy location 1120, and the malignant certainty decreases as the distance from the biopsy location 1120a increases. The rate at which the malignant certainty decreases can depend on the variance of the Gaussian function 1140. Depending on the size of the training data (e.g., training data 140), the variance of the Gaussian function 1140 can be more conservative or less conservative. For example, when there is a smaller amount of training data (e.g., training data 140), the Gaussian variance can be more conservative, but the resulting data labels can have greater certainty. On the other hand, when there is a large amount of training data, the Gaussian variance may be less conservative, but the resulting data labels may have less certainty. Typically, the variance of the Gaussian function 1140 can be variable, for example, depending on the biopsy location 1120. For example, the correlation between the pathological samples acquired at a given biopsy location 1120 and the corresponding ultrasound imaging features can be learned to estimate the optimal variance of the Gaussian function 1140 for the given biopsy location 1120. In some cases, the variance of the Gaussian function 1140 can be defined by the user during the training phase.

[0076] During training, the deep learning network 310 can be applied to the image 1110, for example, using forward propagation, to obtain an output or score for each pixel on the image 1110, wherein the score can indicate the certainty or probability of malignancy for the pixel. The coefficients for the filters 1022 in the convolutional layer 1020 and the weights in the fully connected layer 1030 can be adjusted, for example, by using back propagation to minimize the output error. In other words, the coefficients in the convolutional layer 1020 and / or the weights in the fully connected layer 1030 can be adjusted to match the pathology results 1150 at each of the biopsy locations 1120.

[0077] In one embodiment, the annotated ultrasound image 1140 may also include annotations or labels for the OARs 1130 (e.g., OARs 410 and 912). Similarly, the deep learning network 310 can be applied to the image 1110, for example, using forward propagation, to obtain an output or score for each pixel on the image 1110, wherein the score can indicate whether each pixel on the image 1110 corresponds to a key region. The coefficients in the convolutional layer 1020 and / or the weights in the fully connected layer 1030 can be adjusted to identify the OAR at the location of the annotated OAR 1130.

[0078] In one embodiment, the system 100 can be used to capture images of test tissue with malignant tumors and generate annotations for the images based on biopsy results. The system 100 can save annotated images similar to the image 1110 as a training data set 140 in the memory 138. By using the training data set 140, the scheme 1100 can be applied to train the deep learning network 310.

[0079] In one embodiment, the deep learning network 310 may be trained using a training mechanism similar to the training mechanism in protocol 1100 to predict a biopsy target (e.g., target biopsy location 340 and biopsy target 620) from an input image (e.g., images 302 and 1002) and / or predict a biopsy needle path (e.g., biopsy needle paths 350, 610, and 940) to reach the predicted biopsy target from an input image. In such an embodiment, during training, the deep learning network 310 may be input with images annotated with OARs (e.g., OARs 410 and 1130), biopsy locations (e.g., biopsy locations 340, 913, and 1120), and biopsy needle paths (e.g., biopsy needle paths 350, 610, and 940) for reaching the corresponding biopsy locations.

[0080] In one embodiment, a deep learning network 310 can be trained using a training mechanism similar to the training mechanism in scheme 1100 to predict a motion control configuration (e.g., motion control configuration 820) for moving a probe (e.g., probe 110) from a current position to an optimal imaging position to guide a biopsy given an input image captured by the probe when the probe is in the current position. In such an embodiment, during training, images of certain imaging planes can be input to the deep learning network 310, each imaging plane being associated with one or more motion control configurations for manipulating the probe from the imaging plane of the corresponding image to the optimal imaging plane for guiding a biopsy to a biopsy location (e.g., target biopsy location 340 and target biopsy 620). In one embodiment, the deep learning network 310 can be trained based on ultrasound images captured from multiple patients and / or multiple subjects. The deep learning network 310 can be trained using population-based training on the geometry of the thyroid gland (e.g., thyroid gland 210) and surrounding key structures or OARs to suggest optimal imaging planes.

[0081] In some embodiments, the deep learning network 310 may include multiple CNNs 1010, for example, a first CNN 1010 may be trained to predict the probability of malignancy for each pixel of an input image (e.g., images 302 and 1002) (e.g., malignancy potential maps 320 and 400 and confidence maps 330 and 500), a second CNN 1010 may be trained to predict a target biopsy location (e.g., target biopsy location 340 and 620) and / or a biopsy needle path (e.g., biopsy needle paths 350, 610, and 940), and a third CNN 1010 may be trained to predict a motion control configuration (e.g., motion control configuration 820) for manipulating a probe (e.g., probe 110) to reach an optimal imaging view (e.g., imaging view 920) for guiding a biopsy.

[0082] Fig.12 1 is a schematic diagram illustrating a deep learning network fine-tuning scheme 1200 according to some aspects of the present disclosure. The scheme 1200 is implemented by the system 100. The scheme 1200 can be performed after obtaining a pathology result 1220 (e.g., pathology result 1150) from a biopsy 1210 guided by the scheme 300. For example, a clinician can employ the deep learning network 310 in the scheme 300 to determine a target biopsy location 340. The clinician can perform a biopsy 1210 to collect a tissue sample at the target biopsy location 340. Subsequently, a pathology result 1220 can be obtained from the tissue collected at the target biopsy location 340. The target biopsy location 340 and the corresponding pathology result 1220 can be fed back to the deep learning network 310 to fine-tune the deep learning network 310. The fine-tuning can include adjusting coefficients and / or weights at the convolutional layer 1020 and / or the fully connected layer 1030.

[0083] In some cases, the clinician may select a biopsy location different from the biopsy location 340 suggested by the deep learning network 310. The biopsy location selected by the clinician and the corresponding pathology results can also be fed into the deep learning network 310 to fine-tune the deep learning network 310.

[0084] For example, after each biopsy and when pathology results are available, fine-tuning can be a continuous process. Thus, fine-tuning can improve the predictive performance of the deep learning network 310.

[0085] Similarly, fine-tuning can be applied to needle path predictions and / or imaging probe positioning predictions. For example, the biopsy needle path used to reach a target biopsy location for a biopsy and / or the movement used to steer the probe to the optimal imaging plane for guiding a biopsy can be fed back to the deep learning network 310 for fine-tuning.

[0086] Fig.13 1 is a flowchart of a method 1300 for providing a user warning in a biopsy inspection according to some aspects of the present disclosure. The steps of method 1300 can be run by a computing device (e.g., a processor, a processing circuit, and / or other suitable components) of a host (e.g., host 130). As shown, method 1300 includes a plurality of enumerated steps, but embodiments of method 1300 may include additional steps before, after, and between the enumerated steps. In some embodiments, one or more of the enumerated steps may be omitted or performed in a different order. As described above, a clinician may determine a target biopsy location (e.g., target biopsy location 340 and 620) for performing a biopsy inspection on an anatomical structure (e.g., thyroid gland) of a patient based on a malignant potential map (e.g., malignant potential map 320 and 400) and a corresponding confidence map (e.g., confidence map 330 and 500). In some cases, a clinician may select a biopsy needle insertion point and / or a biopsy needle path to reach the target biopsy location. The method 1300 can be used to provide a clinician with a warning of intersections with critical areas of a patient's anatomy (eg, OARs 410 and 912).

[0087] At step 1310 , method 1300 includes receiving an ultrasound image (eg, images 302 and 1002 ) representing an anatomical structure of a patient (eg, target 105 ) and an insertion of a biopsy needle.

[0088] At step 1320 , method 1300 includes determining a possible trajectory of the biopsy needle that may collide with a critical area of ​​the patient's anatomy.

[0089] At step 1330 , method 1300 includes detecting that the trajectory of the biopsy needle is along one of the determined trajectories that is likely to collide with a critical region of the patient's anatomy.

[0090] At step 1340, method 1300 includes displaying a warning to the clinician based on the detection on a display (eg, display 132). For example, the warning can be in the form of a red light or any other suitable visual indicator on the display.

[0091] In one embodiment, method 1300 can display all determined trajectories that may collide with critical areas of the patient's anatomy on a display (e.g., display 132). When the trajectory of the biopsy needle is detected along one of the trajectories, method 1300 can highlight the detected trajectory and request the clinician to change the location of the entry point. Thus, method 1300 can be used to change the user-selected biopsy needle path so that a biopsy can be performed safely.

[0092] Fig.14 1400 is a flow chart of a method 1400 for guiding biopsy according to some aspects of the present disclosure. The steps of method 1400 can be executed by a computing device (e.g., a processor, processing circuitry, and / or other suitable components) of an ultrasound imaging probe (e.g., probe 110) or a host (e.g., host 130). Method 1400 can be implemented in the same manner as schemes 300, 800, 1100, and 1200, configuration 1000, and respectively related to Figure 3 , Figure 8 , Fig.11 , Fig.12 and Fig.10 The method 1400 described herein is similar to the method 1300 described herein. As shown, the method 1400 includes a plurality of enumerated steps, but embodiments of the method 1400 may include additional steps before, after, and between the enumerated steps. In some embodiments, one or more of the enumerated steps may be omitted or performed in a different order.

[0093] At step 1410 , method 1400 includes receiving a first ultrasound image (eg, images 302 and 1002 ) of an anatomical structure of a subject (eg, target 105 ) from an ultrasound imaging device (eg, probe 110 ).

[0094] At step 1420 , method 1400 includes applying a prediction network (e.g., deep learning network 310 ) to the first ultrasound image to identify a plurality of potential malignant locations (e.g., locations 420 and 914 ) in the anatomical structure of the subject and determining a plurality of malignancy likelihoods at the plurality of potential malignant locations.

[0095] At step 1430 , method 1400 includes displaying, via a display (eg, display 132 ), a plurality of potential malignant locations and a plurality of malignancy probabilities for the plurality of potential malignant locations to guide a biopsy determination of the anatomical structure of the subject.

[0096] In one embodiment, the prediction network can be applied to identify critical regions of the subject's anatomical structure to avoid biopsy (e.g., OARs 410 and 912), and determine multiple confidence levels for multiple malignancy probabilities at multiple potential malignant locations. The display may include a first map (e.g., malignancy potential maps 320, 400, and 600) and a second map (e.g., confidence maps 330 and 500). The first map may include an overlay of multiple potential malignant locations, multiple malignancy probabilities, and indications of the critical regions on an instance (e.g., a copy or version) of a first ultrasound image (e.g., as a Figure 4 The second graph may include an overlay of indications of a plurality of confidence levels corresponding to a plurality of potentially malignant locations on different instances (e.g., different copies or different versions) of the first ultrasound image (e.g., as shown in FIG. 1 ). Figure 5 In some embodiments, the first and second images can be combined on the same instance of the first ultrasound image (e.g., using alpha-blending and / or other suitable image processing) to include an overlay of indications of multiple potential malignant locations, multiple malignancy likelihoods, critical areas, and multiple confidence levels on the first ultrasound image.

[0097] In one embodiment, method 1400 may include determining a target biopsy location (e.g., target biopsy location 340 and 620) for performing a biopsy on an anatomical structure of the subject based on at least one of a plurality of malignancy probabilities at a plurality of potential malignancy locations or a plurality of confidence levels for the plurality of malignancy probabilities. Method 1400 may include determining a biopsy needle path (e.g., biopsy needle paths 350, 610, and 940) for performing a biopsy based on at least one of a plurality of potential malignancy locations, a critical area, or a target biopsy location of the anatomical structure of the subject. Method 1400 may include displaying, via a display, the determined biopsy needle path as an overlay on the first image (e.g., as an image of a first image). Figure 6 shown).

[0098] In one embodiment, method 1400 may include determining that the critical area is along a trajectory path of a biopsy needle directed toward an anatomical structure of the subject. Method 1400 may include displaying an indication (e.g., a warning) of a potential collision between the biopsy needle and the critical area via a display. In one embodiment, the trajectory path of the biopsy needle can be obtained from a second ultrasound image of the anatomical structure of the subject with the biopsy needle inserted. In one embodiment, the trajectory path of the biopsy needle can be obtained based on camera tracking, optical tracking, electromagnetic (EM) tracking, or any suitable tracking in an ultrasound image before the biopsy needle is inserted (e.g., after a user positions the biopsy needle on the patient's skin, but before penetrating the patient's skin).

[0099] In one embodiment, a first ultrasound image is received while the ultrasound imaging device is positioned at a first imaging position relative to an anatomical structure of the subject. Method 1400 can include applying a prediction network to the first ultrasound image to determine a motion control configuration (e.g., motion control configuration 820) for repositioning the ultrasound imaging device from the first imaging position to a second imaging position for performing a biopsy. Method 1400 can include displaying a visual indicator (e.g., visual indicator 902) via a display for repositioning the ultrasound imaging device based on the motion control configuration.

[0100] Various aspects of the present disclosure can provide several benefits. For example, the use of deep learning to automatically identify potential malignant locations and associated malignant possibilities and corresponding confidence levels and displaying malignant potential maps and confidence maps can assist clinicians in determining the need for biopsy and / or target biopsy locations. In addition, the use of deep learning to automatically suggest the optimal biopsy needle path, display the suggested biopsy needle path and / or display warning indications can guide clinicians to perform biopsies to safely reach the target biopsy location. In addition, the use of deep learning to automatically suggest probe repositioning and display suggested movements can guide clinicians to the optimal imaging view for performing biopsies. The disclosed embodiments can allow clinicians to perform successful biopsies without making many attempts (e.g., to reach clinically permitted attempts), thereby also reducing patient trauma. Although the disclosed embodiments are described in the context of thyroid biopsies, the disclosed embodiments can also be applied to guide biopsies in other parts of the body.

[0101] Those skilled in the art will recognize that the above-described apparatus, systems, and methods can be modified in various ways. Therefore, those of ordinary skill in the art will appreciate that the embodiments covered by the present disclosure are not limited to the specific exemplary embodiments described above. In this regard, although exemplary embodiments have been shown and described, various modifications, changes, and substitutions are contemplated in the foregoing disclosure. It should be understood that such changes may be made to the foregoing without departing from the scope of the present disclosure. Therefore, it is appropriate to interpret the claims broadly in a manner consistent with the present disclosure.

Claims

1. An ultrasonic imaging system, comprising: a communication interface in communication with the ultrasound imaging device and configured to receive an ultrasound image of an anatomical structure of the subject; as well as a processor in communication with the communication interface and configured to: The prediction network is applied to the ultrasound image to perform the following operations: identifying a plurality of potentially malignant locations in the subject's anatomy; determining a plurality of malignancy possibilities at the plurality of potentially malignant locations; identifying critical areas of the subject's anatomy to avoid biopsy; and determining a plurality of confidence levels for the plurality of malignancy possibilities at the plurality of potentially malignant locations; and An overlay of the plurality of potentially malignant locations, the plurality of malignancy possibilities, the plurality of confidence levels, and an indication of the critical region on the ultrasound image is output to a display in communication with the processor to guide a biopsy determination of an anatomical structure of the subject.

2. The system according to claim 1, wherein: The ultrasound image includes at least one of: brightness mode (B-mode) information, strain information, elasticity information, or tissue Doppler information.

3. The system according to claim 1, further comprising: The display is configured to display: a first map comprising an overlay of the plurality of potentially malignant locations, the plurality of malignancy probabilities, and an indication of the critical region on a first instance of the ultrasound image; as well as A second graph includes an overlay of indications of the plurality of confidence levels at the plurality of potentially malignant locations on a second instance of the ultrasound image.

4. The system according to claim 1, wherein: The processor is further configured to: A biopsy needle path for biopsy of the anatomical structure of the subject is determined based on at least one of the plurality of potentially malignant locations in the anatomical structure of the subject, the critical area, or a target biopsy location.

5. The system according to claim 4, wherein: The processor is further configured to: determining a target biopsy location for an anatomical structure of the subject based on at least one of the plurality of malignancy probabilities or the plurality of confidence levels for the plurality of malignancy probabilities at the plurality of potential malignant locations, and Wherein, the biopsy needle path is also determined based on the determined target biopsy position.

6. The system according to claim 4, further comprising: A user interface is in communication with the processor and is configured to receive a selection of the target biopsy location.

7. The system according to claim 4, wherein: The processor is further configured to: The determined biopsy needle path is output to the display as an overlay on the ultrasound image.

8. The system according to claim 4, wherein: The processor is also configured to apply the prediction network to the ultrasound image to determine the biopsy needle path.

9. The system according to claim 4, in, The processor is further configured to: determining that the critical area is along a trajectory path of a biopsy needle directed toward an anatomical structure of the subject; and An indication of a potential collision between the biopsy needle and the critical area is output to the display.

10. The system according to claim 4, wherein: The ultrasound image is received when the ultrasound imaging device is positioned at a first imaging position relative to an anatomical structure of the subject, and wherein the processor is further configured to: applying the prediction network to the ultrasound image to determine a motion control configuration for repositioning the ultrasound imaging device from the first imaging position to a second imaging position to perform the biopsy; and A visual indicator for repositioning the ultrasound imaging device based on the motion control configuration is output to the display.

11. The system according to claim 1, wherein: The prediction network is trained by the following operations: providing a plurality of test ultrasound images of an anatomical structure of a test subject, biopsy locations of biopsies performed on the anatomical structure of the test subject, and pathological results of the biopsies performed at the biopsy locations; and Scores are assigned to pixels of the plurality of test ultrasound images based on the biopsy locations and the pathology results.

12. The system of claim 1, wherein: The processor is further configured to: The prediction network is updated based on at least one of the following: a target biopsy location for a biopsy determined based on the plurality of potential malignant locations and the plurality of malignancy possibilities; or Pathological results of the biopsy.

13. The system of claim 1, wherein: The subject's anatomical structure includes at least a portion of a thyroid gland.

14. A computer program product comprising a computer program which, when executed by a computer, causes the computer to: receiving an ultrasound image of an anatomical structure of a subject from an ultrasound imaging device; The prediction network is applied to the ultrasound image to perform the following operations: identifying a plurality of potentially malignant locations in the subject's anatomy; determining a plurality of malignancy possibilities at the plurality of potentially malignant locations; identifying critical areas of the subject's anatomy to avoid biopsy; and determining a plurality of confidence levels for the plurality of malignancy possibilities at the plurality of potentially malignant locations; and An overlay of the multiple potential malignant locations, the multiple malignancy possibilities, the multiple confidence levels, or indications of the critical areas on the ultrasound image is displayed via a display to guide a biopsy determination of the anatomical structure of the subject.

15. The computer program product according to claim 14, in, The display includes: displaying a first map including an overlay of the plurality of potentially malignant locations, the plurality of malignancy probabilities, and an indication of the critical region on a first instance of the ultrasound image; and A second graph is displayed, the second graph including an overlay of indications of the plurality of confidence levels corresponding to the plurality of potentially malignant locations on a second instance of the ultrasound image.

16. The computer program product of claim 15, wherein: The computer program, when executed by the computer, further causes the computer to perform the following operations: determining a target biopsy location for biopsy of an anatomical structure of the subject based on at least one of the plurality of malignancy probabilities or the plurality of confidence levels for the plurality of malignancy probabilities at the plurality of potential malignant locations; determining a biopsy needle path for performing the biopsy based on at least one of the plurality of potentially malignant locations of the subject's anatomy, the critical area, or the target biopsy location; and The determined biopsy needle path is displayed by the display as an overlay on the first map.

17. The computer program product of claim 16, wherein: The computer program, when executed by the computer, further causes the computer to perform the following operations: determining that the critical area is along a trajectory path of a biopsy needle directed toward an anatomical structure of the subject; and An indication of a potential collision between the biopsy needle and the critical area is displayed by the display.

18. The computer program product of claim 16, wherein: receiving the ultrasound image when the ultrasound imaging device is positioned at a first imaging position relative to an anatomical structure of the subject, and wherein the computer program, when executed by the computer, further causes the computer to: applying the prediction network to the ultrasound image to determine a motion control configuration for repositioning the ultrasound imaging device from the first imaging position to a second imaging position to perform the biopsy; and Visual indicators for repositioning the ultrasound imaging device based on the motion control configuration are displayed via the display.

Images