Displaying blood vessel in ultrasound image

Abstract

To provide an ultrasonic system for identifying blood vessels in ultrasonic images and displaying the blood vessels in the ultrasonic images.SOLUTION: An ultrasonic system is configured to execute the steps of: receiving ultrasound images that include a blood vessel; determining, with a neural network implemented at least partially in hardware of the ultrasonic system, diameters of the blood vessel; determining a movement quantity of un ultrasound probe while an ultrasonic image is generated; comparing the movement quantity with a threshold movement value; determining a current diameter of the blood vessel on the basis of the diameter; selecting a color corresponding to a change in shape of the blood vessel on the basis of the current diameter of the blood vessel; and displaying the blood vessel with a colored indicator in one of the ultrasound images.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] Related Applications This application claims the benefit of priority to U.S. Non-Provisional Application No. 17 / 239,335, entitled “Identifying Blood Vessels in Ultrasound Images,” filed on April 23, 2021, the disclosure of which is incorporated by reference in its entirety into this application.

[0002] One or more exemplary embodiments relate to ultrasound machines and methods of operation, and more particularly, to an ultrasound machine that identifies blood vessels in an ultrasound image and enhances the display of the blood vessels by including information about the blood vessels that is useful to an operator of the ultrasound machine. [Background technology]

[0003] An ultrasound system emits ultrasound signals generated from an ultrasound probe into an object, such as a patient, and receives echo signals reflected from the interior of the object. An image of the interior of the object is generated using the received echo signals. More specifically, an ultrasound diagnostic machine generates an ultrasound image by using ultrasound image data acquired from the ultrasound probe, and displays the generated ultrasound image on a screen to provide the ultrasound image to a user. The ultrasound machine may include a control panel for controlling the ultrasound machine to set various functions, such as gain or frequency settings.

[0004] Procedures for which ultrasound machines are often used include ultrasound-guided insertion of interventional devices, such as peripheral intravenous (PIV) catheterization. In performing ultrasound-guided PIV catheterization, clinicians may first use an ultrasound machine to perform a survey scan of the body to locate a suitable vein for cannulation. A suitable vein has the following characteristics: generally greater than 0.3 mm in diameter; generally not located near an artery that may be inadvertently damaged during cannulation; large enough so that the catheter-to-vein diameter ratio is greater than 0.45 (although other rules of thumb exist, such as 0.33); the vein is relatively straight and not tortuous; and the vein does not contain a valve against which the catheter will impinge.

[0005] To determine the diameter and depth of a vein in a subject, a clinician may stare (e.g., estimate) the measurement, which is prone to error, or may turn on calipers and measure to obtain a more accurate diameter and depth. In practice, clinicians rarely use calipers because it is time consuming and they do not want to touch the ultrasound machine while performing this procedure (for both sterility and workflow reasons). Thus, clinicians need to be able to easily see and identify suitable blood vessels for inserting catheters, needles, etc. Summary of the Invention

[0006] In some embodiments, an ultrasound system is provided that includes a memory and at least one processor coupled to the memory, the at least one processor configured to perform the following steps: receive ultrasound images including a blood vessel; determine a diameter of the blood vessel in the ultrasound images using a neural network implemented at least in part in hardware of the ultrasound system, the diameter including a first diameter of the blood vessel in a first one of the ultrasound images and a second diameter of the blood vessel in a second one of the ultrasound images, the first ultrasound image corresponding to a first probe pressure on the blood vessel and the second ultrasound image corresponding to a second probe pressure on the blood vessel that is different from the first probe pressure; determine an amount of movement of the ultrasound probe between generating the first ultrasound image and the second ultrasound image; comparing the amount of movement to a threshold movement value; determine a current diameter of the blood vessel based on the first and second diameters, the current diameter of the blood vessel including applying one of a first function and a second function to the first and second diameters based on the comparison; select a color corresponding to a change in shape of the blood vessel based on the current diameter of the blood vessel; and display the blood vessel with a colored indicator in one of the ultrasound images.

[0007] In some embodiments, an ultrasound system is provided that includes a memory and at least one processor coupled to the memory, the at least one processor configured to perform the steps of receiving ultrasound images including a plurality of blood vessels; determining a diameter and a position of the plurality of blood vessels in a subset of the ultrasound images using a neural network implemented at least in part in hardware of the ultrasound system; determining a missed detection for a first blood vessel of the plurality of blood vessels in a first ultrasound image of one of the ultrasound images not included in the subset, the missed detection indicating an absence of a diameter or position determined by the neural network for the first blood vessel in the first ultrasound image; and in response to determining the missed detection, displaying an indication of the diameter and position of the first blood vessel in the first ultrasound image, the indication being based on the diameter and position determined by the neural network for the subset of the ultrasound images.

[0008] In some embodiments, an ultrasound system includes a memory and at least one processor coupled to the memory, the at least one processor including: receiving ultrasound images including a blood vessel while an ultrasound probe is moving, the ultrasound images being based on ultrasound echo signals received by the ultrasound probe while moving; determining a first ultrasound image of the ultrasound images based on a position of the ultrasound probe relative to the blood vessel when the ultrasound images are received, the first ultrasound image including an indication of a first probe pressure of the blood vessel having a first diameter; determining characteristics for the blood vessel in a further ultrasound image of the ultrasound images including a second diameter; and performing ultrasound processing between generating the first ultrasound image and the further ultrasound image. An ultrasound system is provided that is configured to perform the steps of determining an amount of movement of an ultrasonic probe, comparing the amount of movement to a threshold movement value, determining a first diameter and a second diameter and a current diameter of the blood vessel based on the comparison, and displaying an indicator of a characteristic of the blood vessel in a further one of the ultrasound images, where the indicator in the further ultrasound image indicates a second probe pressure for the blood vessel that is different from the first probe pressure, the indicator changes based on a change from the first probe pressure to the second probe pressure, and the indicator in the further ultrasound image includes a color indicative of the current diameter of the blood vessel, and where determining the current diameter of the blood vessel includes applying one of a first function and a second function to the first diameter and the second diameter based on the comparison, and the color corresponds to a change in shape of the blood vessel.

[0009] The present invention will be more fully understood from the detailed description given below of various embodiments of the invention and the accompanying drawings, which should not be construed as limiting the invention to the particular embodiments, but are for the purposes of illustration and understanding. [Brief description of the drawings]

[0010] [Figure 1A] Here is an example that detects two blood vessels and identifies them with boxes. [Figure 1B] Here is an example that detects two blood vessels and identifies them with boxes. [Figure 2A] 4 shows the detection of vein contours. [Figure 2B] 4 shows the detection of vein contours. [Diagram 3] 1 shows an example of an ellipse with intersecting diameters. [Figure 4A] 1 shows one image with no probe pressure applied. [Figure 4B] 1 shows one image where probe pressure (compression) was applied. [Figure 5A] 4 shows highlighting of detected veins. [Figure 5B] 4 shows highlighting of detected veins. [Figure 6] 1 shows an example of a vein diameter used for the calculation. [Figure 7] FIG. 1 illustrates a flow diagram of one embodiment of a process for displaying veins on an ultrasound image. [Figure 8] (A) shows an example of a process for displaying blood vessels. (B) shows an example of a process for displaying blood vessels. (C) shows an example of a process for displaying blood vessels. (D) shows an example of a process for displaying blood vessels. [Figure 9] FIG. 1 is a flow diagram of one embodiment of a process for manipulating the display of blood vessels (or other body structures) on a monitor. [Figure 10] (A) shows an embodiment of a process for displaying an image of a previous detection result in response to a sudden loss of a detection result, (B) shows an embodiment of a process for displaying an image of a previous detection result in response to a sudden loss of a detection result, and (C) shows an embodiment of a process for displaying an image of a previous detection result in response to a sudden loss of a detection result. [Figure 11A] 1 shows an example of highlighting of an ultrasound image produced when the probe is moving. [Figure 11B] 1 shows an example of highlighting of an ultrasound image produced when the probe is moving. [Figure 11C] 1 shows an example of highlighting of an ultrasound image produced when the probe is moving. [Figure 11D]1 shows an example of highlighting of an ultrasound image produced when the probe is moving. [Figure 12] 1 shows a flow diagram of one embodiment of a process for displaying one or more blood vessels (or other body structures) as detection results on a monitor along with at least one virtual detection result. [Figure 13A] Pressure from the probe is increased to a certain level, indicating a period during which the shape of the vessel changes from a normal shape (eg, oval) to another shape (eg, collapsed oval) (vessel collapse). [Figure 13B] Pressure from the probe is increased to a certain level, indicating a period during which the shape of the vessel changes from a normal shape (eg, oval) to another shape (eg, collapsed oval) (vessel collapse). [Figure 13C] Pressure from the probe is increased to a certain level, indicating a period during which the shape of the vessel changes from a normal shape (eg, oval) to another shape (eg, collapsed oval) (vessel collapse). [Figure 13D] Pressure from the probe is increased to a certain level, indicating a period during which the shape of the vessel changes from a normal shape (eg, oval) to another shape (eg, collapsed oval) (vessel collapse). [Figure 14A] Pressure from the probe is reduced from a particular level to a relaxed state, indicating the period during which the vessel shape changes from another shape back to its normal shape. [Figure 14B] Pressure from the probe is reduced from a particular level to a relaxed state, indicating the period during which the vessel shape changes from another shape back to its normal shape. [Figure 14C] Pressure from the probe is reduced from a particular level to a relaxed state, indicating the period during which the vessel shape changes from another shape back to its normal shape. [Figure 14D] Pressure from the probe is reduced from a particular level to a relaxed state, indicating the period during which the vessel shape changes from another shape back to its normal shape. [Figure 15] Included is a graph showing vessel diameter versus time. [Figure 16]1 illustrates a flow diagram of one embodiment of a process for using detection results from one time period to create detection results in another time period. [Figure 17] An example of an image is shown in which the best image of the detected blood vessels is superimposed on the image. [Figure 18A] 1 shows an example of guide information along with information on best probe location. [Figure 18B] 1 shows an example of guide information along with information on best probe location. [Figure 19A] 1 shows a flow diagram of one embodiment of a process for displaying blood vessels on an ultrasound image. [Figure 19B] 1 shows a flow diagram of another embodiment of a process for displaying blood vessels on an ultrasound image. [Figure 19C] 13 shows a flow diagram of yet another embodiment of a process for displaying blood vessels on an ultrasound image. [Figure 20] 13 shows a flow diagram of yet another embodiment of a process for displaying blood vessels on an ultrasound image. [Figure 21] 13 shows a flow diagram of a further embodiment of a process for displaying blood vessels on an ultrasound image. [Figure 22] FIG. 1 illustrates a block diagram of one embodiment of an ultrasound machine. [Figure 23] FIG. 1 illustrates an example of a handheld ultrasound machine. [Figure 24] FIG. 13 illustrates a data flow diagram for the vessel identification subsystem. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] In the following description, numerous details are set forth to provide a more thorough explanation of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the present invention.

[0012] overview Techniques are disclosed for identifying blood vessels in ultrasound images and displaying blood vessels in ultrasound images. In some embodiments, these techniques are performed by an ultrasound machine. Examples of such ultrasound machines are described in more detail below. For purposes of this specification, the terms "ultrasound machine," "ultrasound system," and "ultrasound imaging system" may be used interchangeably.

[0013] In some embodiments, the ultrasound machine executes detection software to perform vessel detection in areas of the ultrasound image, and the detection results are displayed on a monitor or display of the ultrasound machine, such as, for example, a clinical display or tablet coupled to the ultrasound machine. Execution can be by one or more processors or execution engines. In some embodiments, the ultrasound machine performs vessel detection using template matching, artificial intelligence (AI) or machine learning (e.g., adaptive boosting (adaboost) for processing sequential frame information, deep learning, supervised learning models, support vector machines (SVM), sequence models including recurrent neural networks (RNN), gated recurrent units (GRU), convolutional GRU (ConvGRU), long short-term memory (LSTM), etc.), and / or another suitable detection method. Additionally or alternatively, the ultrasound machine can execute AI algorithms and / or use neural networks to identify veins and arteries and locate them within the ultrasound image.

[0014] In some embodiments, after the detection software detects the blood vessel, the ultrasound machine displays the blood vessel on the ultrasound system monitor or display. In some embodiments, the ultrasound machine displays the detected blood vessel in a highlighted manner to provide information to an operator, e.g., a user of the ultrasound machine. For example, the ultrasound machine may paint an outline or other form of blood vessel representation (e.g., identifier) ​​around or near the blood vessel.

[0015] Displaying the vessels in an enhanced form provides additional information to the operator. For example, the outline of the vein may be changed to match the color coding of the largest catheter that can fit within that vein to the extent of the catheter to vein diameter ratio (e.g., 0.45, or 0.33). Thus, the operator may select a catheter size based on the vessel and their displayed indicia. Additionally or alternatively, the ultrasound machine may identify all veins within the ultrasound image that are suitable for a particular catheter size. For example, the operator may select a catheter size on the user interface of the ultrasound machine, and in response, the ultrasound machine may designate veins within the ultrasound image that are suitable for the catheter size (e.g., based on the ratio of catheter size to vein diameter) by changing the color of the indicia of the vein that is suitable for the catheter size, removing the indicia of the vein that is not suitable for the catheter size, a combination thereof, etc. In one example, the operator may touch a vein in the ultrasound image and have the ultrasound machine display the diameter and depth of that vessel. Additionally or alternatively, the ultrasound machine may automatically identify the most central and shallowest vein and automatically provide the diameter and depth of that vein on the display of the ultrasound machine.

[0016] In some embodiments, certain organization of arteries and veins can be detected as an integrated structure. For example, a triad is a vein-artery-vein collection where a central artery is closely bounded on both sides by veins. In one example, the AI ​​can clearly detect and classify the veins and arteries in the triad as individual components. However, due to the unique configuration of the triad structure, this grouping can be detected as an additional unique classification along the veins and arteries. By doing so, the overall accuracy of the detection can be increased because triads always occur in a vein-artery-vein configuration. A group of three closely detected vessels, for example, a vein, a vein, an artery, where the central vein is bounded by a vein and an artery, is likely to be misclassified or false positive. By detecting this group as a triad, the accurate classification of each vessel can be improved.

[0017] In some embodiments, the ultrasound machine calculates a likelihood value for each detected vessel as an indication of a confidence level associated with the detection result. For example, the likelihood value may include a value between 0 and 1 indicating a confidence level of the vessel's classification as an artery or vein. In some embodiments, the ultrasound machine generates a label (e.g., an outline, a circle, an oval, a box, etc.) of the vessel and adjusts the opacity of the label based on the likelihood value representing the confidence of the prediction of that vessel. This adjustment of opacity provides an improvement over conventional ultrasound systems that do not adjust the opacity of the label based on the confidence level of the label. In one example, a clinician or operator of the ultrasound machine is exposed to additional information that is useful in selecting a vessel for a medical procedure such as catheterization. Additionally or alternatively, the opacity adjustment makes the display more visually appealing and less cluttered, since, based on the change in opacity, the vessels gradually fade in and out over time, rather than suddenly appearing and disappearing.

[0018] In one example, the ultrasound machine tracks a blood vessel across multiple frames (e.g., a frame can represent one ultrasound image in a sequence of ultrasound images). For example, the ultrasound system can determine a blood vessel in one ultrasound image and track the blood vessel in subsequent ultrasound images in the sequence of ultrasound images. The tracking can be based on characteristics of the blood vessel, such as the location of the blood vessel, the diameter of the blood vessel, classification as a vein or artery, combinations thereof, etc. For example, the ultrasound machine can determine that a blood vessel in a first ultrasound image is the same blood vessel in a second ultrasound image based on one or more of the characteristics of the blood vessel in the first and second ultrasound images.

[0019] In some embodiments, the ultrasound machine determines when a vessel detection fails and generates a virtual detection result to display the vessel on the ultrasound image. For example, an ultrasound machine may detect a vessel in multiple ultrasound images and then fail to detect the vessel in a subsequent ultrasound image (e.g., an ultrasound image following multiple ultrasound images in a video sequence). The ultrasound machine may compare the detection results of the ultrasound images in the video sequence and declare a detection failure for the subsequent ultrasound image based on the comparison. Based on the detection failure, the ultrasound system may generate a virtual detection result, such as, for example, a bounding box, and display the virtual detection result in the subsequent ultrasound image to indicate the location of the vessel, even though the vessel was not detected in the subsequent ultrasound image.

[0020] In one example, the ultrasound machine determines a desired entry point for an interventional instrument. The ultrasound machine can then calculate the distance from the transducer face (e.g., the edge of the probe) to the desired entry point and display an indication of the distance on the ultrasound machine's display. For example, the ultrasound machine can display a message with text including the distance and / or an arrow indicating the direction to move the probe according to the distance.

[0021] Thus, the techniques disclosed herein provide for identifying blood vessels in an ultrasound image and displaying the blood vessels in some manner to convey useful information to an operator. Thus, the ultrasound systems disclosed herein may be suitable for medical procedures where traditional ultrasound systems are not suitable because they may result in undesirable outcomes for the patient including discomfort, blood loss from multiple punctures, risk of infection, etc.

[0022] It should be noted that while the description herein focuses on blood vessels, the techniques and systems disclosed herein are not limited to blood vessels and may be used with other body structures such as nerves, muscles, skeletal parts, etc. Additionally, while the description herein focuses on peripheral intravenous catheterization, the techniques and systems disclosed herein are not limited to catheters and may be used with any suitable interventional device, such as needles, stents, clamps, guides, etc.

[0023] Highlight The techniques and systems disclosed herein can detect blood vessels in ultrasound images and generate useful information about the blood vessels in various ways. In some embodiments, the ultrasound system determines the diameter of the blood vessel based on one or more previous ultrasound images. For example, the ultrasound system can determine the diameter of the blood vessel in each of multiple previous frames (e.g., the previous 2, 3, 4 or more frames) and generate the diameter of the blood vessel in the current frame based on the diameter of the blood vessel from the multiple previous frames. Thus, the ultrasound system can prevent undesirable fast changes in the vessel's indicia (e.g., the color of the vessel's bounding box, which may correspond to the catheter size), which allows the operator to understand the true size of the blood vessel and take appropriate action, such as selecting the appropriate size of the catheter.

[0024] In some embodiments, an ultrasound system generates ultrasound images including one or more blood vessels and determines a diameter of the blood vessels in the ultrasound images. The diameters may include a respective diameter of each blood vessel in each ultrasound image of the ultrasound images (e.g., an ultrasound video stream). The ultrasound system may include a neural network that determines a size and location of the blood vessels. The neural network may be implemented at least in part in hardware of a computing device (e.g., an ultrasound machine, a computing device coupled to an ultrasound machine such as a tablet, a combination thereof, etc.). The ultrasound system may calculate a blood vessel diameter based on the diameter of the blood vessel. Based on the blood vessel diameters of the same blood vessel in the multiple images, the ultrasound system may generate a blood vessel diameter for use in displaying the blood vessel. The ultrasound machine may select a color based on the blood vessel diameter, and then generate an indicator based on the blood vessel diameter and the selected color to provide an indication in at least one ultrasound image such that the blood vessel is displayed with an indicator having a color. FIGS. 1A-6 show an example of this process.

[0025] In one example, the process is initiated by a processor of an ultrasound machine executing an image analysis algorithm to perform image analysis on an ultrasound image, resulting in detection of blood vessels in the ultrasound image. The processor may execute two algorithms, one for detecting blood vessels and another for determining the type of detected blood vessel (e.g., whether the blood vessel is a vein or an artery). The algorithm may include one or more neural networks, such as a first neural network trained to detect the location of blood vessels and a second neural network trained to classify blood vessels as veins or arteries based on the detection results of the first neural network. Alternatively, these two neural networks may be combined into a single algorithm. In one example, a single neural network including a first head of detection and a second head of classification is included in a single algorithm, such that the processor may include and execute a single algorithm.

[0026] FIG. 1A shows an example of detecting two blood vessels and identifying them in boxes 101 and 102, respectively, with characteristics such as color, line width, line shape (e.g., solid or dashed). Boxes 101 and 102 are examples of bounding containers of the blood vessels of FIG. 1A, which may be generated in a processor of the ultrasound system using information from a neural network. The ultrasound system may then classify the blood vessels as veins or arteries, such as with additional neural networks. After classifying the blood vessels as veins or arteries, the ultrasound system may modify the characteristics of one or both of boxes 101 and 102 to serve as indicators of the classification. In the example of FIG. 1B, after the ultrasound system classifies both blood vessels as veins, the ultrasound machine modifies the color of boxes 101 and 102 in FIG. 1B (represented by a hash) compared to the color of boxes 101 and 102 in FIG. 1A.

[0027] In one example, the ultrasound system determines a contour of the detected blood vessel and approximates the determined contour as an ellipse. For example, the ultrasound system may match the ellipse to one or both of boxes 101 and 102. Additionally or alternatively, the ultrasound system may include implementation of binarization, contour detection, or segmentation techniques to determine the contour of the blood vessel.

[0028] 2A and 2B illustrate detection of blood vessel (e.g., vein) contours. Referring to FIG. 2A, two blood vessels 201 and 202 are shown. Binarization / edge detection contouring can be performed inside a box-shaped region (not shown) selected in the image of FIG. 2A. Additionally or alternatively, the ultrasound system can fit inner ellipses 211 and 212 to the box, as shown in FIG. 2B. In these examples, the ultrasound system generates an oval shape as the bounding vessel of the blood vessel (e.g., vein).

[0029] The ultrasound system can determine the diameter of the blood vessel using the ellipse generated as the bounding vessel. FIG. 3 shows the major and minor axes (together with the major / minor axes 301 and 302 of each of the two blood vessels). The ultrasound system can determine the diameter of one blood vessel from the major / minor axes 301 and the diameter of the other blood vessel from the major / minor axes 302. For example, the ultrasound system can average the major and minor axes of the major / minor axes 301 to determine the diameter of the blood vessel on the left side of FIG. 3 as the blood vessel diameter, and average the major and minor axes of the major / minor axes 302 to determine the diameter of the blood vessel on the right side of FIG. 3 as the blood vessel diameter. Additionally or alternatively, the ultrasound system can generate the diameter of the blood vessel as the blood vessel diameter from a weighted average of the major and minor axes of an ellipse (e.g., an ellipse) that represents the bounding vessel of the blood vessel. For example, the ultrasound system can weight one of the axes more heavily than the other when forming the average.

[0030] In one example, the ultrasound system determines the diameter of the blood vessel from the major and minor axes of the ellipse by matching the properties of the ellipse to the properties of a circle. For example, the ultrasound system can determine the diameter of the blood vessel as the diameter of a circle having the same area as an ellipse that represents the bounding vessel of the blood vessel. Let the major axis of the ellipse be d1 and the minor axis be d2. The ultrasound system determines the diameter d of the blood vessel as the diameter of a circle having the same area as the ellipse, i.e.,

number

number

[0031] Additionally or alternatively, the ultrasound system may determine the vessel diameter from the diameter of a circle having the same circumference as an ellipse representing the vessel's bounding vessel.

number

number

[0032] In some embodiments, the ultrasound system detects an artery and highlights and displays the artery using a different sign than the sign used to identify the vein. For example, the sign used for the artery may be a red circle with a red line across it to inform the operator not to select the artery for a procedure where the vein is desired. In some embodiments, after detecting the artery, the operator of the ultrasound machine may turn off the calculation of the artery diameter, and thus the calculation of the vessel diameter of the artery. Additionally or alternatively, the operator may enable or disable the highlighting of the vein or artery individually (e.g., one after the other in succession), such as by first disabling the vein or artery classification indicator, and then disabling the displayed diameter of the vein or artery.

[0033] In some embodiments, the ultrasound system applies a tracking technique to the detected blood vessel to determine whether the same blood vessel appears in multiple ultrasound images. For example, the tracking technique can use the diameter and location of the detected blood vessel. In such cases, the blood vessel diameter is pre-calculated before tracking. However, in other embodiments, tracking of the detected blood vessel is performed before calculating the blood vessel diameter. Note that the blood vessel diameter is calculated based on two intersection diameters, so it can be calculated before or after tracking, and tracking does not necessarily require the blood vessel diameter, as long as at least these two diameters can be calculated.

[0034] Also, by using the tracking technique, the current diameter value of the blood vessel can be calculated from the predetermined blood vessel diameter. For example, the ultrasound system can set the current blood vessel diameter of the blood vessel to the maximum value of the previously determined blood vessel diameters of the blood vessel that appeared in a number of previous ultrasound images, such as the previous three to five images. Since the tracking technique allows the determination of the same blood vessel that persists through a series of ultrasound images (frames) (e.g., ultrasound), in some embodiments, the maximum value is calculated for the same blood vessel. Figures 4A and 4B represent two images during pressure of the probe in a time series. In Figure 4A, two veins are present, vein 401 and vein 402. Due to pressure by the probe, vein 401 disappears in Figure 4B, and as a result, the ultrasound system identifies only vein 402 as the same vein between Figures 4A and 4B. Thus, the ultrasound system can determine the current value of the blood vessel diameter of vein 402 in Figure 4B from one or more values ​​of the blood vessel diameter of the identified vein 402 in a predetermined number of previous images, including Figure 4A. For example, if the vessel diameter of vein 402 in FIG. 4B is 1.5 mm and the maximum vessel diameter of vein 402 in a previous image (e.g., FIG. 4A) is 2.0 mm, the ultrasound system can set the current diameter of vein 402 in FIG. 4B to 2.0 mm. A new vein 403 is also shown in FIG. 4B and does not appear in FIG. 4A. Thus, the ultrasound system can determine the vessel diameter of vein 403 in FIG. 4B without using the vessel diameter of vein 403 in the previous image.

[0035] In some embodiments, tracking uses at least one or more types of information about points, lines, or areas of the vascular region. For example, in some embodiments, vascular points are tracked between frames, and the ultrasound system measures the movement of the points. In the case of points, information about the points may include the center, top, bottom, left, right, or centroid point of the vascular region, and the difference in the location of the same points between the two ultrasound images is evaluated. Tracking may use information other than point information. Examples of other information that can be used for tracking include information about lines (e.g., diameter of the vascular region, or contour length, etc.) and information about areas (e.g., shape, area, lighting information (texture information), etc.) of the vascular region. If the movement is within a threshold distance (e.g., less than 2 mm), the ultrasound system considers the vessels of the frames to be the same vessel. In some embodiments, the threshold distance is equal to a percentage (e.g., half) of the diameter of the vessel in the immediately preceding frame (or another previously generated frame), or the vessel diameter (e.g., the average of the diameters) of the vessels in a set of two or more previously generated consecutive frames.

[0036] In some embodiments, the ultrasound system displays the detected blood vessels superimposed with an enhancement, such as a color enhancement. The ultrasound system can determine the color for the color enhancement according to a diameter value determined by the ultrasound system for the blood vessel. Figures 5A and 5B show the same ultrasound image with different color enhancement on the detected vein on the display of the ultrasound system. The images were obtained, for example, under pressure by the probe. Specifically, Figure 5A shows a display of the detected vein when considering only the vessel diameter of the current image, and Figure 5B shows the detected vein with color enhancement according to the calculated vessel diameter according to some embodiments. Assume (i) that the calculated vessel diameter of veins 501A and 501B is 1.5 mm, and (ii) that the configuration of the enhancement rules in the ultrasound system is such that if the current value of the vessel diameter is 2.0 mm or more, the vein is surrounded with a color (e.g., pink), and if the value is less than 2.0 mm, the vein is surrounded with another color (e.g., orange). In FIG. 5A, vein 501A is highlighted in orange (represented by a solid white oval), whereas in FIG. 5B, vein 501B is highlighted in pink (represented by hatching). Vein 501B is surrounded in pink because, in one embodiment, the ultrasound system is configured to calculate the vessel diameter from a previous image that includes a maximum vessel diameter of 2.0 mm. In an image that includes such a maximum vessel diameter, the shape of a vein, e.g., the same as 501A, 501B, becomes more circular. Thus, the highlighting in FIG. 5B corresponds better to the change in the vessel shape than that in FIG. 5A. In other words, an embodiment such as FIG. 5B is useful, for example, when the probe releases and / or squeezes an object under ultrasound scanning such that the vessel shape changes gradually or suddenly.

[0037] In some embodiments, the vessel diameter is calculated for subsequent images (frames). In one example, the ultrasound system detects a vein and performs a calculation of the vessel diameter of the vein only for subsequent images (frames), and then highlights only the vein, thus facilitating understanding of the vessel characteristics in the ultrasound image. FIG. 6 is a diagram showing an example of the vessel diameter used for calculation. With reference to FIG. 6, the ultrasound system identifies a vein and calculates the diameter of the vein 402 for one frame (image) using the axes shown. The artery 403 has a diameter, and the vessel diameters of the vein 402 and the artery 403 are not calculated for subsequent frames.

[0038] In some embodiments, the ultrasound system uses past images (e.g., previously generated and displayed ultrasound images) to determine characteristics of the blood vessel, such as vein size. Using past images, the ultrasound system can determine when the blood vessel is in a normal state (e.g., not compressed by the probe). For example, the ultrasound system can determine an image from multiple ultrasound images that includes a blood vessel with a maximum diameter, and use the maximum diameter to determine the blood vessel diameter, and therefore select the blood vessel at its correct size when determining its blood vessel diameter. Additionally or alternatively, at least three past frames, or at least half the number of frames, can be used to determine when the blood vessel is at its correct size. However, more or less than these frames may be used. Thus, the ultrasound system can determine the actual (or true) value of the blood vessel diameter in its normal state, and therefore select the correct color (e.g., pink in FIG. 5B depending on the diameter) to designate the blood vessel. This can mitigate the effects of pressure from the probe.

[0039] 7 shows a flow diagram of one embodiment of a process for displaying veins on an ultrasound image. The process may be performed by processing logic, which may include hardware (e.g., circuitry, dedicated logic, etc.), software (such as that executed on a general purpose computer system or dedicated machine), firmware (e.g., software programmed into a read-only memory), or a combination thereof. In some embodiments, the process is performed by a processor of an ultrasound machine. Although FIG. 7 shows displaying veins on an ultrasound image, the process may additionally or alternatively be used to display arteries.

[0040] 7, the process begins by processing logic acquiring an image frame, such as an ultrasound image (processing block 701), and performing vein (or vessel) detection on the image frame (processing block 702). Using the results of the vein detection, processing logic extracts vein boundaries (processing block 703) and calculates vessel diameters of the detected veins (processing block 704).

[0041] After calculating the vessel diameter, processing logic performs tracking (processing block 705). During tracking, processing logic determines whether each detected vein was present in a previous frame (processing block 706). If not, the process transitions to processing block 708. If so, processing logic calculates the maximum vessel diameter of that vein based on its vessel diameter in the previous frame (processing block 707) as the current value of the vessel diameter and transitions to processing block 708.

[0042] At processing block 708, processing logic determines whether all veins have been tracked and whether a maximum vessel diameter has been calculated for those veins that have been tracked. If not, the process returns to processing block 705 and the process continues. If so, the process transitions to processing block 709 where processing logic displays each vein in the ultrasound image with a color or other indicator selected based on its maximum diameter.

[0043] In some embodiments, the current value of the vessel diameter is calculated in one of several different ways. For example, the current value of the vessel diameter of a detected vessel may be determined from the maximum, minimum, mean, median, mode, standard deviation, max-min from the vessel diameter of the same vein in the past few frames. In some embodiments, the ultrasound machine determines the calculation algorithm prior to the examination, or prior to imaging the vessel, or using any enhancement (e.g., label, color, etc.) of the vessel in the image. This determination may be part of the configuration or boot-up process of the ultrasound machine. In some embodiments, the calculation algorithm (e.g., maximum, minimum, mean, etc.) is selected by the operator of the ultrasound machine.

[0044] One advantage of selecting the standard deviation or max-min value is that a healthy vein can easily compress and recover during pressure by the probe, and if the vein is not in a healthy state, it is not very elastic. One advantage of selecting the mean, median, or mode is that it eliminates outliers that appear unintentionally or erroneously in one or more frames. Another advantage of selecting the mean, median, or mode is that it is a compromise between two points: (i) a smaller gauge catheter causes less damage to the blood vessel, but (ii) a smaller gauge catheter also transports less fluid. Furthermore, the mean, median, and mode are always less than the maximum diameter of the blood vessel, so that the medical instrument / instrument, such as, for example but not limited to, a needle or catheter, is smaller than the diameter of the blood vessel.

[0045] In one example, the processor is configured to automatically determine the calculation algorithm (e.g., without explicit user selection of the calculation algorithm). In some embodiments, the selection is based on whether the probe is compressing the subject being examined (e.g., as determined by a pressure sensor in the probe), thereby reducing the size of the vessel. In one such embodiment, (i) if the probe is pressing on the same part of the object, the processor selects a maximum calculation algorithm, and (ii) if the probe is moving along the vessel (e.g., longitudinally), the processor selects an average calculation algorithm. In some embodiments, to distinguish between (i) and (ii), the ultrasound machine performs image analysis to determine whether the perimeter of the detected vessel has changed. Additionally or alternatively, a position sensor in the probe (e.g., a magnetic sensor, an accelerometer, a gyro sensor, etc.) can be used to distinguish between (i) and (ii).

[0046] In some embodiments, the ultrasound machine marks the displayed blood vessels with a colored indicator (e.g., circle, oval, etc.). In such cases, in some embodiments, the color can be based on whether the detected blood vessel is an artery or a vein. For example, the ultrasound machine can mark arteries with a red indicator (e.g., a red circle, or a crossed circle icon). Additionally or alternatively, the ultrasound system can mark blood vessels with an indicator having a color based on what size (e.g., gauge) of needle or catheter may be inserted into the blood vessel. The color can follow a standard color chart that assigns unique colors to instrument gauges. Thus, in some embodiments, blood vessels having a different range of diameters (e.g., vessel diameters) are highlighted by using different colors. Additionally, in one example, only veins having a different range of diameters (e.g., vessel diameters) are highlighted by using a different highlighting color than that used for arteries, since in one configuration of the ultrasound system, blood vessel diameters are calculated for veins only.

[0047] As previously mentioned, in some embodiments, the ultrasound machine detects and displays some highlighting on the blood vessels regardless of which blood vessels are of interest to the operator of the ultrasound machine. For example, the highlighting on the blood vessels may include one or more highlighted blood vessels that the operator wishes to see along with one or more other highlighted blood vessels that the operator is not interested in seeing. The operator may input a desire to see or not see certain blood vessels via a user interface of the ultrasound machine.

[0048] In some embodiments, the operator may remove unwanted vessel highlighting from the display. For example, the operator may select a detected vessel to trigger highlighting on the detected vessel and toggle between an ON state in which the vessel is displayed with highlighting (e.g., bounding vessel, color, color based on type (e.g., vein or artery), color based on needle size that can be inserted into the vessel, text indicating diameter, icons such as stop signs, thumbs up, thumbs down, combinations thereof, etc.) and an OFF state in which the detected vessel is displayed without any highlighting.

[0049] 8A-8D show an example of a process for highlighting on blood vessels. Referring to FIG. 8A, blood vessels including vein 811, artery 812, and vein 813 are shown and highlighted on monitor 800. In some embodiments, monitor 800 includes a touch-sensitive screen display or touch surface display device. In one example, three blood vessels 811-813 are detected by artificial intelligence (AI). Additionally or alternatively, the detected blood vessels can be marked by AI with colored indicators (e.g., circles or ellipses). When an operator touches monitor 800 at the location of one of the three blood vessels, such as vein 813 in FIG. 8A, the colored circle disappears as shown in FIG. 8B. Furthermore, when an operator touches monitor 800 at the location of one of the blood vessels previously touched to make the circle disappear as shown in FIG. 8C, the circle reappears on the blood vessel as shown in FIG. 8D.

[0050] 9 shows a flow diagram of one embodiment of a process for manipulating the highlighting of blood vessels (or other body structures) on a monitor. The process can be performed by processing logic that can include hardware (e.g., circuitry, dedicated logic, etc.), software (such as that executed on a general-purpose computer system or dedicated machine), firmware (e.g., software programmed into a read-only memory), or a combination thereof. In some embodiments, the process is performed by a processor of the ultrasound machine.

[0051] 9, the process begins by processing logic displaying highlighting of one or more blood vessels (or other body structures) on an ultrasound machine monitor with a colored circle (or other display of other highlighting aspects) (processing block 901). In some embodiments, the monitor is a touch-sensitive screen type monitor or a touch surface display device.

[0052] After one or more vessels are highlighted, processing logic receives an indication that the user has touched the monitor (processing block 902). Additionally or alternatively, the user can make a selection on the monitor by moving a cursor with a cursor control device to a location on the monitor and pressing a select button.

[0053] In response to receiving an indication that a user has touched the monitor, processing logic determines the location of the touch and whether a blood vessel was highlighted at that location on the monitor when the touch was made (processing block 903). If so, processing logic causes the colored circle displayed with the blood vessel (or any other indicator / highlighting used in conjunction with the display of the blood vessel) to be removed from the display (processing block 904).

[0054] If a blood vessel was not displayed at that location on the monitor when the touch was made, processing logic determines whether a blood vessel has been detected at the location of the touch but its highlighting has been suppressed from the display (processing block 905), and if so, causes the colored circle (or other marking / highlighting) to be re-displayed on the monitor (processing block 906).

[0055] In some embodiments, the blood vessel is displayed on the monitor and the process of FIG. 9 can be repeated when the operator touches the monitor. For example, as the operator moves the ultrasound probe, new blood vessels may appear and previous blood vessels may disappear from view. The ultrasound machine can repeat the process of FIG. 9 for the newly displayed blood vessel when the operator touches the monitor. In one exemplary implementation of the ultrasound system, the ultrasound system continuously generates ultrasound images and continues the vessel detection process and tracking techniques (and vessel diameter calculation) on the generated images regardless of the on or off state of enhancement, so that when the operator touches a blood vessel that was not previously enhanced on the monitor, a circle of the appropriate color is displayed on the blood vessel, and the color is further based on a standard color corresponding to the calculated blood vessel diameter and / or catheter size.

[0056] In some embodiments, the enhancement to the ultrasound image is in the form of indicating blood vessels that were not detected. For example, if a blood vessel is detected in the ultrasound image, the detection result (e.g., a colored circle on the detected blood vessel in the ultrasound image) may suddenly disappear if the examination situation changes suddenly. For example, if the probe is suddenly pushed into the subject under examination, the blood vessel in the ultrasound image may suddenly collapse, and the detection result may no longer be displayed on the monitor. This lack of detection indication may be problematic for the user, since the user may still see the actual location of the collapsed blood vessel in the B-mode image.

[0057] In some embodiments, if the processing logic of the ultrasound machine cannot detect a blood vessel in the current image, the detection results of one or several previous ultrasound images are retrieved. The processing logic can determine a virtual detection result based on the detection results of one or more of the previous ultrasound images, and the virtual detection result can be displayed by the ultrasound machine superimposed on the current image. In one embodiment, the ultrasound machine displays only 2-3 frames of the virtual detection result to prevent flickering.

[0058] In some embodiments, the ultrasound machine determines the diameter and location of blood vessels within a subset of the ultrasound images. As previously described, this determination may be made using a neural network or other detection mechanism implemented at least in part in the hardware of a computing device, such as the ultrasound machine itself, a computing device coupled to the ultrasound machine, or a combination thereof. Alternatively or additionally, the determination may be made using hardware (e.g., circuitry, dedicated logic, etc.), software (such as that executed on a general purpose computer system or a dedicated machine), firmware (e.g., software programmed into a read-only memory), or a combination thereof.

[0059] Based on the diameter and location of the blood vessels seen in each image, the ultrasound machine can determine whether a missed detection occurred for one of the blood vessels in one of the ultrasound images because the blood vessel was not present in the blood vessels identified by the neural network (or other detection mechanism). In response to that determination, the ultrasound machine can attempt to track a blood vessel previously detected by an alternative means and display an indication of the diameter and location of one of the blood vessels in one of the ultrasound images. In some embodiments, the indication is based on the diameter and location determined by the neural network for a subset of the ultrasound images.

[0060] 10A-10C illustrate one embodiment of a process for displaying images of previous detections in response to a sudden loss of a detection. This situation may occur, for example, when the probe is gradually compressing. As the vessel is further compressed, the vessel shape collapses. In such a case, it becomes much more difficult to detect such vessel collapse from only the current image.

[0061] Referring to FIG. 10A, blood vessels 1001 and 1002 are displayed on monitor 800 as a result of the detection process determining their presence. As shown in FIG. 10B, while the detection continues, blood vessels 1001 and 1002 continue to be displayed on monitor 800, but blood vessel 1002 is shown as an oval (as opposed to the circular shape of blood vessel 1002 in FIG. 10A). FIG. 10C illustrates that the detection shows only blood vessel 1001 being displayed, since blood vessel 1002 is not displayed. In other words, FIG. 10C reflects the result of the last successful detection of blood vessel 1002. For example, an ultrasound machine may not be able to detect blood vessel 1002 in FIG. 10C. In such a case, the system displays blood vessel 1002 as a dashed oval representing a virtual detection result, even though the detection process did not yield a result for blood vessel 1002. The advantage of displaying the virtual result with the same shape but different type of outline as the successful result (e.g. solid ellipse vs. dashed ellipse, or solid box vs. dashed box) is to prevent the operator from feeling uncomfortable. In some embodiments, the color of the solid (real) and dashed (virtual) ellipse can be selected from the calculation result of the vessel diameter in the previous image. Since the current value of the vessel diameter is calculated based on the previous image, the color variation can be suppressed, and only the shape line is changed, making it easier for the operator to identify the same vessel (vein).

[0062] More specifically, in some embodiments, the virtual detection results are displayed differently than the real detection results. For example, as shown in FIG. 10C, the virtual detection results are displayed with dashed ovals, while the real detection results are displayed with solid bounding vessels, as shown in the vessel 1002 in FIG. 10A and FIG. 10B. Additionally or alternatively, different colors, different types of filling of the bounding vessels, colors, labels, icons, text, or other types of distinguishing features can be used to distinguish the real detection results from the virtual detection results. Also, note that because the vessel 1001 is an artery, its shape does not change as much as the shape of a vein changes.

[0063] In some embodiments, after identifying a blood vessel in an ultrasound image, a user may wish to search for an optimal location of the blood vessel where a needle or other interventional instrument can be inserted. For example, a blood vessel may extend from the elbow to the wrist, from the upper arm to the elbow, etc., and the operator may wish to be able to search for a particular location or locations of the blood vessel for needle insertion. However, when searching for the best location of the blood vessel, the operator moves the probe from one location to another, thereby successively acquiring multiple ultrasound images that can be checked. In such cases, the operator may miss the best location for examination and / or intervention when the probe is moving.

[0064] In some embodiments, ultrasound images including the blood vessel are acquired while the ultrasound probe is moving. For example, an ultrasound machine can generate the ultrasound images based on ultrasound echo signals received by the ultrasound probe. The ultrasound machine determines at least one ultrasound image of the blood vessel based on a position of the ultrasound probe relative to the blood vessel when the ultrasound image is received, and determines one or more features of the blood vessel in the ultrasound image. The ultrasound machine displays an indication of each feature of the blood vessel in a further ultrasound image (e.g., in the current ultrasound image).

[0065] 11A-11D show an example of ultrasound image highlighting generated when the probe is moving. Referring to FIG. 11A, blood vessels 1101-1103 are shown on an ultrasound image. Blood vessel 1101 is shown as a vein with a circle of a first color, blood vessel 1102 is shown as an artery, and blood vessel 1103 is not identified as either a vein or an artery (e.g., identified with a dotted circle). FIG. 11B shows another ultrasound image of the same location where the shape of blood vessel 1101 is represented by a more elliptical shape with a different highlighting (e.g., different color) than that of FIG. 11A. The shape of blood vessel 1101 may be due to the operator pushing the probe harder into the body at that location. FIG. 11C shows another ultrasound image of the same location where the shape of blood vessel 1101 is represented by a smaller elliptical shape with a different highlighting (e.g., different color) than that of FIG. 11B. Figure 11D shows another ultrasound image of the same location in which the shape of the blood vessel 1101 is represented by a smaller oval shape with a different enhancement (e.g., a different color) and dotted line than in Figure 11C. Figures 11A-11D thus show how the enhancement of the vessel can change due to probe movement and / or pressure.

[0066] 12 shows a flow diagram of one embodiment of a process for displaying one or more blood vessels (or other body structures) as detection results on a monitor having at least one virtual detection result. The process may be performed by processing logic that may include hardware (e.g., circuitry, dedicated logic, etc.), software (such as that executed on a general-purpose computer system or dedicated machine), firmware (e.g., software programmed into a read-only memory), or a combination thereof. In some embodiments, the process is performed by a processor of the ultrasound machine.

[0067] 12, the process begins by processing logic detecting one or more blood vessels from an ultrasound image (processing block 1201). In some embodiments, processing logic detects blood vessels using neural networks, template matching, machine learning (e.g., sequence models including adaboost, deep learning, SVM, RNN, GRU, ConvGRU, LSTM, etc., for processing frame information in a sequence), etc. Additionally or alternatively, processing logic may calculate a likelihood value for each blood vessel indicating a level of confidence that the detection is a blood vessel (processing block 1202). The likelihood value may be between 0 and 1, with 1 representing a high confidence in the detection and 0 representing a low confidence in the detection.

[0068] Processing logic also tracks the detected vessels between frames and identifies the same detected vessels in different images (processing block 1203). In some embodiments, processing logic uses position information and / or vessel type for the tracking and / or identification process.

[0069] Processing logic compares the likelihood value to a threshold, such as 0.7 (processing block 1204). If processing logic determines that the likelihood value for the same vessel falls below the threshold, processing logic determines that vessel detection has failed (processing block 1205) and generates a virtual detection result for display (processing block 1206). In some embodiments, to generate the virtual detection result, processing logic searches one or more previous ultrasound image results and uses the results as the basis for displaying the virtual detection result. The virtual detection result may be calculated using a computational method (algorithm) implemented in several ways. In some embodiments, processing logic takes the most recent successful detection result and uses it for the virtual detection result. Additionally or alternatively, processing logic may select the detection result with the highest likelihood value from a predetermined number of previous frames, such as from the previous two, three, four, or more frames, as the virtual detection result. The number of frames may depend at least in part on the acquisition frame rate, with a higher frame rate allowing more frames to be selected. In one example, processing logic estimates the shape and / or location of the vessel based on results from one or more previous frames generated over a predetermined length of time in the past, such as from 500 msec before the image frame. In one example, the estimation is performed by using motion vector techniques. Additionally or alternatively, the estimation can be performed by using optical flow techniques. Processing logic then displays the virtual detection results in the ultrasound image along with any other detection results, real or virtual (processing block 1207).

[0070] In some embodiments, the probe of the ultrasound machine has a sensor to detect its movement. If the movement of the probe exceeds a certain speed, the processing logic does not display the virtual detection results on the monitor. This suppression of the display of the virtual detection results is because it is more difficult for the processing logic to estimate the virtual detection results when the probe is moving, for example, because it is not possible to estimate symmetric changes in the shape and / or position of the blood vessel when the probe is moving.

[0071] While the region of interest is compressed and relaxed by the probe of the ultrasound system, the shape of the blood vessel may change. For example, Figures 13A-13D show a period (i) in which the pressure by the probe is increased to a certain level and the shape of the blood vessel changes (e.g., collapses) from a normal shape (usually a circular ellipse) to an additional shape (e.g., a collapsed ellipse). Figures 14A-14D show a period (ii) in which the pressure by the probe is decreased from a certain level to a released state and the shape of the blood vessel returns from the additional shape to the normal shape. During periods (i) and (ii), ultrasound images are collected and blood vessels are detected. In some embodiments, the detection of blood vessels is performed using AI (e.g., AI software implementing a neural network).

[0072] In some embodiments, the detection results of the ultrasound images acquired during time period (i) are analyzed to generate results that are used to detect / analyze the ultrasound images during time period (ii). For example, the ultrasound system can use the detection results of time period (i) as conditional inputs to a neural network to generate additional detection results for time period (ii). Additionally or alternatively, the ultrasound system can use the position and size of each blood vessel detected during time period (i) to predict (e.g., bound) an area for searching for the blood vessel during time period (ii), thereby contributing to a fast and accurate detection of the same blood vessel during time period (ii). In one example, the ultrasound system can generate a region of interest based on the position and size of each blood vessel detected during time period (i) and provide the region of interest as a conditional input to the neural network, which can then generate an inference about the position and size of the blood vessel according to the region of interest in time period (ii).

[0073] In one example, the ultrasound machine adjusts the difference between the collapse speed and the recovery speed of the blood vessel when determining the detection result of the period (ii) based on the detection result of the period (i). For example, in an ultrasound examination using a probe, the probe gradually presses the test site in the period (i), and the operator releases the probe in the period (ii). Therefore, the collapse speed generally depends on the pressure caused by the probe based on the pressing action by the operator. On the other hand, the recovery speed mainly depends on the nature of the blood vessel itself (i.e., a nearly constant speed). As the inventors have observed, the recovery speed of the blood vessel itself is usually slower than the collapse speed. Therefore, the processor can correct the detection result of the period (i) to match the actual recovery speed of the blood vessel (e.g., by multiplying the detection result of the period (i) by an appropriate coefficient to delay the recovery function). FIG. 15 includes a graph showing the relationship between blood vessel diameter and time. Referring to FIG. 15, the delay 1501 represents the gap between the actual recovery speed and the predicted speed, which is assumed to be the same as the collapse speed and must be met. In some embodiments, the ultrasound system calculates the collapse velocity by using vessel detection, diameter determination, and tracking techniques, then detects the timing of transition from period (i) and period (ii), then calculates the (actual) recovery velocity by using the first few period frames (ii), then a coefficient between the actual velocity and the predicted velocity is generated. Thus, by using the value of the coefficient, the detection result in period (i) is appropriately reflected in the detection process in period (ii). Details are also described using the flowchart of FIG. 16 below. In one example, such coefficient can be updated by essentially reflecting the detection result in period (ii) as time progresses, since the actual result (i.e., diameter) of the same vessel volume progresses as the detection process progresses, thereby making the gap more accurate. In one example, the vessel diameter can be used instead of the diameter in FIG. 15 and FIG. 16. Furthermore, it should be noted that the technique described with reference to FIG. 15 and FIG. 16 can be based on the case where period (i) refers to the collapse velocity and period (ii) refers to the recovery velocity, but can also be applied to the case where period (i) refers to the recovery velocity and period (ii) refers to the collapse velocity.

[0074] Figure 16 shows a flow diagram of one embodiment of a process for using detection results from one period to generate detection results from another period, for example as described above with respect to Figures 13A-15. The process can be performed by processing logic that can include hardware (e.g., circuitry, dedicated logic, etc.), software (such as that executed on a general purpose computer system or dedicated machine), firmware (e.g., software programmed into a read-only memory), or a combination thereof. In some embodiments, the process is performed by a processor of the ultrasound machine.

[0075] 16, the process begins by processing logic detecting blood vessels from an ultrasound image (processing block 1601). As previously mentioned, many different algorithms can be used to detect blood vessels, including but not limited to template matching, machine learning (e.g., sequence models including adaboost, deep learning, SVM, RNN, GRU, ConvGRU, LSTM, etc., for processing frame information in a sequence, etc.), etc.

[0076] Processing logic also tracks the detected vessels between frames (images) to identify the same vessels in these frames (processing block 1602). During pressure by the probe, the shape of the detected vessels changes, and processing logic analyzes the changes in shape, position, size, combinations thereof, etc. of the same vessels from frame to frame over several frames (processing block 1603). In response to the analysis, processing logic determines the first period (i) in which the shape transitions from a normal ellipse (no pressure) to a most collapsed ellipse (maximum pressure) (processing block 1604). Since the frame rate is predetermined by the ultrasound machine (e.g., 30 frames / sec), processing logic calculates the time of the first period and the collapse velocity of the vessel (processing block 1605). Thus, from processing blocks 1601-1605, processing logic obtains information of the vessels during period (i).

[0077] The processor determines a frame that is a start frame of the recovery of the vessel (processing block 1606). For example, the processing logic can equate the recovery speed of the vessel to the collapse speed of the vessel. Next, the processor can determine a frame that is a start frame of the recovery of the vessel, for example, from this start frame, enter the period (ii). Then, the processor can estimate the position, diameter, or other information of the vessel in the current frame based on the detection result during the period (i) and the time interval between the current time and the time of the start frame. In other words, in processing block 1606, several frames of the period (ii) are also acquired so that the time of the change from the period (i) to the period (ii) is determined. Also, in processing block 1606, the processor can use several frames of the period (ii) to calculate the recovery speed since several frames of the period (ii) have already been acquired, whereby the coefficient (corresponding to the delay 1501 in FIG. 15) to be applied to the detection result during the period (i) is calculated based on several frames of the period (ii).

[0078] In processing block 1607, a new current image (belonging to period (ii)) is obtained. Since the time interval and coefficient between the current time and the time of change from period (i) to period (ii) are known, information about the vessel in the appropriate period (i) is obtained for use in creating the new image. More specifically, the processor may take into account the difference between the vessel's recovery speed and the collapse speed. For example, in some embodiments, the processor modifies the detection result of period (i) by, for example, multiplying the detection result of period (i) by an appropriate coefficient to delay the recovery function to match the vessel's actual recovery speed (processing block 1607). Thus, the detection result of the appropriate time point of period (i) is input to the processor (e.g., AI algorithm) to enhance the detection of the vessel in the new current image. In one example, the position and / or diameter of the vessel at the appropriate time point of period (i) is input to the processor (e.g., AI algorithm) to limit the area designated for search and / or limit the size of the vessel for search, thereby reducing the burden on the processor.

[0079] In some embodiments, after identifying a blood vessel in an ultrasound image, a user may wish to search for an optimal location of the blood vessel where a needle or other interventional instrument can be inserted. For example, a blood vessel may extend from the elbow to the wrist, from the upper arm to the elbow, etc., and the operator may wish to be able to search for a particular location or locations of the blood vessel for needle insertion. However, when searching for the best location of the blood vessel, the operator moves the probe from one location to another, thereby successively acquiring multiple ultrasound images that can be checked. In such cases, the operator may miss the best location for examination and / or intervention when the probe is moving.

[0080] In some embodiments, ultrasound images including the blood vessel are acquired while the ultrasound probe is moving. For example, an ultrasound machine can generate the ultrasound images based on ultrasound echo signals received by the ultrasound probe. The ultrasound machine determines at least one ultrasound image of the blood vessel based on a position of the ultrasound probe relative to the blood vessel when the ultrasound image is received, and determines one or more features of the blood vessel in the ultrasound image. The ultrasound machine displays an indication of each feature of the blood vessel in a further ultrasound image (e.g., in the current ultrasound image).

[0081] In some embodiments, several past ultrasound images are acquired and image analysis, including vessel detection, is performed on each image to indicate the best or more desirable position for insertion that the user has previously performed. In some embodiments, the ultrasound machine selects one image from the past images based on the analysis (hereinafter the selected one image is also referred to as the "best image"). In some embodiments, the selected image is the ultrasound image obtained when the probe is placed in the desired portion of the vessel for insertion. The determination of which position is best or desirable can be based on predetermined criteria. Such predetermined criteria can include the size of the vessel diameter, the position of the predetermined vessel being shallower than a predetermined depth, etc. Examples of such predetermined criteria are described in more detail below.

[0082] Information about the vessel of the best image detected can be superimposed on the current (live) image. This superimposition can be done like a graphic superimposed on the image. FIG. 17 shows an example of a current (live) image with such superimposition. With reference to FIG. 17, a vein 1701 is highlighted with an ellipse with the addition of an L-shaped depth indicator. In one example, the L-shaped depth indicator can be additionally displayed when the position of the detected vessel is shallower than a predetermined depth. However, another vessel 1702 is shown superimposed on the current image as information about the vessel of the best image. Note that a vessel 1703 is also shown on the image and represents an artery. Thus, the vessel 1703 is shown with a bleed-through as opposed to a circle or ellipse without bleed-through to highlight the vessel 1703 as an artery and not a vein.

[0083] To identify and overlay information associated with the best image, the ultrasound machine may include a probe having a position sensor to provide position information that is analyzed by the processor. In some embodiments, the position sensor includes a gyro sensor or an accelerometer. In some embodiments, the image is stored in memory along with the probe's position information.

[0084] The ultrasound machine also includes a processor capable of detecting blood vessels from the ultrasound images and selecting one image from the plurality of images as the best image based on at least one of the following:

[0085] 1) The vessel diameter is large (e.g., larger than a threshold or the largest among multiple images);

[0086] 2) the detected blood vessel is located shallower than the given depth;

[0087] 3) the detected vessel is not located too close (e.g., within 2 mm lateral or 3-4 mm deep) to an anatomical structure (such as, but not limited to, a nerve or artery) that would be damaged or obstructed when inserting the needle or catheter;

[0088] 4) high likelihood value for vessel detection;

[0089] 5) The internal structure of the blood vessel is determined to be clear, in other words, the blood vessel wall or intima is clearly visible in the ultrasound image;

[0090] 6) No abnormal structures (e.g., thrombus, edema, etc.) are observed in the blood vessels of the ultrasound images; and

[0091] 7) The diameter of the vessel is stable around the location of the vessel. In some embodiments, the ultrasound machine makes a selection that takes into account that the location of the vessel is stable around the location. In some embodiments, the word "stable" means that the deviation of the diameter is less than a threshold diameter (e.g., 10-20% of the maximum diameter of the vessel). The word "stable" can also mean that the deviation of the change in position is less than a threshold. Since the instrument (e.g., catheter) is inserted along the longitudinal direction of the vessel, such deviation is preferably small to prevent phlebitis due to friction between the instrument and the vessel. In other words, the vessel is not in a good position for insertion to be constricted, resulting in a large change in position while the diameter is locally large.

[0092] After the best image is determined, information about the best image can be overlaid on the monitor currently displaying the live B-mode image. In some embodiments, the information includes one or more of the following:

[0093] 1) Information about the detected vessel in the best image (e.g., vessel location and / or vessel size). An example of displaying such information is a dashed circle indicating the diameter of the vessel in the previous best image. Another example is displaying an arrow pointing to the location of the vessel in the previous best image. Yet another example is if a detected vessel in the current image is the same as a vessel in the best image, the color to highlight the same vessel is the same (e.g., a dashed green circle can be displayed as the vessel in the best image and a solid green circle can be displayed as the same vessel in the current image). As mentioned above, in one example, the color of the circle can be selected based on standard colors corresponding to at least the vessel diameter and / or catheter size. Additionally or alternatively, vessel identification information can be used instead of the same color;

[0094] 2) time information (e.g., the time when the best image was obtained, relative to the present, etc.);

[0095] 3) Guidance information (e.g., by using the position of the probe). Position information regarding the position where the best image was scanned is stored in memory, so that guide information is determined by comparing the position where the best image was scanned with the current position (e.g., showing a direction on the monitor to facilitate movement to the probe position, or displaying a relative distance). In some embodiments, the guide information is shown on a schema (an illustration of the whole body or a part of the body). For example, if an arm is displayed on the screen, a probe movement direction indicator (e.g., an arrow) can be superimposed on the arm diagram.

[0096] Figures 18A and 18B show an example of guide information together with information about the best position. With reference to Figures 18A and 18B, the solid ovals represent the veins detected in the current image, and the dashed ovals represent the (same) veins detected in the best previous image. Arrows 1801 and 1802 represent the guide information.

[0097] In Figure 18A, arrow 1801 enters a dashed oval with a narrow arrow tip. This arrow configuration indicates that the probe should be moved backwards relative to the monitor. In Figure 18B, arrow 1802 exits a dashed oval with a wide arrow tip. This arrow configuration indicates that the probe should be moved forwards relative to the monitor.

[0098] In some embodiments, the guide information is displayed along with information about the detected blood vessels in the best image. For example, the ultrasound machine may display the guide information as annotations along with an indication of the diameter of the detected blood vessel and / or the classification of the blood vessel as a vein or artery. As another example, the ultrasound machine may display the guide information along with a dashed circle of a color that may be selected based on standard colors corresponding to at least the blood vessel diameter and / or catheter size.

[0099] In addition to or instead of displaying the above information, the ultrasound machine can provide audible and / or tactile information to the operator of the ultrasound machine. For example, in some embodiments, the audible information is changed in frequency and / or volume according to the relative distance between the location of the best image and the current location. Thus, as the operator moves the probe closer to or further from the location of the best image, the audible information changes in one or more ways (e.g., getting louder) to alert the user that it is getting closer, or in one or more other ways (e.g., getting quieter) to alert the user that it is getting further away. In one example, the tactile information is changed in power and / or pattern of vibrations by the body of the probe (e.g., a smartphone or tablet) to indicate the proximity of the probe to the location of the best image.

[0100] If the guide information is displayed on the monitor, the movement of the probe can be tracked to determine whether the probe has moved in accordance with the guide information. For example, if audible information is provided to the operator, the ultrasound machine can track the movement of the probe to determine whether the probe has moved in accordance with the audible information. As another example, if the operator moves the probe in the wrong direction relative to the guide information and / or audible information, the ultrasound system can provide an indication of the wrong direction, such as by displaying a stop sign icon, a caution icon, a reverse direction icon, etc., and / or playing an audio signal that is not unpleasant to the operator, such as words such as "stop," "wrong direction," or a tone with an unpleasant beep. Additionally or alternatively, if the operator moves the probe in the right direction relative to the guide information and / or audible information, the ultrasound system can provide an indication of the right direction, such as by displaying a go icon, a traffic light icon with a green light, etc., and / or playing an audio signal that is pleasant to the operator, such as words such as "go," "right direction," a clap, a cheer, or a series of tones with a pleasant harmony.

[0101] Typical Flow Diagram Figure 19A shows a flow diagram of one embodiment of a process for displaying blood vessels on an ultrasound image. This process relates to the techniques described in connection with Figures 7, 8, and 9. This process can be performed by processing logic that can include hardware (e.g., circuitry, dedicated logic, etc.), software (such as that executed on a general purpose computer system or a dedicated machine), firmware (e.g., software programmed into a read-only memory), or a combination thereof. In some embodiments, the process is performed by a processor of the ultrasound machine.

[0102] 19A, the process begins by processing logic receiving ultrasound images including a blood vessel (processing block 1901). Processing logic determines diameters of the blood vessels in the ultrasound images, the diameters including the respective diameters of each of the ultrasound images (processing block 1902). In some embodiments, the determining is performed using a neural network implemented at least in part in hardware of a computing device, such as an ultrasound machine or a computing device (e.g., a tablet) coupled to the ultrasound machine.

[0103] Processing logic also determines a vessel diameter based on vessel diameters from previous images (processing block 1903). That is, the vessel diameter in the current image may be calculated from a plurality of vessel diameters in a predetermined number of previous images as previously described in connection with FIG. 7. Based on the vessel diameter, processing logic selects a color (processing block 1904). In some embodiments, selecting a color includes matching the color to a standard color that corresponds to the catheter size determined based on the vessel diameter.

[0104] Processing logic then illustrates the blood vessel with a marker having a color in one of the ultrasound images (processing block 1905). In some embodiments, the process also includes processing logic for determining a location of the marker on a touchscreen of the computing device (processing block 1906) and receiving a user selection via the touchscreen proximate the location (processing block 1907). In response to receiving the user selection, the processor disables display of the marker if the marker is currently displayed on the touchscreen and enables display of the marker if the marker is not currently displayed on the touchscreen (processing block 1908).

[0105] 19B shows a flow diagram of another embodiment of a process for displaying blood vessels on an ultrasound image. This process is related to the techniques described in connection with FIG. 7. This process can be performed by processing logic that can include hardware (e.g., circuitry, dedicated logic, etc.), software (such as that executed on a general-purpose computer system or a dedicated machine), firmware (e.g., software programmed into a read-only memory), or a combination thereof. In some embodiments, the process is performed by a processor of the ultrasound machine.

[0106] 19B, the process begins by processing logic receiving ultrasound images including a blood vessel (processing block 1911). Processing logic determines diameters of the blood vessels in the ultrasound images, the diameters including respective diameters of each of the ultrasound images. Processing logic also calculates a blood vessel diameter based on the determined diameters (processing block 1912). In some embodiments, the determining is performed using a neural network implemented at least in part in hardware of a computing device (e.g., an ultrasound machine or a computing device coupled to the ultrasound machine).

[0107] After determining the vessel diameter, in some embodiments, processing logic receives a user selection of a function indicating one of the mean, median, mode, maximum, or minimum. Additionally or alternatively, processing logic may determine the amount of movement of the ultrasound probe used to generate the ultrasound image and select the function as the mean, median, mode, maximum, or minimum based on the amount of movement (processing block 1913). In some embodiments, the selection process is based on comparing the amount of movement to a threshold movement value and selecting the function as the maximum if the amount of movement is less than the threshold movement value and selecting the function as the mean if the amount of movement is equal to or greater than the threshold movement value.

[0108] Processing logic then determines a current value of the vessel diameter based on the vessel diameter in the previous image by applying the function to the vessel diameter in the previous image (processing block 1914). Based on the current value of the vessel diameter, processing logic selects a color (processing block 1915). In some embodiments, selecting a color includes matching the color to a standard color that corresponds to the catheter size determined based on the vessel diameter. Processing logic then shows the vessel in one of the ultrasound images with a marker having the color (processing block 1916). Additionally or alternatively, the selection of the function for calculating the current value for the vessel diameter may be determined as a preset for the configuration of the ultrasound system (e.g., processing block 1916 may come before processing block 1911, and other sets of processing blocks may be repeated).

[0109] 19C shows a flow diagram of yet another embodiment of a process for displaying blood vessels on an ultrasound image. This process is related to the techniques described in connection with FIG. 7. This process can be performed by processing logic that can include hardware (e.g., circuitry, dedicated logic, etc.), software (such as that executed on a general purpose computer system or dedicated machine), firmware (e.g., software programmed into a read-only memory), or a combination thereof. In some embodiments, the process is performed by a processor of the ultrasound machine.

[0110] 19C, the process begins by processing logic receiving ultrasound images including blood vessels (processing block 1921). Processing logic determines diameters of blood vessels in the ultrasound images, the diameters including respective diameters of the blood vessels for each ultrasound image of the ultrasound images (processing block 1922). In some embodiments, the determining is performed using a neural network implemented at least in part in hardware of a computing device (e.g., an ultrasound machine or a computing device coupled to the ultrasound machine).

[0111] Next, processing logic determines a vessel diameter based on the vessel diameter by applying a function to the vessel diameter (processing block 1923). Based on the vessel diameter, processing logic selects a color (processing block 1924). In some embodiments, selecting a color includes matching the color to a standard color that corresponds to the catheter size determined based on the vessel diameter.

[0112] Processing logic then displays the blood vessel in one of the ultrasound images with a marker having a color, the display being based on whether the blood vessel is a vein or an artery (processing block 1925). In some embodiments, if the blood vessel is determined to be a vein, processing logic provides the display by enabling the display of a marker having a color similar to the blood vessel, or if the blood vessel is determined to be an artery, processing logic inhibits the display of the marker having a color. Additionally or alternatively, if the blood vessel is determined to be an artery, processing logic may provide the display of the blood vessel in one of the ultrasound images using an additional marker that distinguishes arteries from veins.

[0113] Additionally or alternatively, processing logic may perform the function of distinguishing arteries from veins without calculating vessel diameter as described in processing block 1922. In such a case, processing logic may use the results of performing the function to provide a user with an indication as to whether a vessel displayed in the ultrasound image is an artery or a vein. Thus, in one example, if the detected vessel is a vein, the highlighting color may be selected depending on the vessel diameter or the current value of the vessel diameter, and if the detected vessel is an artery, the highlighting color may be fixed to one color regardless of diameter.

[0114] Figure 20 shows a flow diagram of yet another embodiment of a process for displaying blood vessels on an ultrasound image. This process is related to the techniques described in connection with Figures 8, 9, and 12. This process can be performed by processing logic that can include hardware (e.g., circuitry, dedicated logic, etc.), software (such as that executed on a general purpose computer system or a dedicated machine), firmware (e.g., software programmed into a read-only memory), or a combination thereof. In some embodiments, the process is performed by a processor of the ultrasound machine.

[0115] 20, the process begins by processing logic receiving an ultrasound image including a blood vessel (processing block 2001). Processing logic determines the diameter and location of the blood vessel within a subset of the ultrasound image (processing block 2002). In some embodiments, the determination is performed using a neural network implemented at least in part in hardware of a computing device (e.g., an ultrasound machine or a computing device coupled to an ultrasound machine). In one example, the processing logic performs the tracking techniques and likelihood value calculations as described above in FIG.

[0116] After the diameter and location of the blood vessel are determined, processing logic determines a missed detection for one of the blood vessels in one of the ultrasound images that is not included in the subset (processing block 2003). A missed detection indicates the absence of a diameter or location being determined, for example by a neural network, for one of the blood vessels in one of the ultrasound images.

[0117] In response to determining a failed detection, processing logic displays in one of the ultrasound images an indication (e.g., an oval, a dot, a bounding vessel, etc.) of the diameter and location of one of the blood vessels in one of the ultrasound images (processing block 2004). The indication is based on the diameter and location determined, for example, by a neural network, for the subset of the ultrasound images. The indication can include one or more indications, such as a bounding box and text.

[0118] In some embodiments, displaying the indicia includes displaying a marker for one of the vessels in a first format (e.g., a solid oval, etc.) to indicate a detection failure, and displaying an additional marker for a further vessel of the vessel in a second format (e.g., a dotted oval, etc.). In one example, the first format of the indicia includes a dashed vessel and the second format of the additional indicia includes a solid vessel. Additionally or alternatively, the first format of the indicia can include a first color and the second format of the additional indicia can include a second color. In some embodiments, the second color corresponds to a standard color representing a catheter size, and the first color does not indicate a standard catheter size.

[0119] In one example, the process includes receiving a user selection via a touch screen of a computing device proximate the marker (processing block 2005) and, in response, disabling the display of the marker (processing block 2006). Processing logic then receives a further user selection via a touch screen of a computing device proximate the location indicated by the marker (processing block 2007) and, in response, enables the display of the marker (processing block 2008). Furthermore, in some embodiments, some processing blocks of FIG. 7 may be suitably incorporated into FIG. 21 such that not only the tracking technique but also the calculation of the vessel diameter and associated highlighting are performed. For example, in block 2004, the marker may be based on a calculation of the vessel diameter using a subset of the images.

[0120] Figure 21 shows a flow diagram of a further embodiment of a process for displaying blood vessels on an ultrasound image. This process is related to the techniques described in connection with Figure 18. This process can be performed by processing logic that can include hardware (e.g., circuitry, dedicated logic, etc.), software (such as that executed on a general purpose computer system or a dedicated machine), firmware (e.g., software programmed into a read-only memory), or a combination thereof. In some embodiments, the process is performed by a processor of the ultrasound machine.

[0121] 21, the process begins by processing logic receiving an ultrasound image including a blood vessel while an ultrasound probe is moving (processing block 2101). The ultrasound image is based on ultrasound echo signals received by the ultrasound probe while in motion.

[0122] After receiving the ultrasound images, processing logic determines one of the ultrasound images based on a position of the ultrasound probe relative to the blood vessel when the ultrasound image was received (processing block 2102) and determines a feature of the blood vessel in the ultrasound image (processing block 2103). In some embodiments, the feature includes a diameter of the blood vessel in the ultrasound image and the indicia include a bounding vessel representing the diameter. Additionally or alternatively, the feature can include a position of the blood vessel in the ultrasound image.

[0123] Processing logic then displays an indicator of the vascular feature in a further ultrasound image of the ultrasound images (processing block 2104). Processing logic may display a further indicator to guide movement of the ultrasound probe to a location in the further ultrasound image of the ultrasound images (processing block 2105). In some embodiments, the further ultrasound image is a current ultrasound image of the ultrasound images.

[0124] The procedures described herein constitute an improvement over procedures that do not display blood vessels in the enhanced ultrasound image. Rather, the procedures described herein provide enhancements that include useful, and potentially important, information, and display the information to the operator of the ultrasound system so that the operator can apply the information to the ultrasound procedure in real time. Thus, the procedures described herein provide a significant improvement in the patient's experience, including reducing or minimizing the number of times that an interventional instrument is inserted into the patient, resulting in pain and risk of infection for the patient, as well as preventing undesired insertion of an instrument, for example, into an artery rather than a vein. Thus, the patient does not perceive unnecessary pain and discomfort, and the risk of one or more of infection, thrombus due to insertion of an oversized catheter, extravasation, and damage to the vessel wall is reduced compared to conventional ultrasound procedures that do not display enhanced blood vessels. Thus, the procedures described herein may be suitable for medical procedures and examinations for which conventional ultrasound procedures are not suitable.

[0125] Examples of Ultrasound Machines FIG. 22 is a block diagram of an embodiment of an ultrasound machine. It should be noted that any type of ultrasound machine, including fixed, portable, handheld, and combinations thereof, can be used to perform the vessel identification and highlighting techniques described herein. The block diagram is one example of an ultrasound machine that can be constructed using the blocks of FIG. 22. For example, signals can be redefined, and blocks can be modified, combined, split, added, or removed to form a modified system without changing the functionality of the block diagram. Thus, such modified systems are considered to be within the scope of this disclosure. Furthermore, the blocks, modules, and units of an ultrasound machine, such as the ultrasound machine shown in FIG. 22, can be implemented as any type of module or component in software (e.g., as software instructions executable on a processing system), hardware, or a combination thereof, as a standalone application, or as a module or component of another device application, and in any type of computing device.

[0126] Referring to FIG. 22, an ultrasound machine includes an ultrasound transducer probe 2201 coupled to an ultrasound device 2202. The ultrasound transducer probe 2201 includes an ultrasound transducer 2201A electrically coupled to a receiver 2201B and a transmitter 2201C. The ultrasound transducer 2201A has one or more transducer elements and, in operation, transmits ultrasound energy from the one or more transducer elements toward a subject in response to a transmit signal from the transmitter 2201C and receives ultrasound echoes from the subject using the receiver 2201B. The receiver 2210B and the transmitter 2210C can include any type of circuit or any type of other form, such as a processor, as described below. The receiver 2210B and the transmitter 2210C may be formed separately or integrated into a single form. The receiver 2210B and the transmitter 2210C can be incorporated into a processor having other functions, such as a processor 2202C, described below. The ultrasonic echoes are converted into electrical signals by the receiver 2201B and electrically coupled to electronics within the ultrasonic device 2202 (e.g., an analog-to-digital (A / D) converter 2202A, one or more processors 2202C, memory module 2202B, a beamformer, FPGA, etc.) configured to process the electrical signals and form one or more ultrasound images.

[0127] In one example, the ultrasonic probe 2201 includes an A / D converter (not shown in FIG. 22) that can be configured to digitize the electrical signals generated by the receiver 2201B based on the ultrasonic echo. Additionally, the ultrasonic probe 2201 can also include a partial or full beamformer (not shown in FIG. 22) configured to sum the electrical signals in a phase-corrected manner. Thus, the ultrasonic probe 2201 can be implemented to couple the digital signal to the ultrasonic device 2202 via a wired or wireless communication link, etc. For example, the ultrasonic probe 2201 and the ultrasonic device 2202 can each include a transceiver (not shown in FIG. 22) that communicates using a communication signal that is digitally modulated based on the digital signal.

[0128] The processor 2202C may include, in part or in whole, any type or number of application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), integrated circuits (ICs), logic, etc. In some embodiments, the processor 2202C includes a beamformer and includes processing for forming an ultrasound image. It should also be noted that the processing described herein may be performed at least in part on the cloud, on an attached or networked processing device, on a discrete or external GPU, or on custom-specific connected hardware such as a USB-connected device.

[0129] Capturing ultrasound data from a subject using an ultrasound probe 2201 generally involves generating ultrasound waves, transmitting ultrasound waves into the subject, and receiving ultrasound waves reflected by the subject. Ultrasound data can be captured using a wide range of frequencies, such as, for example, low frequency ultrasound (e.g., less than 15 MHz) and / or high frequency ultrasound (e.g., 15 MHz or higher) can be used. One skilled in the art can readily determine which frequency range to use based on factors such as, for example, but not limited to, imaging depth and / or desired resolution.

[0130] In some embodiments, the ultrasonic device 2202 includes one or more processors 2202C that transmit electrical currents to the ultrasonic probe 2201 to emit sound waves and receive electrical pulses generated from the returning echoes from the probe 2201. The A / D converter 2202A receives analog signals from the receiver 2201B and converts them to digital data stored in the memory 2202B. The memory 2202B also stores software and other data (e.g., templates, user preferences for enhancements, etc.) related to detection of blood vessels (e.g., veins, arteries, and capillaries), as well as analysis results along with ultrasound images and other data associated therewith.

[0131] The processor 2202C processes raw data associated with the received electrical pulses and forms an image that is transmitted to the display device 2203, which displays the image on the monitor 2203A. Thus, the monitor 2203A can display ultrasound images from ultrasound data processed by the processor 2202C of the ultrasound device 2202. The monitor 2203A can also display vessel detection results, as well as probe guide information, as described herein. In one example, the monitor 2203A includes a clinical display device. Additionally or alternatively, the monitor 2203A can include a display of a computing device coupled to the ultrasound machine, such as a tablet.

[0132] In some embodiments, the processor 2202C performs vessel detection using template matching, artificial intelligence or machine learning (e.g., adaptive boosting, deep learning, support vector machine (SVM), sequence models including RNN, GRU, ConvGRU, LSTM, etc. for processing frame information in a sequence, etc.), and / or another detection method. In some embodiments, the processor 2202C executes AI algorithms and / or uses neural networks to identify veins and arteries and locate them in the ultrasound image.

[0133] In some embodiments, after the processor 2202C detects a blood vessel, the processor 2202C displays the blood vessel on a monitor or display of the ultrasound system. In one example, the detected blood vessel is displayed in a highlighted manner by the processor 2202C to provide information to an operator or user of the ultrasound machine. In some embodiments, the processor 2202C draws an outline or other form of a blood vessel representation (e.g., an identifier) ​​around or near the blood vessel.

[0134] By displaying the vessels in an enhanced manner, additional information can be made available to the operator. For example, in some embodiments, the outline of the vein is changed to match the color coding of the largest catheter that can fit within that vein. The color coding can be the industry standard color coding for catheters. In this manner, the operator can quickly learn which catheters can be used with a particular vessel. In this manner, the operator selects a catheter size based on the vessels displayed. Alternatively, by being able to color code the vessels according to the largest catheter, the operator can select a catheter size and have the ultrasound system identify all vessels (e.g., veins) in the image that are appropriate for that catheter size.

[0135] Additionally or alternatively, the processor 2202C can identify all veins in the image that are suitable for a particular catheter size. In some embodiments, the operator can touch a vein in the image and have the processor 2202C display the diameter and depth of that vessel. In some embodiments, the processor 2202C automatically identifies the most central, most superficial vein and automatically provides the diameter and depth of that vein on the ultrasound machine display.

[0136] In some embodiments, the processor 2202C calculates a likelihood value for each detected vessel as an indication of the confidence level associated with the detection result. In one example, the processor 2202C generates a more or less opacified vein / artery contour to indicate the reliability of the prediction for that vessel.

[0137] In some embodiments, the processor 2202C tracks detected vessels across frames and identifies the same detected vessels in multiple images. The processor 2202C can use vessel position information or vessel type for tracking and / or identification.

[0138] In some embodiments, the ultrasound system has one or more user input devices 2204 (e.g., keyboard, cursor control device, etc.) for inputting data from at least one operation unit 2204A. Along with the memory 2202B, the ultrasound system also has a storage device 2205 including a storage device 2205A (e.g., hard, floppy, compact disk (CD), digital video disk (DVD)) for storing acquired images (e.g., past images, current images, etc.), which can be used to calculate vessel diameter, select the optimal image for insertion, and for other operations described herein. The ultrasound system further includes a data communication device 2206 having a data communication interface 2206A for providing communication between the ultrasound device 2202 and an external server 2207.

[0139] In some embodiments, the ultrasound machine includes a handheld ultrasound device with a probe and a tablet or smartphone connected to each other. The connection may be wireless or wired. FIG. 23 shows an example of such an ultrasound machine. Referring to FIG. 23, a wireless ultrasound probe 2300 wirelessly communicates with a smartphone 2302. In some embodiments, the probe 2300 includes hardware and software operable to perform transmission / reception with an array of transducers, A / D conversion; beamforming; quadrature detection; and wireless transmission. In some embodiments, the smartphone 2302 includes hardware and software operable to generate an ultrasound image (e.g., a B-mode image) and display the image on its display. The smartphone 2302 hardware and software can detect veins, calculate vessel diameters, use multiple vessel diameters to calculate a value (e.g., a maximum value), and generate and apply color enhancement of the detected veins to the monitor.

[0140] Figure 24 shows a data flow diagram of the vessel identification subsystem. In some embodiments, the subsystem is part of an ultrasound machine, such as that previously described in Figure 22. Additionally or alternatively, the subsystem may operate in conjunction with the ultrasound machine by providing information to and receiving information from the ultrasound machine for use and / or display.

[0141] 24, memory 2401 stores ultrasound image 2402. Neural network (or other vessel detection mechanism) 2403 receives or acquires ultrasound image 2402. In some embodiments, neural network 2403 uses AI or other machine learning algorithms to determine vessel location and diameter within ultrasound image 2402 and outputs this information as vessel location / diameter information 2404 to display generator 2406.

[0142] In some embodiments, neural network 2403 also uses AI or other machine learning algorithms to determine when there has been a failed vessel detection and outputs this information as failed vessel detection information 2405 to display generator 2406. In some embodiments, neural network 2403 includes one or more processors.

[0143] In response to the vessel location / diameter information 2404 and / or vessel missed detection information 2405, a display generator 2406 generates vessel graphics or other displays for superimposition on one or more ultrasound images.

[0144] For further details, see U.S. patent application Ser. No. 2005 / 013363 entitled "Identifying Blood Vessels in Ultrasound Images" and U.S. patent application Ser. No. 2005 / 013363 entitled "Instrument Insertion Guide," filed concurrently herewith.

[0145] The system described herein constitutes an improvement over systems that do not display blood vessels in the enhanced ultrasound image. Rather, the system described herein generates enhancements that include useful, and potentially important, information and displays the information to the operator of the ultrasound system so that the operator can apply the information to the ultrasound procedure in actual real-time. Thus, the system described herein can significantly improve the patient experience, including reducing or minimizing the number of times that an interventional instrument is inserted into the patient, resulting in pain and risk of infection for the patient, as well as preventing undesired insertion of an instrument, e.g., insertion into an artery rather than a vein. Thus, the patient does not perceive unnecessary pain and discomfort, and the risk of one or more of infection, blood clots due to insertion of an oversized catheter, extravasation, and damage to the vessel wall is reduced compared to conventional ultrasound systems that do not display enhanced blood vessels. Thus, the system described herein may be suitable for medical procedures and examinations for which conventional ultrasound systems are not suitable.

[0146] Several example embodiments are described herein.

[0147] Example 1 is a method implemented by a computing device, the method including the steps of receiving ultrasound images including blood vessels; determining diameters of blood vessels in the ultrasound images using a neural network implemented at least in part in hardware of the computing device, the diameters including respective diameters in each of the ultrasound images; determining blood vessel diameters based on the diameters of the blood vessels; selecting a color based on the blood vessel diameters; and displaying the blood vessels with a colored marker in one of the ultrasound images.

[0148] Example 2 is the method of Example 1, optionally including receiving a user selection of a function indicating one of a mean, a median, a mode, a maximum, or a minimum, and determining the vessel diameter includes applying the function to the vessel diameter.

[0149] Example 3 is the method of Example 1, which may optionally include determining an amount of movement of an ultrasound probe used to generate the ultrasound image and selecting a function as a mean, median, mode, maximum, or minimum based on the amount of movement, and wherein determining the vessel diameter includes applying the function to the vessel diameter.

[0150] Example 4 is the method of Example 3, which can optionally include comparing the amount of movement to a threshold movement value, and wherein selecting the function is based on the comparison and includes selecting the function as the maximum if the amount of movement is less than the threshold movement value, or selecting the function as the average if the amount of movement is equal to or greater than the threshold movement value.

[0151] Example 5 is the method of Example 1, which can optionally include a step of determining a catheter size based on a vessel diameter, and wherein selecting a color includes matching the color to a standard color corresponding to the catheter size.

[0152] Example 6 is the method of Example 1, which may optionally include a step of determining whether the blood vessel is a vein or an artery, and the displaying step includes a step of enabling display of a marker having a color similar to that of the blood vessel if the blood vessel is determined to be a vein, or a step of prohibiting display of a marker having a color if the blood vessel is determined to be an artery.

[0153] Example 7 is the method of Example 1, which may optionally include a step of determining whether the blood vessel is a vein or an artery, and if the blood vessel is determined to be an artery, showing the blood vessel in one of the ultrasound images with a further mark that distinguishes between arteries and veins.

[0154] Example 8 is the method of Example 1, which may optionally include determining a location of a sign on a touchscreen of the computing device, receiving a user selection via the touchscreen proximate to the location, and in response to receiving the user selection, disabling display of the sign if the sign is currently displayed on the touchscreen or enabling display of the sign if the sign is not currently displayed on the touchscreen.

[0155] Example 9 is a method implemented by a computing device, the method including the steps of receiving ultrasound images including a blood vessel; determining a diameter and a position of the blood vessel in a subset of the ultrasound images using a neural network implemented at least in part in hardware of the computing device; determining a failure to detect for one of the blood vessels in one of the ultrasound images not included in the subset, where the failure to detect indicates an absence of a diameter or a position determined by the neural network for one of the blood vessels in one of the ultrasound images; and displaying, in one of the ultrasound images, an indication of the diameter and the position of one of the blood vessels in one of the ultrasound images in response to the determination of the failure to detect, where the indication is based on the diameter and the position determined by the neural network for the subset of the ultrasound images.

[0156] Example 10 is the method of Example 9, which may optionally include receiving a user selection via a touch screen of a computing device proximate to the sign, and disabling display of the sign in response to receiving the user selection.

[0157] Example 11 is the method of Example 10, which may optionally include receiving a further user selection via a touch screen of a computing device proximate to the location indicated by the indicator, and enabling display of the indicator in response to receiving the further user selection.

[0158] Example 12 is the method of Example 9, which may optionally include, where the displaying step includes displaying a marker in one of the blood vessels in a first format and displaying additional markers in additional blood vessels of the blood vessels in a second format to indicate a detection failure.

[0159] Example 13 is the method of example 12, which can optionally include, where the first format of the indicia includes the container with a dashed line and the second format of the further indicia includes the container with a solid line.

[0160] Example 14 is the method of example 12, which can optionally include: the first format of the indicia includes a first color; and the second format of the further indicia includes a second color.

[0161] Example 15 is the method of example 14, which can optionally include the second color corresponding to a standard color representing a catheter size and the first color not indicating a standard catheter size.

[0162] Example 16 is a method implemented by a computing device, the method including the steps of receiving ultrasound images including a blood vessel while an ultrasound probe is moving, the ultrasound images being based on ultrasound echo signals received by the ultrasound probe while it is moving, determining an ultrasound image of one of the ultrasound images based on a position of the ultrasound probe relative to the blood vessel when the ultrasound image is received, determining a feature of the blood vessel in the ultrasound images, and displaying an indication of the feature of the blood vessel in a further ultrasound image of the one of the ultrasound images.

[0163] Example 17 is the method of example 16, which may optionally include, the features include a diameter of a blood vessel in the ultrasound image and the markings include a bounding vessel representing the diameter.

[0164] Example 18 is the method of example 16, which may optionally include, the features including a location of a blood vessel within the ultrasound image.

[0165] Example 19 is the method of example 16 that can optionally include, where the further ultrasound image is a most recent ultrasound image of the ultrasound images.

[0166] Example 20 is the method of example 16, optionally including displaying additional indicia in the additional ultrasound image to guide movement of the ultrasound probe to a location.

[0167] Some portions of the above detailed descriptions are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0168] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities.Unless otherwise noted, as will be apparent from the following description, throughout the description, descriptions utilizing terms such as "processing" or "computing" or "calculating" or "determining" or "displaying" will be understood to refer to operations and processes of a computer system or similar electronic computing device that manipulates and converts data represented as physical (electronic) quantities in the computer system's registers and memory into other data similarly represented as physical quantities in the computer system's memory or registers or other such information storage, transmission or display devices.

[0169] The present invention also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored on a computer-readable storage medium, such as any type of disk, including, but not limited to, a floppy disk, optical disk, CD-ROM, and magneto-optical disk, read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic or optical card, or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus.

[0170] The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will appear from the description below. Further, the present invention is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages ​​can be used to implement the teachings of the present invention as described herein.

[0171] A machine-readable medium includes any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer). For example, machine-readable media includes read-only memory ("ROM"), random-access memory ("RAM"), magnetic disk storage media, optical storage media, flash memory devices, electrical, optical, acoustical or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.), and the like.

[0172] Although many variations and modifications of the present invention will become apparent to those skilled in the art after reading the foregoing description, it should be understood that any particular embodiments shown and described by way of illustration are not intended to be considered limiting in any way. Accordingly, references to details of various embodiments are not intended to limit the scope of the claims, which in themselves recite only those features deemed essential to the invention.

Claims

1. 1. An ultrasound system comprising: Memory, at least one processor coupled to the memory; The at least one processor: receiving an ultrasound image including a blood vessel; determining a diameter of the blood vessel in the ultrasound images using a neural network implemented at least in part in hardware of the ultrasound system, the diameters including a first diameter of the blood vessel in a first one of the ultrasound images and a second diameter of the blood vessel in a second one of the ultrasound images, the first ultrasound image corresponding to a first probe pressure on the blood vessel and the second ultrasound image corresponding to a second probe pressure on the blood vessel that is different from the first probe pressure; determining an amount of movement of an ultrasound probe between generating the first ultrasound image and the second ultrasound image; comparing the amount of movement to a threshold movement value; determining a current diameter of the blood vessel based on the first diameter and the second diameter, the determining step including applying one of a first function and a second function to the first diameter and the second diameter based on the comparison; selecting a color corresponding to a change in shape of the blood vessel based on the current diameter of the blood vessel; displaying the blood vessels with a marker having the color in one of the ultrasound images; 13. An ultrasound system configured to:

2. the first function is one of a mean, a median, a mode, a maximum, or a minimum of the first diameter and the second diameter of the blood vessel; The ultrasound system of claim 1 .

3. the second function being one of a mean, a median, a mode, a maximum, or a minimum of the first diameter and the second diameter of the blood vessel; The ultrasound system of claim 1 .

4. The at least one processor: selecting the first function as the maximum of the first diameter and the second diameter of the blood vessel if the difference is less than the threshold movement value; selecting the second function as an average of the first and second diameters of the blood vessel if the difference is greater than or equal to the threshold shift value; 10. The ultrasound system of claim 1 further configured to perform:

5. The at least one processor is further configured to perform the step of determining a catheter size based on the third diameter of the blood vessel; the step of selecting a color includes matching the color to a standard color corresponding to the catheter size. The ultrasound system of claim 1 .

6. The at least one processor is further configured to perform the step of determining whether the blood vessel is a vein or an artery; The step of displaying comprises: enabling display of the marker having a color similar to that of the blood vessel if the blood vessel is determined to be a vein; or prohibiting display of said marking having said color if said blood vessel is determined to be an artery; 10. The ultrasound system of claim 1 , comprising:

7. The at least one processor: determining whether the blood vessel is a vein or an artery; if said blood vessel is determined to be an artery, indicating said blood vessel in one of said ultrasound images with a further label distinguishing between arteries and veins; 10. The ultrasound system of claim 1 further configured to perform:

8. The at least one processor: determining a location of the blood vessel on a touch screen of the ultrasound system; receiving a user selection via the touch screen proximate to the location; In response to said receiving the user selection, Disabling the display of the indicia if the indicia is currently displayed on the touch screen; or enabling the display of the indicia if the indicia is not currently displayed on the touch screen; 10. The ultrasound system of claim 1 further configured to perform:

9. 1. An ultrasound system comprising: Memory, at least one processor coupled to the memory; The at least one processor: receiving an ultrasound image including a plurality of blood vessels; determining diameters and locations of the plurality of blood vessels within the subset of ultrasound images using a neural network implemented at least in part in the hardware of the ultrasound system; determining a missed detection for a first blood vessel of the plurality of blood vessels in a first ultrasound image, the first ultrasound image being one of the ultrasound images not included in the subset, the missed detection indicating a lack of a diameter or a location determined by the neural network for the first blood vessel in the first ultrasound image; responsive to determining the failed detection, displaying an indication of a diameter and location of the first blood vessel in the first ultrasound image, the indication being based on the diameter and location determined by the neural network for the subset of the ultrasound images; 13. An ultrasound system configured to:

10. The at least one processor: receiving a user selection via a touch screen of the ultrasound system proximate to the landmark; responsive to said receiving said user selection, disabling display of said indicia; 10. The ultrasound system of claim 9, further configured to perform:

11. The at least one processor: receiving a further user selection via the touch screen proximate to a location indicated by the indicium; in response to said receiving said further user selection, enabling display of said indicia; 11. The ultrasound system of claim 10, further configured to perform:

12. 10. The ultrasound system of claim 9, wherein the displaying step includes displaying the marker in the first blood vessel in a first format and displaying additional markers in additional blood vessels of the plurality of blood vessels in a second format to indicate the missed detection.

13. 13. The ultrasound system of claim 12, wherein the first format of the indicia includes a dashed vessel and the second format of the further indicia includes a solid vessel.

14. 13. The ultrasound system of claim 12, wherein the first format of the indicia includes a first color and the second format of the further indicia includes a second color.

15. 15. The ultrasound system of claim 14, wherein the second color corresponds to a standard color representing a catheter size and the first color does not indicate a standard catheter size.

16. 1. An ultrasound system comprising: Memory, at least one processor coupled to the memory; The at least one processor: receiving an ultrasound image including a blood vessel while an ultrasound probe is moving, the ultrasound image being based on ultrasound echo signals received by the ultrasound probe during the movement; determining a first one of the ultrasound images based on a position of the ultrasound probe relative to the blood vessel when the ultrasound images are received, the first ultrasound image including an indication of a first probe pressure for the blood vessel having a first diameter; determining characteristics of the blood vessel in a further one of the ultrasound images, the characteristics comprising a second diameter; determining an amount of movement of an ultrasound probe between generating the first ultrasound image and the further ultrasound image; comparing the amount of movement to a threshold movement value; determining a current diameter of the blood vessel based on the first diameter and the second diameter and the comparison; displaying an indication of the feature of the blood vessel in the further one of the ultrasound images; is configured to run the indicia in the further ultrasound image indicating a second probe pressure on the blood vessel that is different from the first probe pressure; the indicia changes based on a change from the first probe pressure to the second probe pressure; the indicia in the further ultrasound image includes a color indicative of the current diameter of the blood vessel; determining the current diameter of the blood vessel includes applying one of a first function and a second function to the first diameter and the second diameter based on the comparison; The color corresponds to a change in the shape of the blood vessel. Ultrasound systems.

17. The ultrasound system of claim 16 , wherein the indicia include a bounding container.

18. The ultrasound system of claim 16 , wherein the features include a location of the blood vessel within the further ultrasound image.

19. 17. The ultrasound system of claim 16, wherein the further ultrasound image is a current ultrasound image of the ultrasound image.

20. 17. The ultrasound system of claim 16, wherein the at least one processor is configured to perform the step of displaying additional indicia in the additional ultrasound image to guide movement of the ultrasound probe to the location.

Images