Visualization of blood vessels in ultrasound images

Abstract

To provide an ultrasonic system for identifying blood vessels in ultrasonic images and displaying the blood vessels in the ultrasonic images.SOLUTION: An ultrasonic system is configured to execute the steps of: receiving ultrasound images that include a blood vessel; determining, with a neural network implemented at least partially in hardware of the ultrasonic system, diameters of the blood vessel; determining a movement quantity of un ultrasound probe while an ultrasonic image is generated; comparing the movement quantity with a threshold movement value; determining a current diameter of the blood vessel on the basis of the diameter; selecting a color corresponding to a change in shape of the blood vessel on the basis of the current diameter of the blood vessel; and displaying the blood vessel with a colored indicator in one of the ultrasound images.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] Related Applications This application claims the benefit of priority of U.S. Non-Provisional Application No. 17 / 239,335, entitled "Identifying Blood Vessels in Ultrasound Images", filed on April 23, 2021, the disclosure of which is hereby incorporated by reference in its entirety.

[0002] One or more exemplary embodiments relate to an ultrasound machine and a method of operating the same, and more particularly, to an ultrasound machine that identifies blood vessels in an ultrasound image and highlights and displays the blood vessels by including information about the blood vessels useful to an operator of the ultrasound machine.

Background Art

[0003] An ultrasound system radiates an ultrasonic signal generated from an ultrasound probe to an object such as a patient and receives an echo signal reflected from inside the object. An image of the inside of the object is generated using the received echo signal. More specifically, an ultrasonic diagnostic machine generates an ultrasonic image by using ultrasonic image data obtained from an ultrasound probe, and provides the generated ultrasonic image to a user by displaying it on a screen. The ultrasound machine can include a control panel for controlling the ultrasound machine to set various functions such as gain or frequency settings.

[0004] 1] One procedure in which ultrasound machines are frequently used is ultrasound-guided insertion of interventional devices, such as peripheral intravenous (PIV) catheterization. When performing ultrasound-guided PIV catheterization, clinicians can first use an ultrasound machine to perform a survey scan of the body to locate a suitable vein for cannula insertion. A suitable vein has the following characteristics: generally larger than 0.3 mm in diameter; generally not located near an artery that could be accidentally damaged during cannula insertion; sufficiently large so that the diameter ratio of the catheter to the vein is greater than 0.45 (although other rules of thumb exist, such as 0.33); the vein is relatively straight and not tortuous; and the vein does not contain a valve that the catheter will strike.

[0005] To determine the diameter and depth of veins within a patient, clinicians may stare at the measurements (e.g., estimate), which is prone to errors, or they may turn on a caliper to obtain more accurate diameter and depth measurements. In practice, clinicians rarely use a caliper because it is time-consuming and they do not want to touch the ultrasound machine while performing this procedure (for both sterility and workflow reasons). Therefore, clinicians need to be able to easily see and identify suitable vessels for inserting catheters, needles, etc. [Overview of the project]

[0006] In some embodiments, an ultrasound system is provided, comprising a memory and at least one processor coupled to the memory, the at least one processor configured to perform the steps of: receiving an ultrasound image including a blood vessel; determining the diameter of a blood vessel in the ultrasound image using a neural network at least partially implemented in the hardware of the ultrasound system, wherein the diameter includes a first diameter of the blood vessel in a first ultrasound image of the ultrasound images and a second diameter of the blood vessel in a second ultrasound image of the ultrasound images, where the first ultrasound image corresponds to a first probe pressure on the blood vessel and the second ultrasound image corresponds to a second probe pressure on a blood vessel different from the first probe pressure; determining the amount of movement of an ultrasound probe between generating the first ultrasound image and the second ultrasound image; comparing the amount of movement with a threshold movement value; determining the current diameter of the blood vessel based on the first and second diameters, comprising applying one of a first function and a second function to the first and second diameters based on the comparison; selecting a color corresponding to a change in the shape of the blood vessel based on the current diameter of the blood vessel; and displaying the blood vessel with a colored marker in one of the ultrasound images.

[0007] In some embodiments, an ultrasound system is provided, comprising a memory and at least one processor coupled to the memory, wherein the at least one processor is configured to perform the steps of: receiving an ultrasound image including a plurality of blood vessels; determining the diameter and location of a plurality of blood vessels in a subset of ultrasound images using a neural network at least partially implemented in the hardware of the ultrasound system; determining a detection failure for a first blood vessel, which is one of the plurality of blood vessels, in a first ultrasound image, which is one of the ultrasound images not included in the subset, wherein the detection failure indicates the absence of a diameter or location determined by the neural network for the first blood vessel in the first ultrasound image; and, in response to the determination of a detection failure, displaying a label for the diameter and location of the first blood vessel in the first ultrasound image, wherein the label is based on the diameter and location determined by the neural network for the subset of ultrasound images.

[0008] In some embodiments, an ultrasound system includes a memory and at least one processor coupled to the memory, the at least one processor comprising: receiving an ultrasound image including a blood vessel while an ultrasound probe is moving, wherein the ultrasound image is based on ultrasound echo signals received while the ultrasound probe is moving; determining a first ultrasound image from among the ultrasound images based on the position of the ultrasound probe relative to the blood vessel when the ultrasound image is received, wherein the first ultrasound image includes marking of a first probe pressure of a blood vessel having a first diameter; determining features of a blood vessel in a further ultrasound image from among the ultrasound images, including a second diameter; and generating the first ultrasound image and the further ultrasound image. An ultrasound system is provided which is configured to perform the steps of determining the amount of movement of an ultrasound probe, comparing the amount of movement with a threshold movement value, determining a first diameter and a second diameter, and determining the current diameter of a blood vessel based on the comparison, and displaying a label of the blood vessel features in a further ultrasound image among the ultrasound images, wherein the label in the further ultrasound image indicates a second probe pressure for a blood vessel different from the first probe pressure, the label changes based on the change from the first probe pressure to the second probe pressure, the label in the further ultrasound image includes a color indicating the current diameter of the blood vessel, and the step of determining the current diameter of the blood vessel includes applying one of a first function and a second function to the first diameter and the second diameter based on the comparison, and the color corresponds to a change in the shape of the blood vessel.

[0009] The present invention will be better understood from the detailed description and accompanying drawings given below of various embodiments of the invention, but these are for illustrative and illustrative purposes only and should not be construed as limiting the invention to specific embodiments. [Brief explanation of the drawing]

[0010] [Figure 1A] This example shows how to detect two blood vessels and identify them using boxes. [Figure 1B] This example shows how to detect two blood vessels and identify them using boxes. [Figure 2A] This shows the detection of vein contours. [Figure 2B] This shows the detection of vein contours. [Figure 3] An example of an ellipse with intersecting diameters is shown. [Figure 4A] This image shows a single image where no probe pressure (compression) is applied. [Figure 4B] This image shows a single image with probe pressure (compression) applied. [Figure 5A] The detected veins are highlighted. [Figure 5B] The detected veins are highlighted. [Figure 6] An example of a vein diameter used for calculations is shown. [Figure 7] This diagram shows a flowchart of one embodiment of the process for displaying veins on an ultrasound image. [Figure 8] (A) An example of the process for displaying blood vessels is shown. (B) An example of the process for displaying blood vessels is shown. (C) An example of the process for displaying blood vessels is shown. (D) An example of the process for displaying blood vessels is shown. [Figure 9] This is a flowchart illustrating one embodiment of a process for manipulating the display of blood vessels (or other body structures) on a monitor. [Figure 10] (A) Shows one embodiment of a process that displays images of previous detection results in response to the sudden disappearance of detection results. (B) Shows one embodiment of a process that displays images of previous detection results in response to the sudden disappearance of detection results. (C) Shows one embodiment of a process that displays images of previous detection results in response to the sudden disappearance of detection results. [Figure 11A] This shows an example of ultrasound image enhancement generated when the probe is moving. [Figure 11B] This shows an example of ultrasound image enhancement generated when the probe is moving. [Figure 11C] This shows an example of ultrasound image enhancement generated when the probe is moving. [Figure 11D]An example of enhanced display of an ultrasonic image generated when the probe is moving is shown. [Figure 12] A flowchart of one embodiment of a process for displaying one or more blood vessels (or other body structures) as detection results on a monitor together with at least one virtual detection result is shown. [Figure 13A] The period during which the pressure by the probe rises to a specific level and the shape of the blood vessel changes from a normal shape (e.g., elliptical) to another shape (e.g., collapsed elliptical) (the blood vessel collapses) is shown. [Figure 13B] The period during which the pressure by the probe rises to a specific level and the shape of the blood vessel changes from a normal shape (e.g., elliptical) to another shape (e.g., collapsed elliptical) (the blood vessel collapses) is shown. [Figure 13C] The period during which the pressure by the probe rises to a specific level and the shape of the blood vessel changes from a normal shape (e.g., elliptical) to another shape (e.g., collapsed elliptical) (the blood vessel collapses) is shown. [Figure 13D] The period during which the pressure by the probe rises to a specific level and the shape of the blood vessel changes from a normal shape (e.g., elliptical) to another shape (e.g., collapsed elliptical) (the blood vessel collapses) is shown. [Figure 14A] The period during which the pressure by the probe drops from a specific level to a released state and the shape of the blood vessel changes from another shape to the original normal shape is shown. [Figure 14B] The period during which the pressure by the probe drops from a specific level to a released state and the shape of the blood vessel changes from another shape to the original normal shape is shown. [Figure 14C] The period during which the pressure by the probe drops from a specific level to a released state and the shape of the blood vessel changes from another shape to the original normal shape is shown. [Figure 14D] The period during which the pressure by the probe drops from a specific level to a released state and the shape of the blood vessel changes from another shape to the original normal shape is shown. [Figure 15] It includes a graph showing the relationship between blood vessel diameter and time. [Figure 16]A flowchart of one embodiment of a process for creating detection results in another period using detection results of a certain period is shown. [Figure 17] An example of an image obtained by superimposing the best image of the detected blood vessel on the image is shown. [Figure 18A] An example of guide information together with information regarding the best probe position is shown. [Figure 18B] An example of guide information together with information regarding the best probe position is shown. [Figure 19A] A flowchart of one embodiment of a process for displaying a blood vessel on an ultrasonic image is shown. [Figure 19B] A flowchart of another embodiment of a process for displaying a blood vessel on an ultrasonic image is shown. [Figure 19C] A flowchart of yet another embodiment of a process for displaying a blood vessel on an ultrasonic image is shown. [Figure 20] A flowchart of still another embodiment of a process for displaying a blood vessel on an ultrasonic image is shown. [Figure 21] A flowchart of a further embodiment of a process for displaying a blood vessel on an ultrasonic image is shown. [Figure 22] A block diagram of one embodiment of an ultrasonic machine is shown. [Figure 23] A diagram showing an example of a handheld ultrasonic machine. [Figure 24] A data flow diagram of a blood vessel identification display subsystem is shown.

BEST MODE FOR CARRYING OUT THE INVENTION

[0011] In the following description, numerous specific details are set forth in order to provide a more thorough explanation of the present invention. However, as will be apparent to one skilled in the art, the present invention may be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form rather than in detail in order to avoid obscuring the present invention.

[0012] Overview Techniques for identifying and displaying blood vessels in ultrasound images are disclosed. In some embodiments, these techniques are performed by an ultrasound machine. Examples of such ultrasound machines are described in more detail below. For the purposes of this specification, the terms “ultrasound machine,” “ultrasound system,” and “ultrasound imaging system” may be used interchangeably.

[0013] In some embodiments, the ultrasound machine runs detection software to perform vascular detection in a region of the ultrasound image, and the detection results are displayed on the ultrasound machine's monitor or display, such as a clinical display or tablet coupled to the ultrasound machine. Execution can be performed by one or more processors or execution engines. In some embodiments, the ultrasound machine performs vascular detection using template matching, artificial intelligence (AI) or machine learning (e.g., adaptive boosting (adaboost), deep learning, supervised learning models, support vector machines (SVM), sequence models including recurrent neural networks (RNN), gated recurrent units (GRU), convolutional GRU (ConvGRU), long-term short-term memory (LSTM), etc. for processing sequential frame information), and / or other suitable detection methods. In addition to or instead of this, the ultrasound machine can run AI algorithms and / or use neural networks to identify veins and arteries and position them in the ultrasound image.

[0014] In some embodiments, after detection software detects blood vessels, the ultrasound machine displays the blood vessels on the ultrasound system's monitor or display. In some embodiments, the ultrasound machine displays the detected blood vessels in an highlighted manner to provide information to the operator, e.g., the user of the ultrasound machine. For example, the ultrasound machine may draw an outline or other form of blood vessel indication (e.g., an identifier) ​​around or near the blood vessels.

[0015] By highlighting blood vessels, additional information is provided to the operator. For example, the outline of a vein can be changed to match the color coding of the largest catheter that can fit into that vein, up to a catheter-to-vein diameter ratio (e.g., 0.45 or 0.33). Thus, the operator can select a catheter size based on the blood vessel and their displayed markings. In addition to or alternatively, the ultrasound machine can identify all veins in the ultrasound image that are suitable for a particular catheter size. For example, the operator can select a catheter size on the ultrasound machine's user interface, and in response, the ultrasound machine can specify veins in the ultrasound image that are suitable for the catheter size by changing the color of the markings for veins that are suitable for the catheter size, removing the markings for veins that are not suitable for the catheter size, or a combination thereof (e.g., based on the ratio of catheter size to vein diameter). In one example, the operator can touch a vein in the ultrasound image and have the ultrasound machine display the diameter and depth of that blood vessel. In addition to or alternatively, the ultrasound machine can automatically identify the most central and shallowest vein and automatically provide its diameter and depth on the ultrasound machine's display.

[0016] In some embodiments, specific tissues of arteries and veins can be detected as integrated structures. For example, a triad is a vein-artery-vein cluster where the central artery is closely bounded on both sides by veins. In one example, AI can clearly detect and classify the veins and arteries within a triad as individual components. However, due to the inherent configuration of the triad structure, this grouping can be detected as an additional inherent classification along the veins and arteries. By doing so, the overall accuracy of detection can be improved because triads always occur in a vein-artery-vein configuration. A group of three closely detected vessels, for example, a vein-vein-artery cluster where the central vein is bounded by veins and arteries, is likely to be a misclassification or false detection. By detecting this base as a triad, the accurate classification of each vessel can be improved.

[0017] In some embodiments, the ultrasound machine calculates a likelihood value for each detected vessel as an indicator of the confidence level associated with the detection result. For example, the likelihood value may include a value between 0 and 1 indicating the confidence level of the vessel's classification as artery or vein. In some embodiments, the ultrasound machine generates a label for the vessel (e.g., outline, circle, ellipse, box, etc.) and adjusts the opacity of the label based on the likelihood value representing the confidence level of the prediction for that vessel. This adjustment of opacity provides an improvement over conventional ultrasound systems that do not adjust the opacity of the label based on the confidence level of the label. In one example, the clinician or operator of the ultrasound machine is exposed to additional information useful in selecting vessels for medical procedures such as catheterization. In addition to or instead of this, the opacity adjustment makes the display more visually appealing and less confusing, as the vessels gradually fade in and out over time rather than suddenly appearing and disappearing based on the change in opacity.

[0018] For example, an ultrasound machine can track a blood vessel across multiple frames (for instance, a frame may represent one ultrasound image within a series of ultrasound images). For instance, an ultrasound system can identify a blood vessel in one ultrasound image and track it in subsequent ultrasound images within a series of ultrasound images. The tracking can be based on characteristics of the blood vessel, such as its location, diameter, classification as a vein or artery, or a combination thereof. For example, an ultrasound machine can determine that a blood vessel in the first ultrasound image is the same blood vessel in the second ultrasound image based on one or more characteristics of the blood vessel in the first and second ultrasound images.

[0019] In some embodiments, the ultrasound machine determines when blood vessel detection fails and generates a virtual detection result to display the blood vessel on the ultrasound image. For example, the ultrasound machine may detect a blood vessel in multiple ultrasound images and then fail to detect it in subsequent ultrasound images (e.g., ultrasound images following multiple ultrasound images in a video sequence). The ultrasound machine can compare the detection results of the ultrasound images in the video sequence and, based on the comparison, declare a detection failure in the subsequent ultrasound image. Based on the detection failure, the ultrasound system can generate a virtual detection result, such as a bounding box, and display the virtual detection result in the subsequent ultrasound image to indicate the location of the blood vessel, even though no blood vessel was detected in the subsequent ultrasound image.

[0020] In one example, the ultrasound machine determines the desired entry point of the intervention device. The ultrasound machine then calculates the distance from the transducer surface (e.g., the edge of the probe) to the desired entry point and can display a distance indicator on the ultrasound machine's display. For example, the ultrasound machine may display a message with text containing the distance and / or an arrow indicating the direction in which to move the probe according to the distance.

[0021] Accordingly, the techniques disclosed herein provide for identifying blood vessels in ultrasound images and displaying them in several ways to convey useful information to the operator. Therefore, the ultrasound systems disclosed herein may be suitable for medical procedures where conventional ultrasound systems are unsuitable, as conventional ultrasound systems may result in undesirable outcomes for the patient, including discomfort, blood loss from multiple punctures, and the risk of infection.

[0022] While this specification focuses on blood vessels, it should be noted that the technologies and systems disclosed herein are not limited to blood vessels and can be used with other body structures such as nerves, muscles, and skeletal parts. Furthermore, while this specification focuses on peripheral intravenous catheterization, the technologies and systems disclosed herein are not limited to catheters and can be used with any appropriate intervention device such as needles, stents, clamps, and guides.

[0023] Highlight The technologies and systems disclosed herein can detect blood vessels in ultrasound images and generate useful information about the vessels in various ways. In some embodiments, the ultrasound system determines the diameter of a blood vessel based on one or more previous ultrasound images. For example, the ultrasound system can determine the diameter of a blood vessel in each of several previous frames (e.g., two, three, four or more previous frames) and generate the diameter of the blood vessel in the current frame based on the diameters of the blood vessels from the several previous frames. Thus, the ultrasound system can prevent undesirable rapid changes in the labeling of the blood vessels (e.g., the color of the blood vessel boundary box which can correspond to the catheter size), thereby allowing the operator to understand the true size of the blood vessel and take appropriate actions, such as selecting the appropriate size catheter.

[0024] In some embodiments, an ultrasound system generates an ultrasound image containing one or more blood vessels and determines the diameter of the blood vessels in the ultrasound image. The diameter may include the respective diameter of each blood vessel in each ultrasound image (e.g., an ultrasound video stream). The ultrasound system may include a neural network that determines the size and location of the blood vessels. The neural network may be implemented in hardware at least partially of a computing device (e.g., an ultrasound machine, a computing device coupled to an ultrasound machine such as a tablet, or a combination thereof). The ultrasound system can calculate the blood vessel diameter based on the diameter of the blood vessels. Based on the blood vessel diameters of the same blood vessel in multiple images, the ultrasound system can generate blood vessel diameters for use in displaying the blood vessels. The ultrasound machine may select a color based on the blood vessel diameter and then generate a label based on the blood vessel diameter and the selected color to provide a display in at least one ultrasound image so that the blood vessels are displayed with colored labels. Figures 1A to 6 illustrate an example of this process.

[0025] In one example, the process is initiated by a processor in an ultrasound machine that executes an image analysis algorithm to perform image analysis on an ultrasound image, thereby detecting blood vessels in the ultrasound image. The processor can execute two algorithms: one for detecting blood vessels and another for determining the type of blood vessel detected (e.g., whether the vessel is a vein or an artery). The algorithm may include one or more neural networks, such as a first neural network trained to detect the location of blood vessels and a second neural network trained to classify the blood vessels as veins or arteries based on the detection results of the first neural network. Alternatively, these two neural networks can be combined into a single algorithm. In one example, a single neural network containing a first head for detection and a second head for classification is included in a single algorithm, thereby allowing the processor to include and execute a single algorithm.

[0026] Figure 1A shows an example of detecting two blood vessels and identifying them, respectively, with boxes 101 and 102 having characteristics such as color, line width, and line shape (e.g., solid or dashed). Boxes 101 and 102 are examples of vessel boundary containers in Figure 1A and can be generated by the ultrasound system's processor using information from a neural network. The ultrasound system can then classify the blood vessels as veins or arteries, for example, using an additional neural network. After classifying the blood vessels as veins or arteries, the ultrasound system can modify the characteristics of one or both of boxes 101 and 102 to function as classification markers. In the example in Figure 1B, after the ultrasound system has classified both blood vessels as veins, the ultrasound machine changes the colors of boxes 101 and 102 in Figure 1B (represented by hashes) compared to the colors of boxes 101 and 102 in Figure 1A.

[0027] In one example, the ultrasound system determines the contour of the detected blood vessel and approximates the determined contour as an ellipse. For example, the ultrasound system can match the ellipse to one or both of boxes 101 and 102. In addition to or instead of this, the ultrasound system may include implementations of binarization, contour detection, or segmentation techniques for determining the contour of the blood vessel.

[0028] Figures 2A and 2B illustrate the detection of blood vessel (e.g., vein) contours. Referring to Figure 2A, two blood vessels 201 and 202 are shown. Contouring by binarization / edge detection can be performed inside the selected box-shaped region (not shown) in the image of Figure 2A. In addition to or instead of this, the ultrasound system can match inner ellipses 211 and 212 to the box, as shown in Figure 2B. In these examples, the ultrasound system generates an oval shape as the boundary container for the blood vessel (e.g., vein).

[0029] An ultrasound system can determine the diameter of a blood vessel using an ellipse generated as a boundary vessel. Figure 3 shows the major and minor axes (the major / minor axes 301 and 302 of the two blood vessels together). The ultrasound system can determine the diameter of one blood vessel from the major / minor axis 301 and the diameter of the other blood vessel from the major / minor axis 302. For example, the ultrasound system can determine the diameter of the blood vessel on the left side of Figure 3 as the blood vessel diameter by averaging the major and minor axes of the major / minor axis 301, and determine the diameter of the blood vessel on the right side of Figure 3 as the blood vessel diameter by averaging the major and minor axes of the major / minor axis 302. In addition to or instead of this, the ultrasound system can generate the blood vessel diameter as the blood vessel diameter from a weighted average of the major and minor axes of an ellipse (e.g., an ellipse) representing the boundary vessel of the blood vessel. For example, when forming the average, the ultrasound system can weight one axis more heavily than the other axis.

[0030] In one example, an ultrasound system determines the diameter of a blood vessel by matching the properties of an ellipse to those of a circle, using the major and minor axes of the ellipse. For example, an ultrasound system can determine the diameter of a blood vessel as the diameter of a circle having the same area as the ellipse representing the boundary vessel of the blood vessel. Let the major axis of the ellipse be d1 and the minor axis be d2. The ultrasound system determines the diameter d of the blood vessel as the diameter of a circle having the same area as the ellipse, that is,

number

number

[0031] In addition to or instead of this, the ultrasound system can determine the diameter of a blood vessel as the diameter of a circle having the same circumference as the ellipse representing the boundary vessel of the blood vessel. In this example, the ultrasound system,

number

number

[0032] In some embodiments, the ultrasound system highlights and displays arteries using a different marker than the one used to detect and identify veins. For example, the marker used for arteries may be a red circle with a red line crossing the circle to inform the operator that veins should not be selected for arteries for the desired procedure. In some embodiments, after detecting an artery, the operator of the ultrasound machine can turn off the calculation of the arterial diameter and therefore the calculation of the arterial vessel diameter. In addition to or instead of this, the operator can individually (e.g., sequentially) enable or disable the highlighting of veins or arteries, for example, by first disabling the indicator for vein or artery classification, and then disabling the displayed diameter of the vein or artery.

[0033] In some embodiments, the ultrasound system applies tracking techniques to detected vessels to determine whether the same vessel appears in multiple ultrasound images. For example, the tracking technique can use the diameter and location of the detected vessel. In such cases, the vessel diameter is calculated in advance before tracking. However, in other embodiments, tracking of detected vessels is performed before calculating the vessel diameter. Since the vessel diameter is calculated based on two cross diameters, it can be calculated before and after tracking, and tracking only requires the calculation of at least these two diameters, and does not necessarily require the vessel diameter.

[0034] Furthermore, by using tracking techniques, the current diameter value of a vessel can be calculated from a predetermined vessel diameter. For example, the ultrasound system can set the current vessel diameter of a vessel to the maximum value of a previously determined vessel diameter of a vessel that appeared in multiple previous ultrasound images, such as the previous three to five images. Since tracking techniques allow for the determination of the same vessel that persists throughout a series of ultrasound images (frames) (e.g., ultrasound), in some embodiments, the maximum value is calculated for the same vessel. Figures 4A and 4B represent two images in a time series of probe pressure. Figure 4A contains two veins, vein 401 and vein 402. Due to the pressure from the probe, vein 401 disappears in Figure 4B, and as a result, the ultrasound system identifies only vein 402 as the same vein between Figure 4A and Figure 4B. Thus, the ultrasound system can determine the current value of the vessel diameter of vein 402 in Figure 4B from one or more values ​​of the vessel diameter of identified vein 402 in a predetermined number of previous images, including Figure 4A. For example, if the vessel diameter of vein 402 in Figure 4B is 1.5 mm, and the maximum vessel diameter of vein 402 in a previous image (e.g., Figure 4A) is 2.0 mm, the ultrasound system can set the current diameter of vein 402 in Figure 4B to 2.0 mm. A new vein 403 is also shown in Figure 4B and not in Figure 4A. Therefore, the ultrasound system can determine the vessel diameter of vein 403 in Figure 4B without using the vessel diameter of vein 403 in the previous image.

[0035] In some embodiments, tracking uses at least one type of information relating to points, lines, or areas of a vascular region. For example, in some embodiments, a point in a vascular region is tracked between frames, and the ultrasound system measures the movement of the point. In the case of a point, the information about the point may include the center, top, bottom, left, right, or centroid of the vascular region, and the difference in the position of the same point between two ultrasound images is evaluated. Information other than point information may be used for tracking. Examples of other information that can be used for tracking include information about lines (e.g., diameter of the vascular region, or length of the contour) and information about regions (e.g., shape, area, illumination information (texture information) of the vascular region). If the movement is within a threshold distance (e.g., less than 2 mm), the ultrasound system considers the vascular region in the frame to be the same vascular region. In some embodiments, the threshold distance is equal to a percentage (e.g., half) of the diameter of the vascular region in the immediately preceding frame (or another previously generated frame), or the vascular diameter (e.g., the average diameter) of the vascular region in a set of two or more previously generated consecutive frames.

[0036] In some embodiments, the ultrasound system displays detected blood vessels superimposed with enhancements such as color enhancement. The ultrasound system can determine the color for color enhancement according to the diameter value determined by the ultrasound system for blood vessels. Figures 5A and 5B show the same ultrasound images with different color enhancements on the detected veins on the ultrasound system's display. The images were obtained, for example, under pressure from the probe. Specifically, Figure 5A shows the display of the detected veins considering only the blood vessel diameter of the current image, and Figure 5B shows the detected veins with color enhancements related to the blood vessel diameter calculated according to some embodiments. Assume that (i) the calculated blood vessel diameters of veins 501A and 501B are 1.5 mm, and (ii) the configuration of the enhancement rule in the ultrasound system is that if the current value of the blood vessel diameter is 2.0 mm or greater, the vein is surrounded by a color (e.g., pink), and if that value is less than 2.0 mm, the vein is surrounded by a different color (e.g., orange). In Figure 5A, vein 501A is highlighted in orange (represented by a solid white oval), while in Figure 5B, vein 501B is highlighted in pink (represented by hatching). The reason vein 501B is surrounded in pink is that, in one embodiment, the ultrasound system is configured to calculate the vessel diameter from a previous image that includes a maximum vessel diameter of 2.0 mm. In an image including such a maximum vessel diameter, the shape of the same vein, for example, 501A and 501B, becomes more circular. Therefore, the highlighting in Figure 5B corresponds to a better representation of the change in vessel shape than that in Figure 5A. In other words, embodiments such as those in Figure 5B are useful, for example, when the probe releases and / or compresses an object under an ultrasound scan so that the shape of the vessel changes gradually or abruptly.

[0037] In some embodiments, the vessel diameter is calculated for subsequent images (frames). In one example, the ultrasound system can detect a vein, perform calculation of only the vein's vessel diameter for subsequent images (frames), and then highlight only the vein, thus facilitating the understanding of the characteristics of the vessels in the ultrasound image. Figure 6 shows an example of the vessel diameter used for calculation. Referring to Figure 6, the ultrasound system identifies a vein and calculates the diameter of vein 402 for one frame (image) using the indicated axis. Artery 403 has a diameter, and the vessel diameters of vein 402 and artery 403 are not calculated for subsequent frames.

[0038] In some embodiments, the ultrasound system uses past images (e.g., previously generated and displayed ultrasound images) to determine vascular characteristics such as vein size. By using past images, the ultrasound system can determine when a blood vessel is in a normal state (e.g., not compressed by the probe). For example, the ultrasound system can determine an image from multiple ultrasound images that contains the blood vessel with the largest diameter, and use that maximum diameter to determine the blood vessel diameter, thus allowing it to select a blood vessel of the correct size when determining its diameter. In addition to or instead of this, at least three past frames or at least half a frame can be used to determine when a blood vessel is of its correct size, although this number may be greater or less. Thus, the ultrasound system can determine the actual (or true) value of the blood vessel diameter in its normal state, and thus select the correct color (e.g., pink in Figure 5B according to diameter) to identify the blood vessel. This can mitigate the effects of pressure from the probe.

[0039] Figure 7 shows a flowchart of one embodiment of a process for displaying veins on an ultrasound image. This process can be carried out by processing logic that may include hardware (e.g., circuits, dedicated logic, etc.), software (e.g., run on a general-purpose computer system or dedicated machine), firmware (e.g., software programmed in read-only memory), or a combination thereof. In some embodiments, the process is carried out by the processor of an ultrasound machine. Although Figure 7 shows the display of veins on an ultrasound image, this process can be used to display arteries in addition to or instead of this.

[0040] Referring to Figure 7, the process begins with the processing logic acquiring an image frame, such as an ultrasound image (processing block 701), and performing vein (or blood vessel) detection on the image frame (processing block 702). Using the results of the vein detection, the processing logic extracts the boundaries of the veins (processing block 703) and calculates the blood vessel diameter of the detected veins (processing block 704).

[0041] After calculating the vessel diameter, the processing logic performs tracking (processing block 705). During tracking, the processing logic determines whether each detected vein exists in past frames (processing block 706). If not, the process proceeds to processing block 708. If it does, the processing logic calculates the maximum vessel diameter of that vein as the current value of the vessel diameter based on its vessel diameter in the previous frame (processing block 707), and proceeds to processing block 708.

[0042] In processing block 708, the processing logic determines whether all veins have been tracked and whether the maximum vascular diameter has been calculated for those veins that have been tracked. If not, the process returns to processing block 705 and continues. If it has, the process moves to processing block 709, where the processing logic displays each vein in the ultrasound image with a color or other label selected based on its maximum diameter.

[0043] In some embodiments, the current value of the vessel diameter is calculated using one of several different methods. For example, the current value of the vessel diameter of a detected vessel can be determined from the maximum, minimum, mean, median, mode, standard deviation, and maximum-minimum values ​​from the vessel diameter of the same vein over the past few frames. In some embodiments, the ultrasound machine determines the calculation algorithm before the examination, or before imaging the vessel, or using any enhancement of the vessel in the image (e.g., labeling, color, etc.). This determination can be part of the ultrasound machine configuration or boot-up process. In some embodiments, the calculation algorithm (e.g., maximum, minimum, mean, etc.) is selected by the ultrasound machine operator.

[0044] One advantage of choosing the standard deviation or maximum-minimum is that a normal vein can be easily compressed and released during pressure applied by the probe, whereas if the vein is not in a healthy state, it will not be as elastic. One advantage of choosing the mean, median, or mode is that it eliminates abnormal values ​​that appear unintentionally or erroneously within one or more frames. Another advantage of choosing the mean, median, or mode is that it is a compromise between two points: (i) smaller gauge catheters cause less damage to the blood vessel, but (ii) smaller gauge catheters also transport less fluid. Furthermore, the mean, median, and mode are always less than the maximum value of the blood vessel, so for example, medical instruments / devices such as needles or catheters, though not limited to, are smaller than the diameter of the blood vessel.

[0045] In one example, the processor is configured to automatically determine the calculation algorithm (e.g., without explicit user selection of the calculation algorithm). In some embodiments, the selection is based on whether the probe is compressing the subject being examined (e.g., determined by a pressure sensor in the probe) and thereby reducing the size of the blood vessel. In one such embodiment, (i) if the probe is pressing on the same part of the object, the processor selects the maximum calculation algorithm, and (ii) if the probe is moving along the blood vessel (e.g., longitudinally), the processor selects the average calculation algorithm. In some embodiments, to distinguish between (i) and (ii), the ultrasound machine performs image analysis to determine whether the area around the detected blood vessel has changed. In addition to or instead of this, position sensors in the probe (e.g., magnetic sensors, accelerometers, gyroscopes, etc.) can be used to distinguish between (i) and (ii).

[0046] In some embodiments, the ultrasound machine marks the displayed blood vessels with colored markers (e.g., circles, ellipses, etc.). In such cases, in some embodiments, the color may be based on whether the detected blood vessel is an artery or a vein. For example, the ultrasound machine may mark arteries with red markers (e.g., a red circle, or an icon of a circle with a horizontal line drawn across it). In addition to or instead of this, the ultrasound system may mark blood vessels with markers that have a color based on what size (e.g., gauge) of needle or catheter can be inserted into the blood vessel. The color may follow a standard color chart that assigns a color specific to the instrument gauge. Thus, in some embodiments, blood vessels with different ranges of diameter (e.g., blood vessel diameter) are highlighted by using different colors. Furthermore, in one example, only veins with different ranges of diameter (e.g., blood vessel diameter) are highlighted using a different color than the highlighting color used for arteries, because in one configuration of the ultrasound system, the blood vessel diameter is calculated only for veins.

[0047] As mentioned above, in some embodiments, the ultrasound machine detects and displays some highlighting on blood vessels, regardless of which blood vessels are of interest to the operator of the ultrasound machine. For example, the highlighting on blood vessels may include one or more highlighted blood vessels that the operator wishes to see, along with one or more other highlighted blood vessels that the operator is not interested in seeing. The operator can input a request to see or not see specific blood vessels through the user interface of the ultrasound machine.

[0048] In some embodiments, the operator can remove unwanted highlighting of blood vessels from the display. For example, the operator may select a detected blood vessel and switch between an ON state, where the detected blood vessel is highlighted and displayed with highlighting (e.g., boundary container, color, color based on type (e.g., vein or artery), color based on the size of the needle that can be inserted into the vessel, text indicating the diameter, stop sign, icons such as thumbs up or thumbs down, or a combination thereof), and an OFF state, where the detected blood vessel is displayed without any highlighting.

[0049] Figures 8A to 8D illustrate an example of the process for highlighting blood vessels. Referring to Figure 8A, blood vessels, including vein 811, artery 812, and vein 813, are shown and highlighted on the monitor 800. In some embodiments, the monitor 800 includes a touch-sensitive screen display or touch-surface display device. In one example, the three blood vessels 811-813 are detected by artificial intelligence (AI). In addition to or instead of this, the detected blood vessels can be marked by the AI ​​with colored markers (e.g., circles or ellipses). When the operator touches the monitor 800 at the location of one of the three blood vessels, such as vein 813 in Figure 8A, the colored circle disappears, as shown in Figure 8B. Furthermore, when the operator touches the monitor 800 at the location of one of the blood vessels previously touched to make the circle disappear, as shown in Figure 8C, the circle reappears on the blood vessel, as shown in Figure 8D.

[0050] Figure 9 shows a flowchart of one embodiment of a process for manipulating the highlighting of blood vessels (or other body structures) on a monitor. This process can be performed by processing logic that may include hardware (e.g., circuits, dedicated logic), software (e.g., running on a general-purpose computer system or dedicated machine), firmware (e.g., software programmed in read-only memory), or a combination thereof. In some embodiments, the process is performed by the processor of an ultrasonic machine.

[0051] Referring to Figure 9, the process begins with the processing logic displaying a highlight on one or more blood vessels (or other body structures) on the ultrasound machine's monitor as a colored circle (or in another display of other highlighting modes) (processing block 901). In some embodiments, the monitor is a touch-sensitive screen type monitor or touch surface display device.

[0052] After one or more blood vessels are highlighted, the processing logic receives an indication that the user has touched the monitor (processing block 902). In addition to or instead of this, the user can make a selection on the monitor by moving a cursor with a cursor control device to a position on the monitor and pressing a select button.

[0053] In response to receiving an indication that the user has touched the monitor, the processing logic determines the location of the touch and whether a blood vessel was highlighted at that location on the monitor when the touch occurred (processing block 903). If so, the processing logic removes the colored circle (or any other marker / highlight used in conjunction with the blood vessel display) displayed with the blood vessel from the display (processing block 904).

[0054] If a blood vessel is not displayed on the monitor at that location when a touch is made, the processing logic determines whether a blood vessel is detected at the touch location but its emphasis is suppressed from the display (processing block 905), and if so, causes the colored circle (or other mark / highlight) to reappear on the monitor (processing block 906).

[0055] In some embodiments, blood vessels are displayed on a monitor, and the process shown in Figure 9 can be repeated when the operator touches the monitor. For example, when the operator moves the ultrasound probe, new blood vessels may appear and previous vessels may disappear from view. The ultrasound machine can repeat the process shown in Figure 9 for newly displayed blood vessels when the operator touches the monitor. In one typical implementation of the ultrasound system, the ultrasound system continuously generates ultrasound images and, regardless of whether the enhancement is on or off, continues the blood vessel detection process and tracking technique (and further, calculation of blood vessel diameter) on the generated images, so that when the operator touches a blood vessel that was not previously highlighted on the monitor, a circle of the appropriate color is displayed around the blood vessel, and the color is further based on standard colors corresponding to the calculated blood vessel diameter and / or catheter size.

[0056] In some embodiments, enhancement to the ultrasound image is performed in the form of indicating undetected blood vessels. For example, if blood vessels are detected in the ultrasound image, a sudden change in the examination situation may cause the detection result (e.g., a colored circle on the detected blood vessel in the ultrasound image) to abruptly disappear. For instance, if the probe is suddenly pressed against the subject during the examination, the blood vessel in the ultrasound image may suddenly collapse, and the detection result may no longer be displayed on the monitor. This lack of detection indication can be problematic for the user, as they may still be able to see the actual location of the collapsed blood vessel in the B-mode image.

[0057] In some embodiments, if the processing logic of the ultrasound machine cannot detect blood vessels in the current image, detection results from one or more previous ultrasound images are retrieved. The processing logic can determine a virtual detection result based on one or more detection results from those previous ultrasound images, and the virtual detection result can be displayed by the ultrasound machine superimposed on the current image. In one embodiment, the ultrasound machine displays only two or three frames of the virtual detection result to prevent flickering.

[0058] In some embodiments, an ultrasound machine determines the diameter and location of blood vessels within a subset of ultrasound images. As previously mentioned, this determination can be made using a neural network or other detection mechanism at least partially implemented in the hardware of a computing device, such as the ultrasound machine itself, a computing device coupled to the ultrasound machine, or a combination thereof. Alternatively, or in addition to the above, the determination can be made using hardware (e.g., circuits, dedicated logic, etc.), software (e.g., running on a general-purpose computer system or a dedicated machine), firmware (e.g., software programmed into read-only memory), or a combination thereof.

[0059] Based on the diameter and location of the blood vessels observed in each image, the ultrasound machine can determine whether a detection failure occurred for one of the blood vessels in one of the ultrasound images because the blood vessel was not present in the area identified by the neural network (or other detection mechanism). In response to this determination, the ultrasound machine can attempt to track previously detected blood vessels by alternative means and display a marker for the diameter and location of one of the blood vessels in one of the ultrasound images. In some embodiments, the marker is based on the diameter and location determined by the neural network for a subset of the ultrasound images.

[0060] Figures 10A to 10C illustrate one embodiment of a process for displaying images of previous detection results in response to the sudden disappearance of detection results. This situation can occur, for example, when the probe is gradually compressing the vessel. If the vessel is further compressed, its shape will collapse. In such cases, it becomes even more difficult to detect such vessel collapse from the current image alone.

[0061] Referring to Figure 10A, vessels 1001 and 1002 are displayed on monitor 800 as a result of the detection process determining their presence. As shown in Figure 10B, vessels 1001 and 1002 continue to be displayed on monitor 800 while detection is ongoing, but vessel 1002 is shown as an ellipse (in contrast to the circular shape of vessel 1002 in Figure 10A). Figure 10C illustrates that since vessel 1002 is not displayed, only vessel 1001 is shown for which detection is available. In other words, Figure 10C reflects the result of the last successful detection of vessel 1002. For example, the ultrasound machine may not be able to detect vessel 1002 in Figure 10C. In such a case, the system displays vessel 1002 as a dashed ellipse representing a hypothetical detection result, even though the detection process did not yield results for vessel 1002. The advantage of displaying virtual results with the same shape as successful results (e.g., solid ellipses versus dashed ellipses, or solid boxes versus dashed boxes) but with different types of contours is to prevent the operator from feeling uncomfortable. In some embodiments, the colors of solid (actual) and dashed (virtual) ellipses can be selected from the calculated blood vessel diameters in previous images. Since the current value of the blood vessel diameter is calculated based on previous images, color variations can be suppressed, and by changing only the shape of the lines, it becomes easier for the operator to identify the same blood vessel (vein).

[0062] More specifically, in some embodiments, the virtual detection result is displayed in a different way from the actual detection result. For example, as shown in Figure 10C, the virtual detection result is displayed as a dashed ellipse, while the actual detection result is displayed in a solid boundary container, as shown for the blood vessel 1002 in Figures 10A and 10B. In addition to or instead of this, different colors, different types of filling for the boundary container, colors, labels, icons, text, or other distinguishing features can be used to distinguish the actual detection result from the virtual detection result. Also note that since the blood vessel 1001 is an artery, its shape will not change as much as the shape of a vein.

[0063] In some embodiments, after identifying a blood vessel in an ultrasound image, the user may want to find the optimal location in the vessel from which a needle or other interventional device can be inserted. For example, blood vessels can extend from the elbow to the wrist, from the upper arm to the elbow, etc., and the operator may want to explore a specific location or position in the blood vessel for needle insertion. However, when exploring the best location in a blood vessel, the operator moves the probe from one location to another, thereby acquiring a series of ultrasound images that can be checked. In such cases, the operator may miss the best location for examination and / or intervention while the probe is moving.

[0064] In some embodiments, ultrasound images including blood vessels are acquired while the ultrasound probe is moving. For example, an ultrasound machine can generate ultrasound images based on ultrasound echo signals received by the ultrasound probe. The ultrasound machine determines at least one ultrasound image based on the position of the ultrasound probe relative to the blood vessel when the ultrasound image was received, and determines one or more features of the blood vessel in the ultrasound image. The ultrasound machine displays a marker for each feature of the blood vessel in a subsequent ultrasound image (for example, in the current ultrasound image).

[0065] Figures 11A–11D show examples of enhancement of ultrasound images generated when the probe is moving. Referring to Figure 11A, blood vessels 1101–1103 are shown on the ultrasound image. Blood vessel 1101 is shown as a vein with a circle of the first color, blood vessel 1102 is shown as an artery, and blood vessel 1103 is not identified as either a vein or an artery (e.g., identified by a dotted circle). Figure 11B shows another ultrasound image of the same location where the shape of blood vessel 1101 is represented as a more elliptical shape with different enhancement (e.g., different color) than in Figure 11A. The shape of blood vessel 1101 may be due to the operator pressing the probe more firmly into the body at that location. Figure 11C shows another ultrasound image of the same location where the shape of blood vessel 1101 is represented as a smaller elliptical shape with different enhancement (e.g., different color) than in Figure 11B. Figure 11D shows another ultrasound image of the same location where the shape of vessel 1101 is represented by a smaller ellipse with different emphasis (e.g., different color) and dotted lines than in Figure 11C. Thus, Figures 11A to 11D illustrate how the emphasis of vessels can change due to probe movement and / or pressure.

[0066] Figure 12 shows a flowchart of one embodiment of a process for displaying one or more blood vessels (or other body structures) as detection results on a monitor having at least one virtual detection result. This process can be carried out by processing logic that may include hardware (e.g., circuits, dedicated logic, etc.), software (e.g., run on a general-purpose computer system or dedicated machine), firmware (e.g., software programmed in read-only memory), or a combination thereof. In some embodiments, the process is carried out by the processor of an ultrasonic machine.

[0067] Referring to Figure 12, the process begins with the processing logic detecting one or more blood vessels from the ultrasound image (processing block 1201). In some embodiments, the processing logic detects blood vessels using neural networks, template matching, machine learning (e.g., sequence models including adaboost, deep learning, SVM, RNN, GRU, ConvGRU, LSTM, etc., for processing frame information in a sequence). In addition to or instead of this, the processing logic may calculate a likelihood value for each blood vessel, indicating the level of confidence that the detection result is a blood vessel (processing block 1202). The likelihood value can be between 0 and 1, where 1 represents a high level of confidence in the detection result and 0 represents a low level of confidence in the detection result.

[0068] The processing logic also tracks detected blood vessels between frames and identifies the same detected blood vessels in different images (processing block 1203). In some embodiments, the processing logic uses location information and / or blood vessel type for the tracking and / or identification process.

[0069] The processing logic compares the likelihood value to a threshold, such as 0.7 (processing block 1204). If the processing logic determines that the likelihood value for the same vessel is below the threshold, the processing logic determines that the vessel detection has failed (processing block 1205) and generates a virtual detection result for display (processing block 1206). In some embodiments, to generate a virtual detection result, the processing logic retrieves the results of one or more previous ultrasound images and uses the results as the basis for displaying the virtual detection result. The virtual detection result can be calculated using a calculation method (algorithm) that is performed in several ways. In some embodiments, the processing logic retrieves the most recent successful detection result and uses it as the virtual detection result. In addition to or instead of this, the processing logic may select the detection result with the highest likelihood value from a predetermined number of previous frames, such as from the previous two, three, four, or more frames, as the virtual detection result. The number of frames may depend at least in part on the acquisition frame rate, with higher frame rates allowing for the selection of more frames. In one example, the processing logic estimates the shape and / or location of a blood vessel based on the results of one or more previous frames generated during a predetermined past time period, such as 500 msec prior to the image frame. In one example, this estimation is performed by using motion vector techniques. In addition to or instead of this, this estimation can be performed by using optical flow techniques. The processing logic then displays the virtual detection results in the ultrasound image along with any other actual or virtual detection results (processing block 1207).

[0070] In some embodiments, the probe of an ultrasonic machine has a sensor for detecting its movement. If the probe moves at a certain speed, the processing logic does not display the virtual detection result on the monitor. This suppression of the display of the virtual detection result is because it is more difficult for the processing logic to estimate the virtual detection result when the probe is moving, for example, because it is not possible to estimate the symmetrical changes in the shape and / or position of the blood vessel when the probe is moving.

[0071] The shape of a blood vessel may change while the region of interest is compressed and released by the probe of the ultrasound system. For example, Figures 13A to 13D show period (i) in which the pressure from the probe increases to a certain level and the shape of the blood vessel changes from its normal shape (usually a circular ellipse) to an additional shape (e.g., a flattened ellipse) (e.g., collapse). Figures 14A to 14D show period (ii) in which the compression from the probe decreases from a certain level to a released state and the shape of the blood vessel returns from the additional shape to a normal shape. During periods (i) and (ii), ultrasound images are collected and blood vessels are detected. In some embodiments, blood vessel detection is performed using AI (e.g., AI software implementing a neural network).

[0072] In some embodiments, the detection results of ultrasound images acquired during period (i) are analyzed, results are created, and used for detection / analysis of ultrasound images during period (ii). For example, the ultrasound system can use the detection results of period (i) as conditional input to a neural network to generate additional detection results for period (ii). In addition to or instead of this, the ultrasound system can use the location and size of each vessel detected during period (i) to predict (e.g., limits) areas for searching for vessels during period (ii), thereby contributing to the rapid and accurate detection of the same vessels during period (ii). In one example, the ultrasound system can generate a region of interest based on the location and size of each vessel detected during period (i) and provide this region of interest as conditional input to a neural network, which can then generate inferences about the location and size of vessels according to the region of interest in period (ii).

[0073] In one example, an ultrasound machine adjusts for the difference between the collapse rate and recovery rate of a blood vessel when determining the detection result for period (ii) based on the detection result for period (i). For example, in ultrasound examination using a probe, during period (i) the probe gradually presses against the examination site, and during period (ii) the operator releases the probe. Therefore, the collapse rate generally depends on the pressure caused by the probe based on the operator's pressing action. On the other hand, the recovery rate depends mainly on the properties of the blood vessel itself (i.e., a nearly constant rate). As the inventors have observed, the recovery rate of the blood vessel itself is usually slower than the collapse rate. Therefore, the processor can modify the detection result for period (i) to match the actual recovery rate of the blood vessel (for example, by multiplying the detection result for period (i) by an appropriate coefficient to delay the recovery function). Figure 15 includes a graph showing the relationship between blood vessel diameter and time. Referring to Figure 15, the delay 1501 represents the gap between the actual recovery rate and the predicted rate that must be satisfied, assuming it is the same as the collapse rate. In some embodiments, the ultrasound system calculates the collapse rate by using techniques for detecting, determining the diameter of, and tracking blood vessels, then detects the timing of the transition from period (i) and period (ii), then calculates the (actual) recovery rate by using the first few period frames (ii), and then generates a coefficient between the actual rate and the predicted rate. Thus, by using the value of this coefficient, the detection result in period (i) is appropriately reflected in the detection process in period (ii). Further details are also illustrated using the flowchart in Figure 16 below. In one example, as the detection process progresses, the actual result of the same blood vessel volume (i.e., diameter) progresses, thereby making the gap more accurate, and so such a coefficient can be updated by essentially reflecting the detection result in period (ii) as time progresses. In one example, the blood vessel diameter can be used instead of the diameter in Figures 15 and 16. Furthermore, it should be noted that the techniques described with reference to Figures 15 and 16 can be based on the case where period (i) refers to the collapse rate and period (ii) refers to the recovery rate, but can also be applied when period (i) refers to the recovery rate and period (ii) refers to the collapse rate.

[0074] Figure 16 shows a flowchart of one embodiment of a process for using detection results from one period to create detection results for another period, as previously described with respect to Figures 13A to 15. This process can be carried out by processing logic that may include hardware (e.g., circuits, dedicated logic), software (such as that run on a general-purpose computer system or dedicated machine), firmware (e.g., software programmed in read-only memory), or a combination thereof. In some embodiments, the process is carried out by the processor of an ultrasonic machine.

[0075] Referring to Figure 16, the process begins with the processing logic detecting blood vessels from the ultrasound image (processing block 1601). As mentioned earlier, many different algorithms can be used to detect blood vessels, including but not limited to template matching and machine learning (e.g., sequence models including adaboost, deep learning, SVM, RNN, GRU, ConvGRU, LSTM, etc., for processing frame information within the sequence).

[0076] The processing logic also tracks the detected blood vessels between frames (images) and identifies the same blood vessels within those frames (processing block 1602). During pressure applied by the probe, the shape of the detected blood vessels changes, and the processing logic analyzes the changes in the shape, position, size, and combinations thereof of the same blood vessel frame by frame across several frames (processing block 1603). In response to the analysis, the processing logic determines the first period (i) in which the shape transitions from a normal ellipse (no pressure) to the most collapsed ellipse (maximum pressure) (processing block 1604). Since the frame rate is predetermined by the ultrasound machine (e.g., 30 frames / second), the processing logic calculates the duration of the first period and the rate of blood vessel collapse (processing block 1605). Thus, from processing blocks 1601-1605, the processing logic obtains information about the blood vessels during period (i).

[0077] The processor determines a frame that is the starting frame for vascular recovery (processing block 1606). For example, the processing logic can make the vascular recovery rate equivalent to the vascular collapse rate. Next, the processor can determine a frame that is the starting frame for vascular recovery, and from this starting frame, for example, enter period (ii). Then, the processor can estimate the position, diameter, or other information of the vascular in the current frame based on the detection results during period (i) and the time interval between the current time and the time of the starting frame. In other words, several frames of period (ii) are also acquired in processing block 1606 so that the time of change from period (i) to period (ii) is determined. Also in processing block 1606, since several frames of period (ii) have already been acquired, the processor can use these frames to calculate the recovery rate, thereby calculating a coefficient (corresponding to delay 1501 in Figure 15) that should be applied to the detection results during period (i), based on several frames of period (ii).

[0078] In processing block 1607, a new current image (belonging to period (ii)) is acquired. Since the time interval and coefficient between the current time and the time of change from period (i) to period (ii) are known, information about the vessels in the appropriate period (i) is acquired for use when creating the new image. More specifically, the processor can take into account the difference between the recovery rate and collapse rate of the vessels. For example, in some embodiments, the processor modifies the detection result of period (i) by multiplying the detection result of period (i) by an appropriate coefficient to delay the recovery function to match the actual recovery rate of the vessels (processing block 1607). Thus, the detection result at the appropriate point in time of period (i) is input to the processor (e.g., an AI algorithm) and the detection of vessels in the new current image is enhanced. In one example, the position and / or diameter of the vessels at the appropriate point in time of period (i) is input to the processor (e.g., an AI algorithm) so that a specified area for exploration can be limited and / or the size of the vessels for exploration can be limited, thereby reducing the processor's load.

[0079] In some embodiments, after identifying a blood vessel in an ultrasound image, the user may want to find the optimal location in the vessel from which a needle or other interventional device can be inserted. For example, blood vessels can extend from the elbow to the wrist, from the upper arm to the elbow, etc., and the operator may want to explore a specific location or position in the blood vessel for needle insertion. However, when exploring the best location in a blood vessel, the operator moves the probe from one location to another, thereby acquiring a series of ultrasound images that can be checked. In such cases, the operator may miss the best location for examination and / or intervention while the probe is moving.

[0080] In some embodiments, ultrasound images including blood vessels are acquired while the ultrasound probe is moving. For example, an ultrasound machine can generate ultrasound images based on ultrasound echo signals received by the ultrasound probe. The ultrasound machine determines at least one ultrasound image based on the position of the ultrasound probe relative to the blood vessel when the ultrasound image was received, and determines one or more features of the blood vessel in the ultrasound image. The ultrasound machine displays a marker for each feature of the blood vessel in a subsequent ultrasound image (for example, in the current ultrasound image).

[0081] In some embodiments, several past ultrasound images are acquired to indicate the best or more desirable position for insertion previously performed by the user, and image analysis, including the detection of blood vessels, is performed on each image. In some embodiments, the ultrasound machine selects one image from the past images based on the analysis (hereinafter, the selected image is also referred to as the “best image”). In some embodiments, the selected image is an ultrasound image obtained when the probe is positioned in the desired portion of the blood vessel for insertion. The determination of which position is best or desirable can be based on predetermined criteria. Such predetermined criteria may include the size of the diameter of the blood vessel, the location of a given blood vessel being shallower than a predetermined depth, etc. Examples of such predetermined criteria are described in more detail below.

[0082] Information about the blood vessels in the best image detected can be superimposed on the current (live) image. This superimposition can be done as a graphic superimposed on the image. Figure 17 shows an example of a current (live) image with such superimposition. Referring to Figure 17, vein 1701 is highlighted with an ellipse, along with the addition of an L-shaped depth marker. In one example, the L-shaped depth marker may be additionally displayed if the location of the detected blood vessel is shallower than a given depth. However, other blood vessels 1702 are shown superimposed on the current image as information about the blood vessels in the best image. Note that blood vessel 1703 is also shown on the image and represents an artery. Therefore, blood vessel 1703 is shown with a backslide, as opposed to a circle or ellipse without backslides, in order to highlight blood vessel 1703 as an artery rather than a vein.

[0083] To identify and overlay information associated with the best image, the ultrasonic machine may include a probe having a position sensor for providing positional information to be analyzed by a processor. In some embodiments, the position sensor includes a gyroscope or an accelerometer. In some embodiments, the image is stored in memory along with the probe's positional information.

[0084] The ultrasound machine also includes a processor that can detect blood vessels from ultrasound images and select the best image from among multiple images based on at least one of the following:

[0085] 1) The diameter of the blood vessel is large (e.g., larger than the threshold or the largest among multiple images);

[0086] 2) The detected blood vessels are located at a shallower depth than the specified depth;

[0087] 3) The detected blood vessel is not located too close (e.g., within 2 mm laterally or 3-4 mm deep) to an anatomical structure (e.g., nerve or artery) that could be damaged or obstructed when inserting a needle or catheter;

[0088] 4) High likelihood value for detecting blood vessels;

[0089] 5) The internal structure of the blood vessel is determined to be clear. In other words, the blood vessel wall or intima is clearly visible in the ultrasound image;

[0090] 6) No abnormal structures (e.g., thrombosis, edema, etc.) are observed in the blood vessels on the ultrasound image; and

[0091] 7) The diameter of the vessel is stable around the location of the vessel. In some embodiments, the ultrasound machine makes selections that take into account that the location of the vessel is stable around that location. In some embodiments, the word “stable” means that the diameter deviation is less than a threshold diameter (e.g., 10-20% of the maximum diameter of the vessel). The word “stable” may also mean that the displacement of the change in location is less than a threshold. Since the instrument (e.g., catheter) is inserted along the longitudinal direction of the vessel, such displacement is preferably small in order to prevent phlebitis due to friction between the instrument and the vessel. In other words, the vessel is not in a good position for insertion where it is narrowed, and as a result the diameter is locally large, but the location is changing significantly.

[0092] After determining the best image, information about the best image can be overlaid on the monitor currently displaying the live B-mode image. In some embodiments, the information includes one or more of the following:

[0093] 1) Information about the blood vessels in the best image detected (e.g., the location and / or size of the blood vessels). An example of displaying such information is a dashed circle indicating the diameter of the blood vessel in the best image from the past. Another example is displaying an arrow pointing to the location of the blood vessel in the best image from the past. Yet another example is using the same color to highlight the same blood vessel if the blood vessel in the current image detected is the same as the blood vessel in the best image (e.g., a dashed green circle can be used to indicate the blood vessel in the best image, and a solid green circle can be used to indicate the same blood vessel in the current image). As mentioned above, in one example, the color of the circle can be selected based on a standard color corresponding to at least the blood vessel diameter and / or catheter size. In addition to or instead of the same color, blood vessel identification information can be used;

[0094] 2) Time information (e.g., the time when the best image was obtained, relative time from the present);

[0095] 3) Guidance information (e.g., by using the probe's position). Since positional information regarding the location where the best image was scanned is stored in memory, the guidance information is determined by comparing the location where the best image was scanned with the current position (e.g., indicating direction on the monitor to facilitate movement to the probe's position, or displaying the relative distance). In some embodiments, the guidance information is shown on a schema (an illustration of the whole or part of the human body). For example, if an arm is displayed on the screen, probe movement direction indicators (e.g., arrows) can be superimposed on the arm diagram.

[0096] Figures 18A and 18B show an example of guide information along with information on the best position. Referring to Figures 18A and 18B, solid ellipses represent veins detected in the current image, and dashed ellipses represent the (same) veins detected in the best image in the past. Arrows 1801 and 1802 represent guide information.

[0097] In Figure 18A, arrow 1801 is inside the dashed ellipse, with a narrowed tip. This arrow configuration indicates that the probe should move to the rear side of the monitor. In Figure 18B, arrow 1802 is outside the dashed ellipse, with a widened tip. This arrow configuration indicates that the probe should move to the front side of the monitor.

[0098] In some embodiments, guide information is displayed along with information about the detected blood vessels in the best image. For example, an ultrasound machine may display guide information as an annotation along with an indication of the diameter of the detected blood vessels and / or their classification as veins or arteries. In another example, an ultrasound machine may display guide information along with a dashed circle of a color that can be selected based on standard colors corresponding to at least the blood vessel diameter and / or catheter size.

[0099] In addition to or instead of displaying the above information, the ultrasonic machine may provide the operator with audible and / or tactile information. For example, in some embodiments, as audible information, frequency and / or volume are changed according to the relative distance between the best image position and the current position. Thus, as the operator moves the probe closer to or further away from the best image position, the audible information changes in one or more ways to warn the user that it is getting closer (e.g., becoming louder) or in one or more other ways to warn the user that it is moving away (e.g., becoming quieter). In one example, as tactile information, the power and / or pattern of vibrations from the body of the probe (e.g., a smartphone or tablet) are changed to indicate that the probe is approaching the best image position.

[0100] If guide information is displayed on the monitor, tracking the probe's movement can determine whether the probe moved in accordance with the guide information. For example, if audible information is provided to the operator, the ultrasonic machine can track the probe's movement to determine whether the probe moved in accordance with the audible information. As another example, if the operator moves the probe in the wrong direction relative to the guide information and / or audible information, the ultrasonic system can provide a sign of the wrong direction by displaying a stop sign icon, a warning icon, a reverse direction icon, etc., and / or playing an audio signal that is not unpleasant to the operator, such as words like "stop," "wrong direction," or a tone accompanied by an unpleasant beep. In addition to or instead of this, if the operator moves the probe in the correct direction relative to the guide information and / or audible information, the ultrasonic system can provide a sign of the correct direction by displaying a forward icon, a traffic light icon with a green light on, etc., and / or playing an audio signal that is satisfactory to the operator, such as words like "proceed," "correct direction," applause, cheers, or a series of tones accompanied by a pleasant harmony.

[0101] Typical flowchart Figure 19A shows a flowchart of one embodiment of a process for displaying blood vessels on an ultrasound image. This process relates to the technology described in relation to Figures 7, 8, and 9. This process can be executed by processing logic that may include hardware (e.g., circuits, dedicated logic), software (e.g., run on a general-purpose computer system or dedicated machine), firmware (e.g., software programmed in read-only memory), or a combination thereof. In some embodiments, the process is executed by the processor of an ultrasound machine.

[0102] Referring to Figure 19A, the process begins when processing logic receives an ultrasound image containing blood vessels (processing block 1901). The processing logic determines the diameter of the blood vessels in the ultrasound image, the diameters being the respective diameters of each ultrasound image (processing block 1902). In some embodiments, the determination is performed using a neural network at least partially implemented in the hardware of a computing device such as an ultrasound machine or a computing device coupled to an ultrasound machine (e.g., a tablet).

[0103] The processing logic also determines the vessel diameter based on the diameter of vessels from past images (processing block 1903). That is, the vessel diameter of the current image can be calculated from multiple vessel diameters of a predetermined number of past images as described earlier, as mentioned in relation to Figure 7. Based on the vessel diameter, the processing logic selects a color (processing block 1904). In some embodiments, selecting a color includes matching the color to a standard color corresponding to the catheter size determined based on the vessel diameter.

[0104] Next, the processing logic indicates a blood vessel with a colored label in one of the ultrasound images (processing block 1905). In some embodiments, the process also includes processing logic that determines the position of the label on the touchscreen of the computing device (processing block 1906) and receives a user selection via the touchscreen adjacent to the position (processing block 1907). In response to receiving the user selection, the processor disables the display of the label if the label is currently displayed on the touchscreen, and enables the display of the label if the label is not currently displayed on the touchscreen (processing block 1908).

[0105] Figure 19B shows a flowchart of another embodiment of the process for displaying blood vessels on an ultrasound image. This process relates to the technique described in connection with Figure 7. This process can be performed by processing logic that may include hardware (e.g., circuits, dedicated logic, etc.), software (e.g., run on a general-purpose computer system or dedicated machine), firmware (e.g., software programmed in read-only memory), or a combination thereof. In some embodiments, the process is performed by the processor of an ultrasound machine.

[0106] Referring to Figure 19B, the process begins when processing logic receives an ultrasound image containing blood vessels (processing block 1911). The processing logic determines the diameter of the blood vessels in the ultrasound image, the diameter being the respective diameter of each ultrasound image. The processing logic also calculates the blood vessel diameter based on the determined diameter (processing block 1912). In some embodiments, the determination is performed using a neural network at least partially implemented in the hardware of a computing device (e.g., an ultrasound machine, or a computing device coupled to an ultrasound machine).

[0107] After determining the vessel diameter, in some embodiments, the processing logic receives a user selection of a function that represents one of the mean, median, mode, maximum, or minimum values. In addition to or instead of this, the processing logic may determine the amount of movement of the ultrasound probe used to generate the ultrasound image and, based on the amount of movement, select a function as the mean, median, mode, maximum, or minimum value (processing block 1913). In some embodiments, the selection process is based on comparing the amount of movement to a threshold movement value and selecting the function as the maximum value if the amount of movement is less than the threshold movement value, and selecting the function as the mean value if the amount of movement is greater than or equal to the threshold movement value.

[0108] Next, the processing logic determines the current value of the vessel diameter based on the vessel diameter in the previous image by applying a function to the vessel diameter in the previous image (processing block 1914). Based on the current value of the vessel diameter, the processing logic selects a color (processing block 1915). In some embodiments, selecting a color includes matching the color to a standard color corresponding to the catheter size determined based on the vessel diameter. The processing logic then indicates the vessel with a colored marker in one of the ultrasound images (processing block 1916). In addition to or instead of this, the selection of a function for calculating the current value for the vessel diameter can be determined as a preset of the ultrasound system configuration (for example, processing block 1916 may come before processing block 1911, and other sets of processing blocks may be repeated).

[0109] Figure 19C shows a flowchart of yet another embodiment of the process for displaying blood vessels on an ultrasound image. This process relates to the technique described in connection with Figure 7. This process can be performed by processing logic that may include hardware (e.g., circuits, dedicated logic, etc.), software (e.g., run on a general-purpose computer system or dedicated machine), firmware (e.g., software programmed in read-only memory), or a combination thereof. In some embodiments, the process is performed by the processor of an ultrasound machine.

[0110] Referring to Figure 19C, the process begins when processing logic receives an ultrasound image containing blood vessels (processing block 1921). The processing logic determines the diameter of the blood vessels in the ultrasound image, the diameters of which include the respective diameters of the blood vessels for each ultrasound image (processing block 1922). In some embodiments, the determination is performed using a neural network at least partially implemented in the hardware of a computing device (e.g., an ultrasound machine, or a computing device coupled to an ultrasound machine).

[0111] Next, the processing logic determines the vessel diameter based on the vessel diameter by applying a function to the vessel diameter (processing block 1923). Based on the vessel diameter, the processing logic selects a color (processing block 1924). In some embodiments, selecting a color includes matching the color to a standard color corresponding to the catheter size determined based on the vessel diameter.

[0112] Next, the processing logic displays the blood vessel in one of the ultrasound images with a colored label, and the display is based on whether the blood vessel is a vein or an artery (processing block 1925). In some embodiments, if the blood vessel is determined to be a vein, the processing logic provides the display by enabling the display of a colored label that is close to the blood vessel, or if the blood vessel is determined to be an artery, the processing logic prohibits the display of a colored label. In addition to or instead of this, if the blood vessel is determined to be an artery, the processing logic may provide the display of the blood vessel in one of the ultrasound images using further labeling that distinguishes the artery from the vein.

[0113] In addition to or instead of this, the processing logic can perform the function of distinguishing arteries from veins without calculating the vessel diameter, as described in processing block 1922. In such cases, the processing logic can use the results of performing the function to provide the user with an indication of whether the vessel shown in the ultrasound image is an artery or a vein. For example, if the detected vessel is a vein, a color to highlight it is selected according to the vessel diameter or the current value of the vessel diameter, and if the detected vessel is an artery, the color to highlight it is fixed to a single color regardless of the diameter.

[0114] Figure 20 shows a flowchart of yet another embodiment of the process for displaying blood vessels on an ultrasound image. This process relates to the techniques described in relation to Figures 8, 9, and 12. This process can be carried out by processing logic that may include hardware (e.g., circuits, dedicated logic, etc.), software (e.g., run on a general-purpose computer system or dedicated machine), firmware (e.g., software programmed in read-only memory), or a combination thereof. In some embodiments, the process is carried out by the processor of an ultrasound machine.

[0115] Referring to Figure 20, the process begins with the processing logic receiving an ultrasound image containing blood vessels (processing block 2001). The processing logic determines the diameter and location of the blood vessels within a subset of the ultrasound image (processing block 2002). In some embodiments, the determination is performed using a neural network at least partially implemented in the hardware of a computing device (e.g., an ultrasound machine, or a computing device coupled to an ultrasound machine). In one example, the processing logic performs tracking techniques and calculates likelihood values, as described above in Figure 12.

[0116] After the diameter and location of the blood vessels are determined, the processing logic determines a detection failure for one of the blood vessels in one of the ultrasound images that is not included in the subset (processing block 2003). A detection failure indicates the absence of a diameter or location for one of the blood vessels in one of the ultrasound images, which is determined, for example, by a neural network.

[0117] In response to a detection failure, the processing logic displays a label (e.g., ellipse, dot, boundary container, etc.) on one of the ultrasound images indicating the diameter and location of one of the blood vessels in that ultrasound image (processing block 2004). The label is based on the diameter and location determined, for example, by a neural network, for a subset of the ultrasound images. The label may include one or more labels such as a bounding box and text.

[0118] In some embodiments, displaying a marker includes displaying a marker for one of the vessels in a first format (e.g., a solid ellipse) to indicate detection failure, and displaying further markers for further vessels in a second format (e.g., a dotted ellipse). In one example, the first format of the marker includes a dashed container, and the second format of the further marker includes a solid container. In addition to or instead of this, the first format of the marker may include a first color, and the second format of the further marker may include a second color. In some embodiments, the second color corresponds to a standard color representing a catheter size, while the first color does not indicate a standard catheter size.

[0119] In one example, the process includes receiving a user selection via the touchscreen of a computing device located near the marker (processing block 2005) and, in response, disabling the display of the marker (processing block 2006). The processing logic then receives a further user selection via the touchscreen of a computing device located near the position indicated by the marker (processing block 2007) and, in response, re-enables the display of the marker (processing block 2008). Furthermore, in some embodiments, some processing blocks in Figure 7 can be appropriately incorporated into Figure 21 so that not only tracking techniques but also calculation of vessel diameter and associated highlighting are performed. For example, in block 2004, the marker may be based on calculation of vessel diameter using a subset of images.

[0120] Figure 21 shows a flowchart of a further embodiment of a process for displaying blood vessels on an ultrasound image. This process relates to the technology described in connection with Figure 18. This process can be carried out by processing logic that may include hardware (e.g., circuits, dedicated logic, etc.), software (e.g., run on a general-purpose computer system or dedicated machine), firmware (e.g., software programmed in read-only memory), or a combination thereof. In some embodiments, the process is carried out by the processor of an ultrasound machine.

[0121] Referring to Figure 21, the process begins with the processing logic receiving an ultrasound image containing blood vessels while the ultrasound probe is moving (processing block 2101). The ultrasound image is based on the ultrasound echo signal received by the ultrasound probe while it is moving.

[0122] After receiving an ultrasound image, the processing logic determines one of the ultrasound images based on the position of the ultrasound probe relative to the blood vessel when the ultrasound image was received (processing block 2102), and determines the features of the blood vessel in the ultrasound image (processing block 2103). In some embodiments, the features include the diameter of the blood vessel in the ultrasound image, and the labeling includes a boundary container representing the diameter. In addition to or instead of this, the features may include the position of the blood vessel in the ultrasound image.

[0123] Subsequently, the processing logic displays markers for vascular features within a further ultrasound image of the ultrasound image (processing block 2104). The processing logic may display further markers to guide the movement of the ultrasound probe to a position within the further ultrasound image of the ultrasound image (processing block 2105). In some embodiments, the further ultrasound image is the most recent ultrasound image of the ultrasound image.

[0124] The procedures described herein constitute an improvement over procedures that do not display blood vessels in enhanced ultrasound images. Rather, the procedures described herein provide enhancements that contain useful, and in some cases important, information, and display this information to the operator of the ultrasound system so that the operator can apply the information to the ultrasound procedure in real time. Thus, the procedures described herein result in a significant improvement in the patient experience, including reducing or minimizing the number of times the interventional device is inserted into the patient, reducing the patient's pain and infection risk, and preventing undesirable insertion of the device, such as insertion into an artery instead of a vein. Thus, the patient does not perceive unnecessary pain and discomfort, and the risk of one or more of the following is reduced compared to conventional ultrasound procedures that do not display enhanced blood vessels: infection, thrombosis due to insertion of an oversized catheter, extravasation, and damage to the blood vessel wall. Therefore, the procedures described herein may be suitable for medical procedures and examinations in which conventional ultrasound procedures are unsuitable.

[0125] Examples of ultrasonic machines Figure 22 is a block diagram of one embodiment of an ultrasonic machine. Note that any type of ultrasonic machine, including fixed, portable, handheld, and combinations thereof, can be used to perform the vascular identification and highlighting techniques described herein. The block diagram is an example of an ultrasonic machine that can be constructed using the blocks in Figure 22. For example, signals can be redefined, and modified systems can be formed by modifying, combining, splitting, adding, or deleting blocks without changing the functionality of the block diagram. Thus, such modified systems are considered to be within the scope of this disclosure. Furthermore, blocks, modules, and units of ultrasonic machines, such as the ultrasonic machine shown in Figure 22, can be implemented as any type of module or component in software (e.g., as software instructions executable in a processing system), hardware, or a combination thereof, as a standalone application, or as a module or component in another device application, and in any type of computing device.

[0126] Referring to Figure 22, the ultrasonic machine includes an ultrasonic transducer probe 2201 coupled to an ultrasonic device 2202. The ultrasonic transducer probe 2201 includes an ultrasonic transducer 2201A electrically coupled to a receiver 2201B and a transmitter 2201C. The ultrasonic transducer 2201A has one or more transducer elements and, during operation, transmits ultrasonic energy from one or more transducer elements toward a subject in response to a transmission signal from the transmitter 2201C, and receives ultrasonic echoes from the subject using the receiver 2201B. The receiver 2210B and transmitter 2210C may include any type of circuit or any other type of form, such as a processor, as will be described later. The receiver 2210B and transmitter 2210C may be formed separately or integrated into a single form. The receiver 2210B and transmitter 2210C may be incorporated into a processor with other functions, such as the processor 2202C described later. The ultrasound echo is converted into an electrical signal by the receiver 2201B and electrically coupled to electronic components within the ultrasound device 2202 (e.g., an analog-to-digital (A / D) converter 2202A, one or more processors 2202C, a memory module 2202B, a beamformer, an FPGA, etc.) configured to process the electrical signal and form one or more ultrasound images.

[0127] In one example, the ultrasonic probe 2201 may include an A / D converter (not shown in Figure 22) that can be configured to digitize the electrical signal generated by the receiver 2201B based on the ultrasonic echo. Furthermore, the ultrasonic probe 2201 may also include a partial or complete beamformer (not shown in Figure 22) configured to sum the electrical signals in a phase-compensated manner. Thus, the ultrasonic probe 2201 may be implemented to couple the digital signal to the ultrasonic device 2202 via a wired or wireless communication link, for example. For instance, the ultrasonic probe 2201 and the ultrasonic device 2202 may each include a transceiver (not shown in Figure 22) that communicates using a digitally modulated communication signal based on the digital signal.

[0128] The processor 2202C may include, in part or in whole, any type or any number of application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), integrated circuits (ICs), logic, etc. In some embodiments, the processor 2202C includes a beamformer and includes processing for forming an ultrasonic image. It should also be noted that the processing described herein may be performed at least in part on custom-specific connected hardware such as a cloud, mounted or networked processing devices, discrete or external GPUs, or USB-connected devices.

[0129] Acquiring ultrasound data from a subject using the ultrasound probe 2201 generally involves generating ultrasound, transmitting ultrasound into the subject, and receiving ultrasound reflected by the subject. A wide range of ultrasound frequencies can be used to acquire ultrasound data, for example, low-frequency ultrasound (e.g., less than 15 MHz) and / or high-frequency ultrasound (e.g., 15 MHz or higher). Those skilled in the art can easily determine which frequency range to use based on factors such as imaging depth and / or desired resolution, for example, but not limited to these.

[0130] In some embodiments, the ultrasound device 2202 includes one or more processors 2202C that cause an ultrasound probe 2201 to transmit an electric current to emit sound waves and receive electrical pulses generated from the echoes returning from the probe 2201. An A / D converter 2202A receives an analog signal from the receiver 2201B and converts it into digital data stored in memory 2202B. Memory 2202B also stores software and other data related to the detection of blood vessels (e.g., veins, arteries, and capillaries) (e.g., templates, user preferences for enhancement, etc.), as well as analysis results along with ultrasound images and other related data.

[0131] The processor 2202C processes the raw data associated with the received electrical pulses and forms an image that is transmitted to the display device 2203, which displays the image on the monitor 2203A. Thus, the monitor 2203A can display an ultrasound image from the ultrasound data processed by the processor 2202C of the ultrasound device 2202. The monitor 2203A can also display vascular detection results, as well as probe guidance information, as described herein. In one example, the monitor 2203A includes a clinical display device. In addition to or instead of this, the monitor 2203A may include a display of a computing device coupled to the ultrasound machine, such as a tablet.

[0132] In some embodiments, the processor 2202C performs vascular detection using template matching, artificial intelligence or machine learning (e.g., adaptive boosting (adaboost), deep learning, support vector machine (SVM), sequence models including RNN, GRU, ConvGRU, LSTM, etc. for processing frame information in the sequence), and / or other detection methods. In some embodiments, the processor 2202C executes AI algorithms and / or uses neural networks to identify veins and arteries and position them in the ultrasound image.

[0133] In some embodiments, after the processor 2202C detects a blood vessel, the processor 2202C displays the blood vessel on the ultrasound system's monitor or display. In one example, the detected blood vessel is displayed in an highlighted manner by the processor 2202C to provide information to the operator or user of the ultrasound machine. In some embodiments, the processor 2202C draws an outline or other form of blood vessel representation (e.g., an identifier) ​​around or near the blood vessel.

[0134] By highlighting blood vessels, additional information can be made available to the operator. For example, in some embodiments, the outline of a vein is modified to match the color coding of the largest catheter that can fit into that vein. The color coding may be the industry standard color coding for catheters. In this way, the operator can quickly identify the catheter that can be used with a particular blood vessel. In this way, the operator selects the catheter size based on the displayed blood vessel. Alternatively, by color-coding the blood vessels according to the largest catheter, the operator can select the catheter size and have the ultrasound system identify all blood vessels (e.g., veins) in the image that are suitable for that catheter size.

[0135] In addition to or instead of this, the processor 2202C can identify all veins in an image suitable for a particular catheter size. In some embodiments, the operator can touch a vein in the image and have the processor 2202C display the diameter and depth of that vessel. In some embodiments, the processor 2202C automatically identifies the most central and shallowest vein and automatically provides its diameter and depth on the ultrasound machine's display.

[0136] In some embodiments, the processor 2202C calculates a likelihood value for each detected vessel as an indicator of the confidence level associated with the detection result. In one example, the processor 2202C generates a more or less opaque vein / artery contour to indicate the reliability of the prediction for that vessel.

[0137] In some embodiments, the processor 2202C tracks detected vessels across frames and identifies the same detected vessels in multiple images. The processor 2202C may use vessel location information or vessel type for tracking and / or identification.

[0138] In some embodiments, the ultrasound system has one or more user input devices 2204 (e.g., keyboard, cursor control device, etc.) that receive data from at least one operating unit 2204A. Along with memory 2202B, the ultrasound system also has a storage device 2205 including a storage device 2205A (e.g., hard disk, floppy disk®, compact disk (CD), digital video disk (DVD)) for storing acquired images (e.g., past images, current images, etc.), and the acquired images can be used to calculate the diameter of blood vessels, select the best image for insertion, and for other operations described herein. The ultrasound system further includes a data communication device 2206 having a data communication interface 2206A for providing communication between the ultrasound device 2202 and an external server 2207.

[0139] In some embodiments, the ultrasound machine includes a handheld ultrasound device having probes and a tablet or smartphone connected to each other. The connection may be wireless or wired. Figure 23 shows an example of such an ultrasound machine. Referring to Figure 23, the wireless ultrasound probe 2300 communicates wirelessly with the smartphone 2302. In some embodiments, the probe 2300 includes hardware and software that functions to perform transmission / reception by an array of transducers, A / D conversion, beamforming, quadrature detection, and wireless transmission. In some embodiments, the smartphone 2302 includes hardware and software that functions to generate an ultrasound image (e.g., a B-mode image) and display the image on its display. The hardware and software of the smartphone 2302 can detect veins, calculate the vessel diameter, calculate a value (e.g., a maximum value) using multiple vessel diameters, and generate color highlighting of the detected veins to apply to the monitor.

[0140] Figure 24 shows a data flow diagram of the vascular identification display subsystem. In some embodiments, the subsystem is part of the ultrasound machine as described in Figure 22. In addition to or instead of this, the subsystem can operate in conjunction with the ultrasound machine by providing information to the ultrasound machine for use and / or display, and by receiving information from the ultrasound machine.

[0141] Referring to Figure 24, memory 2401 stores the ultrasound image 2402. A neural network (or other vascular detection mechanism) 2403 receives or acquires the ultrasound image 2402. In some embodiments, the neural network 2403 uses AI or other machine learning algorithms to determine the location and diameter of the blood vessels in the ultrasound image 2402 and outputs this information as blood vessel location / diameter information 2404 to a display generator 2406.

[0142] In some embodiments, the neural network 2403 also uses AI or other machine learning algorithms to determine when a blood vessel detection failure has occurred and outputs this information to the display generator 2406 as blood vessel detection failure information 2405. In some embodiments, the neural network 2403 includes one or more processors.

[0143] In response to the vessel position / diameter information 2404 and / or vessel detection failure information 2405, the display generator 2406 generates vessel graphics or other displays to be superimposed on one or more ultrasound images.

[0144] For further details, please refer to U.S. Patent Application No. [number] titled "Identifying Blood Vessels in Ultrasound Images" and U.S. Patent Application No. [number] titled "Guidance for Instrument Insertion," which was filed concurrently with this specification.

[0145] The system described herein constitutes an improvement over systems that do not display blood vessels in enhanced ultrasound images. Rather, the system described herein generates enhancements that contain useful, and in some cases important, information and displays this information to the operator of the ultrasound system so that the operator can apply the information to the actual ultrasound procedure in real time. Thus, the system described herein can significantly improve the patient experience, including reducing or minimizing the number of times the interventional device is inserted into the patient, thereby reducing the patient's pain and the risk of infection, as well as preventing undesirable insertion of the device, such as insertion into an artery instead of a vein. Thus, the patient does not perceive unnecessary pain and discomfort, and the risk of one or more of the following is reduced compared to conventional ultrasound systems that do not display enhanced blood vessels: infection, thrombosis due to insertion of an oversized catheter, extravasation, and damage to the blood vessel wall. Thus, the system described herein may be suitable for medical procedures and examinations for which conventional ultrasound systems are unsuitable.

[0146] This specification describes several examples of embodiments.

[0147] Example 1 is a method performed by a computing device, the method comprising: receiving an ultrasound image including blood vessels; determining the diameter of blood vessels in the ultrasound image using a neural network at least partially implemented in the hardware of the computing device, wherein the diameter includes the respective diameter in each ultrasound image of the ultrasound image; determining the diameter of blood vessels based on the diameter of blood vessels; selecting a color based on the diameter of blood vessels; and displaying the blood vessels with colored markers in one of the ultrasound images.

[0148] Example 2 is the method of Example 1, which may include the step of receiving a user selection of a function that optionally represents one of the mean, median, mode, maximum, or minimum, and the step of determining the vessel diameter includes the step of applying the function to the vessel diameter.

[0149] Example 3 is the method of Example 1, which optionally includes the steps of determining the amount of movement of the ultrasound probe used to generate the ultrasound image, and selecting a function as the mean, median, mode, maximum, or minimum based on the amount of movement, and the step of determining the vessel diameter includes applying the function to the vessel diameter.

[0150] Example 4 is the method of Example 3, which optionally includes a step of comparing the amount of movement with a threshold moving value, and the step of selecting a function is based on the comparison and includes a step of selecting the function as the maximum value if the amount of movement is less than the threshold moving value, or a step of selecting the function as the average value if the amount of movement is greater than or equal to the threshold moving value.

[0151] Example 5 is the method of Example 1, which optionally includes a step of determining the catheter size based on the blood vessel diameter, and a step of selecting a color, which includes matching the color to a standard color corresponding to the catheter size.

[0152] Example 6 is the method of Example 1, which optionally includes a step of determining whether a blood vessel is a vein or an artery, and the display step includes a step of enabling the display of a marker having a color similar to that of a blood vessel when the blood vessel is determined to be a vein, or prohibiting the display of a colored marker when the blood vessel is determined to be an artery.

[0153] Example 7 is the method of Example 1, which optionally includes the steps of determining whether a blood vessel is a vein or an artery, and, if the blood vessel is determined to be an artery, indicating the blood vessel in one of the ultrasound images with further markings that distinguish arteries from veins.

[0154] Example 8 is a method of Example 1 that optionally includes the steps of determining the position of a sign on the touchscreen of a computing device, receiving a user selection via a touchscreen adjacent to the position, and, in response to receiving the user selection, disabling the display of the sign if the sign is currently displayed on the touchscreen, or enabling the display of the sign if the sign is not currently displayed on the touchscreen.

[0155] Example 9 is a method implemented by a computing device, the method comprising: receiving an ultrasound image including blood vessels; determining the diameter and location of blood vessels in a subset of ultrasound images using a neural network at least partially implemented in the hardware of the computing device; determining a detection failure for one of the blood vessels in one of the ultrasound images that is not included in the subset, wherein the detection failure indicates the absence of a diameter or location determined by the neural network for one of the blood vessels in one of the ultrasound images; and, in response to the detection failure determination, displaying a label for the diameter and location of one of the blood vessels in one of the ultrasound images, wherein the label is based on the diameter and location determined by the neural network for the subset of ultrasound images.

[0156] Example 10 is a method of Example 9 that optionally includes the steps of receiving a user selection via the touchscreen of a computing device adjacent to the sign, and, in response to receiving the user selection, disabling the display of the sign.

[0157] Example 11 is a method of Example 10 that optionally includes the steps of receiving a further user selection via the touchscreen of a computing device located near the position indicated by the sign, and enabling the display of the sign in response to the receipt of the further user selection.

[0158] Example 12 is a method of Example 9, which may optionally include a display step that includes displaying a label in one of the blood vessels in a first format to indicate detection failure, and displaying further labels in further blood vessels of the blood vessel in a second format.

[0159] Example 13 is the method of Example 12, which may optionally include a first format of the sign containing a dashed line container and a second format of the further indication containing a solid line container.

[0160] Example 14 is a method of Example 12 which may optionally include a first format of the sign containing a first color and a second format of the further display containing a second color.

[0161] Example 15 is a method of Example 14 that may optionally include a second color corresponding to a standard color representing a catheter size, while the first color does not indicate a standard catheter size.

[0162] Example 16 is a method performed by a computing device, the method comprising: receiving an ultrasound image including a blood vessel while an ultrasound probe is moving, the ultrasound image being based on ultrasound echo signals received while the ultrasound probe is moving; determining one of the ultrasound images based on the position of the ultrasound probe relative to the blood vessel when the ultrasound image is received; determining the features of the blood vessel in the ultrasound image; and displaying a marker for the features of the blood vessel in one of the further ultrasound images.

[0163] Example 17 is the method of Example 16, which may optionally include the feature including the diameter of a blood vessel in an ultrasound image and the label including a boundary container representing the diameter.

[0164] Example 18 is a method of Example 16, which optionally includes the feature of including the location of blood vessels in an ultrasound image.

[0165] Example 19 is the method of Example 16, which may optionally include the option that the additional ultrasound image is the most recent ultrasound image.

[0166] Example 20 is a method of Example 16 that optionally includes the step of displaying additional markers in the additional ultrasound image to guide the movement of the ultrasound probe to a position.

[0167] Some parts of the detailed explanation above are presented with respect to algorithms and symbolic representations of operations on data bits in computer memory. The description and representation of these algorithms are means used by those skilled in the field of data processing to most effectively convey the content of their research to others skilled in the field. Here, an algorithm is generally considered to be a self-consistent set of steps that produce a desired result. These steps require the physical manipulation of physical quantities. These quantities, though not always, take the form of electrical or magnetic signals that can be stored, transferred, combined, compared, and otherwise manipulated. Referring to these signals as bits, values, elements, symbols, characters, terms, numbers, etc., has sometimes proven convenient, primarily for reasons of general use.

[0168] However, it should be noted that all these and similar terms should be associated with appropriate physical quantities and are merely convenient labels applied to those quantities. As will be evident from the following explanation, unless otherwise specifically stated, any explanation using terms such as “processing,” “computing,” “calculating,” “determining,” or “displaying” throughout the explanation is understood to refer to the operation and processes of a computer system or similar electronic computing device that manipulates data represented as physical (electronic) quantities in the registers and memory of a computer system and converts it into other data similarly represented as physical quantities in the memory or registers of a computer system or other such information storage device, transmission device, or display device.

[0169] The present invention also relates to an apparatus for performing the operations described herein. This apparatus may be specifically configured for a particular purpose, or may include a general-purpose computer that is selectively started or reconfigured by a computer program stored in the computer. Such computer programs may be stored on computer-readable storage media, including, but are not limited to, any type of disk suitable for storing electronic instructions and each coupled to a computer system bus, such as floppy disks, optical disks, CD-ROMs, and magneto-optical disks, read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic or optical cards, or any type of disk containing any type of medium.

[0170] The algorithms and displays presented herein are not inherently related to any particular computer or other device. Various general-purpose systems may be used with the program in accordance with the teachings herein, or it may be advantageous to construct more specialized devices to perform the necessary method steps. The structures required for various such systems will become apparent from the following description. Furthermore, the present invention is not described with reference to any particular programming language. It can be understood that various programming languages ​​may be used to carry out the teachings of the present invention as described herein.

[0171] Machine-readable media include any mechanism for storing or transmitting information in a format readable by a machine (e.g., a computer). For example, machine-readable media include read-only memory ("ROM"), random-access memory ("RAM"), magnetic disk storage media, optical storage media, flash memory devices, and propagating signals in electrical, optical, acoustic or other forms (e.g., carrier waves, infrared signals, digital signals, etc.).

[0172] Many changes and modifications to the present invention will become apparent to those skilled in the art after reading the foregoing description, but it should be understood that any particular embodiment shown and described as an example is not intended to be considered limiting in any way. Accordingly, references to the details of various embodiments are not intended to limit the claims to enumerate only features that are considered essential to the present invention.

Claims

1. It is an ultrasonic system, Memory and Includes at least one processor coupled to the memory, The at least one processor is The steps include receiving an ultrasound image that includes blood vessels, A step of determining the diameter of the blood vessel in the ultrasound image using a neural network at least partially implemented in the hardware of the ultrasound system, wherein the diameter includes a first diameter of the blood vessel in a first ultrasound image and a second diameter of the blood vessel in a second ultrasound image, the first ultrasound image corresponds to a first probe pressure on the blood vessel, and the second ultrasound image corresponds to a second probe pressure on the blood vessel that is different from the first probe pressure; A step of determining the amount of movement of the ultrasound probe between generating the first ultrasound image and the second ultrasound image, The steps include comparing the amount of movement with a threshold movement value, A step of determining the current diameter of the blood vessel based on the first diameter and the second diameter, comprising applying one of the first function and the second function to the first diameter and the second diameter based on the comparison, The steps include selecting a color corresponding to a change in the shape of the blood vessel based on the current diameter of the blood vessel, The steps include: displaying the blood vessel with a colored label in one of the ultrasound images; An ultrasonic system configured to perform an action.

2. The first function is one of the mean, median, mode, maximum, or minimum values ​​of the first and second diameters of the blood vessel. The ultrasonic system according to claim 1.

3. The second function is one of the mean, median, mode, maximum, or minimum values ​​of the first and second diameters of the blood vessel. The ultrasonic system according to claim 1.

4. The at least one processor is The steps include selecting the first function as the maximum value of the first and second diameters of the blood vessel when the amount of displacement is less than the threshold displacement value, If the amount of displacement is greater than or equal to the threshold displacement value, the second function is selected as the average value of the first diameter and the second diameter of the blood vessel. The ultrasonic system according to claim 1, further configured to perform the following:

5. The at least one processor is configured to further perform the step of determining the catheter size based on the current diameter of the blood vessel, The step of selecting a color includes matching the color to a standard color corresponding to the catheter size. The ultrasonic system according to claim 1.

6. The at least one processor is configured to further perform the step of determining whether the blood vessel is a vein or an artery, The step of displaying is, When the aforementioned blood vessel is determined to be a vein, the marker having a color similar to that of the blood vessel may be displayed. To make something possible, or When the blood vessel is determined to be an artery, the display of the marker having the aforementioned color is prohibited. The ultrasonic system according to claim 1, including the following:

7. The at least one processor is A step of determining whether the blood vessel is a vein or an artery, When the blood vessel is determined to be an artery, the step of indicating the blood vessel in one of the ultrasound images with further markings that distinguish between arteries and veins, The ultrasonic system according to claim 1, further configured to perform the following:

8. The at least one processor is The steps include determining the position of the blood vessel on the touchscreen of the ultrasound system, The steps include receiving a user selection via the touchscreen located near the aforementioned position, In response to the reception of the user selection, If the sign is currently displayed on the touchscreen, disable the display of the sign, or The steps include enabling the display of the sign when the sign is not currently displayed on the touchscreen, The ultrasonic system according to claim 1, further configured to perform the following:

9. It is an ultrasonic system, Memory and Includes at least one processor coupled to the memory, The at least one processor is The steps include receiving an ultrasound image containing multiple blood vessels, The steps include determining the diameter and location of the plurality of blood vessels in a subset of the ultrasound images using a neural network at least partially implemented in the hardware of the ultrasound system, A step of determining a detection failure for a first blood vessel, which is one of the plurality of blood vessels, in a first ultrasound image, which is one of the ultrasound images not included in the subset, wherein the detection failure indicates the absence of a diameter or position determined by the neural network with respect to the first blood vessel in the first ultrasound image. In response to the determination of the detection failure, a step of displaying a label for the diameter and position of the first blood vessel in the first ultrasound image, wherein the label is based on the diameter and position determined by the neural network with respect to the subset of the ultrasound image, An ultrasonic system configured to perform an action.

10. The at least one processor is The steps include receiving a user selection via the touchscreen of the ultrasonic system located adjacent to the sign, In response to the reception of the user selection, the step of disabling the display of the indicator, The ultrasonic system according to claim 9, further configured to perform the following:

11. The at least one processor is The steps include receiving further user selections via the touchscreen located near the position indicated by the sign, In response to the reception of the further user selection, the step of enabling the display of the indicator, The ultrasonic system according to claim 10, configured to further perform the following.

12. The ultrasound system according to claim 9, wherein the display step includes displaying the label in the first vessel in a first format to indicate the detection failure, and displaying further labels in further vessels among the plurality of vessels in a second format.

13. The ultrasonic system according to claim 12, wherein the first format of the sign includes a dashed container, and the second format of the further sign includes a solid container.

14. The ultrasonic system according to claim 12, wherein the first format of the sign includes a first color, and the second format of the further sign includes a second color.

15. The ultrasound system according to claim 14, wherein the second color corresponds to a standard color representing a catheter size, and the first color does not indicate a standard catheter size.

16. It is an ultrasonic system, Memory and Includes at least one processor coupled to the memory, The at least one processor is A step of receiving an ultrasound image including blood vessels while an ultrasound probe is in motion, wherein the ultrasound image is based on ultrasound echo signals received by the ultrasound probe while it is in motion. A step of determining a first ultrasound image from among the ultrasound images based on the position of the ultrasound probe relative to the blood vessel when the ultrasound image is received, wherein the first ultrasound image includes marking of a first probe pressure of the blood vessel having a first diameter, The steps include determining the characteristics of the blood vessel within a further ultrasound image from the aforementioned ultrasound images, including the second diameter, A step of determining the amount of movement of the ultrasound probe between generating the first ultrasound image and the further ultrasound image, The steps include comparing the amount of movement with a threshold movement value, A step of determining the current diameter of the blood vessel based on the first diameter and the second diameter, and the comparison; The step of displaying the label of the features of the blood vessel in the further ultrasound image among the ultrasound images, It is configured to perform the following actions: The label in the further ultrasound image indicates a second probe pressure for the blood vessel that is different from the first probe pressure. Based on the change from the first probe pressure to the second probe pressure, the indicator changes. The label in the further ultrasound image includes a color indicating the current diameter of the blood vessel, The step of determining the current diameter of the blood vessel includes applying one of the first and second functions to the first and second diameters based on the comparison, The aforementioned colors correspond to the changes in the shape of the blood vessels. Ultrasonic system.

17. The ultrasonic system according to claim 16, wherein the marker includes a boundary container.

18. The ultrasound system according to claim 16, wherein the aforementioned features include the location of the blood vessel in the further ultrasound image.

19. The ultrasound system according to claim 16, wherein the further ultrasound image is the latest ultrasound image of the ultrasound image.

20. The ultrasonic system according to claim 16, wherein the at least one processor is configured to perform the step of displaying further markers in the further ultrasonic image that guide the movement of the ultrasonic probe to the position.

Images