Ultrasonic guidance method and system based on deep learning

Abstract

The invention discloses an ultrasonic guidance method and system based on deep learning, and the method comprises the steps: collecting ultrasonic dynamic image data through a collection terminal, carrying out the framing processing of the collected ultrasonic dynamic image data, and carrying out the real-time detection of the framed ultrasonic dynamic image data; positioning the section structure in a GPS navigation manner according to a detection result, and performing real-time coordinate identification on the detected section structure; recording and establishing a section structure feature frame index of each frame of ultrasonic dynamic image data; when the system is started, all the standard sections are displayed by default, the section state is displayed as undetected, each frame of detection is finished, the detected standard sections and the section with the characteristic state are displayed as detected, a user is guided to scan the undetected standard sections with a probe, and the ultrasonic standard section guiding effect is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of ultrasonic dynamic image data processing, and in particular to an ultrasonic guidance method and system based on deep learning. Background Art

[0002] Ultrasonic dynamic image data detection equipment is a medical device that uses ultrasonic technology to generate images of human tissues. It can observe the internal organs and tissue structures of the human body, and realize functions such as disease diagnosis, organ function monitoring, and surgical guidance. Ultrasonic equipment generally scans the long axis, short axis, coronal plane and other areas of the body along the direction of the probe setting, and the operator determines whether there are structural feature abnormalities in the scanned section.

[0003] Ultrasonic dynamic image data detection equipment is in the process of continuous image shooting when working, which will produce a considerable number of scanned images. These images need to rely on the operator's personal experience to judge the abnormalities of the characteristic structures in the images and select the appropriate section to scan again. This inspection method has high requirements for the inspection personnel and is prone to problems such as false detection and missed detection, which affects the efficiency of ultrasonic inspection.

[0004] The existing active image recognition system can only perform simple image segmentation on video images, but ultrasound videos often have a lot of noise. It is easy to detect the noise and noise in the image as ultrasonic features during image segmentation. In addition, the existing system can only output the planar shape of the ultrasonic feature and it is difficult to output the three-dimensional structure of the scanned object. Summary of the invention

[0005] The purpose of the present invention is to provide an ultrasound guidance method and system based on deep learning to solve the problems raised in the prior art.

[0006] To achieve the above object, the present invention provides the following technical solution: an ultrasound guidance method and system based on deep learning, wherein the guidance method comprises the following steps:

[0007] Step 101. Using an acquisition terminal to acquire ultrasound dynamic image data, performing frame processing on the acquired ultrasound dynamic image data, and performing real-time detection on the ultrasound image data pictures after frame processing;

[0008] Step 102: After the detection of each frame of ultrasonic image data is completed, the characteristic structure is located by GPS navigation according to the detection result, and the detected features are marked with real-time coordinates;

[0009] Step 103. Record and establish a frame index of each frame of ultrasound image data slice and cross-sectional structure feature, where each frame index corresponds to a slice of ultrasound dynamic image data;

[0010] Step 104. When the system is started, all ultrasound standard sections are displayed by default, and the section status is displayed as unchecked. After each frame of detection, the detected standard sections and section structure feature status are displayed as checked, guiding the user to use the probe to scan the unchecked standard sections, which plays a role in guiding ultrasound standard sections;

[0011] Step 105. Detect and analyze the undetected sections based on deep learning technology;

[0012] Step 106. After the ultrasound imaging case detection is completed, the ultrasound imaging detection results are analyzed, the system summarizes the detected standard sections and the cross-sectional structural features they contain, and automatically generates the best standard section based on the standard section classification and confidence.

[0013] According to the above technical solution, the ultrasound guidance method further includes the following contents:

[0014] During ultrasound-guided testing, by clicking the playback function key, the system loads the previously recorded video and frame index document section and cross-sectional structural feature information, plays the case video with marked section and cross-sectional structural features, and realizes the real-time case playback function;

[0015] At the end of the ultrasound-guided test, the user can choose to replay a historical case. The system will load the recorded video and frame index document sections and cross-sectional structural features of the case, and play the historical case video with annotated sections and cross-sectional structural features.

[0016] According to the above technical solution, in step 101, the ultrasound doctor installs a sensor on the ultrasound probe to obtain the positioning coordinates of the probe, and obtains the movement speed and movement direction of the probe by relying on the dynamic sensing function of the sensor;

[0017] The adjustment coefficient t is determined according to the angle between the sound beam scanning plane and the horizontal plane and the motion vector of the probe, where t=| |·cos(θ+α), where is the motion vector of the probe, θ is the angle between the motion vector of the probe and the scanning plane of the acoustic beam, and α is the angle between the scanning plane of the acoustic beam and the horizontal plane;

[0018] Before the equipment is working, the shooting frame rate r1 when the probe is stationary is set, and the real-time frame rate r2 during the probe movement is determined in combination with the adjustment coefficient of the probe movement, so that r2 satisfies r2=r1+r1·t, and the ultrasound dynamic image data is shot at the real-time frame rate;

[0019] Frame the ultrasonic dynamic image data, split all frame images, and analyze the quality of the framed images to discard unqualified images, including pure white, pure black, and images with blurred and irregular edges;

[0020] Cut the ultrasound image into grids, the number of which is preset by the operator according to the corresponding parameters in the measurement feature list, and each grid contains an independently detected ultrasound section and its cross-sectional structural features;

[0021] The convolutional neural network is trained using historical ultrasound image data with annotated cross-sectional structural features as a corpus. The entire framed image is used as input, multiple objects are detected simultaneously in one forward pass, all grids in the image are processed in parallel, and a set of predictions is generated for each grid.

[0022] According to the above technical solution, in step 105, ultrasonic features are detected in adjacent grids to obtain contour vector sets of adjacent grids, and the union of the current grid and the contour vector sets of adjacent grids is output to determine whether the union satisfies the vector closure condition. If all vector elements are closed, the closure condition is satisfied, and the union is output as the ultrasonic feature vector set. Otherwise, the detected ultrasonic features are discarded.

[0023] According to the contour formed by the vectors in the ultrasonic feature vector set, the area to be inspected is marked on the frame image, and the frame before and after the frame where the movement occurs is selected as the auxiliary frame. According to the frame rate and movement speed of the probe when the auxiliary frame is taken, the movement distance between the two frames is obtained;

[0024] According to the following formula, the contour similarity of the ultrasonic features in the current frame and the auxiliary frame is determined:

[0025] ;

[0026] Among them, SIM represents the contour similarity of ultrasound features, represents the jth vector in the ultrasonic feature vector set of the frame, represents the h+jth vector in the auxiliary frame ultrasound feature vector set, k is the number of elements in the frame ultrasound feature vector set, h is the assumed point number, and h∈{1, 2, ..., k};

[0027] Traverse all values ​​of h and find the maximum value of SIM as the confidence of the ultrasonic structural feature;

[0028] Marking the ultrasonic structural features in each frame of the image, and restoring the marked frame of the image to ultrasonic dynamic image data output;

[0029] Perform stereo mapping inversion based on the ultrasonic feature vector set of the current frame and the auxiliary frame, the probe height, the section height, and the moving distance between the two frames to obtain a three-dimensional model of the ultrasonic feature, extract the surface shape of the three-dimensional model, and output the surface vector flow of the ultrasonic feature;

[0030] The similarity between the surface vector flow of the feature area and various typical feature vector flows is detected through a large database, and all typical feature vector flow types with similarities higher than a threshold are output.

[0031] Ultrasound guidance method and system based on deep learning, the guidance system includes the following contents: ultrasonic dynamic image data acquisition terminal, image frame management module, ultrasonic dynamic image data real-time detection module, GPS navigation positioning feature module, ultrasonic dynamic image data section structure feature index module, ultrasonic standard section guidance module and ultrasonic dynamic image detection and analysis module;

[0032] The ultrasonic dynamic image data acquisition terminal is used to acquire ultrasonic dynamic image data; the image frame management module is used to perform frame processing on the acquired ultrasonic dynamic image data; the ultrasonic dynamic image data real-time detection module is used to detect the acquired ultrasonic dynamic image data; the GPS navigation positioning feature module locates and identifies the characteristic structure on the dynamic image according to the measured and calculated target width, height coordinates and confidence of the framed image, so as to help the ultrasound doctor to quickly diagnose the section and cross-sectional tissue structure; the ultrasonic image cross-sectional structure feature index module is used to divide the ultrasonic video dynamic image into multiple frames according to the frame parameters, and then perform target detection on each frame of the image and establish an index frame; the ultrasonic standard section guidance module is used to determine whether the real-time detection of ultrasonic dynamic image data has been detected, so as to guide the ultrasound doctor to conduct targeted scanning; the ultrasonic dynamic image detection and analysis module is used to analyze the inspected section and cross-sectional structure feature data after the ultrasonic image detection is completed.

[0033] According to the above technical solution, the guidance system also includes a real-time case playback module and a historical case playback module;

[0034] During ultrasound-guided testing, by clicking the playback function key, the system loads the previously recorded video and frame index document section and cross-sectional structural feature information, plays the case video with marked section and cross-sectional structural features, and realizes the real-time case playback function;

[0035] At the end of the ultrasound-guided test, the user can choose to replay a historical case. The system will load the recorded video and frame index document sections and cross-sectional structural features of the case, and play the historical case video with annotated sections and cross-sectional structural features.

[0036] According to the above technical solution, the image framing management module includes: an image framing unit, an image screening unit and a grayscale extraction unit;

[0037] The image framing unit is used to convert the ultrasound video into frame images and extract the corresponding frame images according to the preset framing parameters;

[0038] The image screening unit is used to clean the framed images and remove blurred images, completely black images and transparent images in each frame of the image;

[0039] The grayscale extraction unit is used to divide the image into a plurality of grids and identify the grayscale of each pixel in the grid.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The present invention can select adjacent frames before and after the frame where the characteristic contour is located, determine the confidence of the structural characteristic contour according to the ultrasound contour in the adjacent frames and the grayscale change, obtain the surface vector flow of the structural characteristic contour, and perform stereoscopic recognition of the ultrasound structural features, so as to obtain more precise and accurate tissue structure information and improve the detection and diagnosis efficiency of ultrasound dynamic imaging data.

[0042] The present invention can calculate the optimal section angle according to the tissue type, guide the ultrasound doctor to control the probe offset to the optimal angle for ultrasound measurement, mark the currently inspected tissue in all images containing the ultrasound structural features after measurement, and realize the function of ultrasound equipment to mark the detection section in real time and the detection structure features in real time, so as to better improve the ultrasound image quality, diagnostic accuracy and feature recognition accuracy.

[0043] The present invention can guide ultrasound doctors to control the probe, correctly position the probe, scan perfect cross-sectional images, and give reminders when the frame rate deviates from the normal range. This can obtain more detailed and accurate ultrasound images and help better identify abnormal areas in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic diagram of the steps of the ultrasound-guided method based on deep learning of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] Embodiment 1: Figure 1 As shown, an ultrasound guidance method based on deep learning comprises the following steps:

[0047] Step 101: using an acquisition terminal to acquire ultrasound dynamic image data, performing frame processing on the acquired ultrasound dynamic image data, and performing real-time detection on the frame-divided ultrasound dynamic image data;

[0048] In step 101, the ultrasound doctor installs a sensor on the ultrasound probe, obtains the location coordinates of the probe by relying on the satellite positioning or physical positioning function of the sensor, and obtains the movement speed and movement direction of the probe by relying on the dynamic sensing function of the sensor;

[0049] The adjustment coefficient t is determined according to the angle between the sound beam scanning plane and the horizontal plane and the motion vector of the probe, where t=| |·cos(θ+α), where is the motion vector of the probe, θ is the angle between the motion vector of the probe and the scanning plane of the acoustic beam, and α is the angle between the scanning plane of the acoustic beam and the horizontal plane;

[0050] Before the equipment is working, the shooting frame rate r1 when the probe is stationary is set, and the real-time frame rate r2 during the probe movement is determined in combination with the adjustment coefficient of the probe movement, so that r2 satisfies r2=r1+r1·t, and the ultrasound dynamic image data is shot at the real-time frame rate;

[0051] Frame the ultrasonic dynamic image data, split all frame images, and analyze the quality of the framed images to discard unqualified images, including pure white, pure black, and images with blurred and irregular edges;

[0052] The convolutional neural network is trained using historical ultrasound image data with annotated structural features as a corpus, taking the entire framed image as input, detecting multiple objects simultaneously in one forward pass, processing all grids in the image in parallel, and generating a set of predictions for each grid.

[0053] In an embodiment of the present invention, the ultrasound device is connected to the ultrasound acquisition terminal through a standard interface, wherein the standard interface may include any one of a VGA interface, a DVI interface, an HDMI interface, a CVBS interface, an S-Video interface, a color difference interface, and a USB interface, which is not limited in the embodiment of the present invention.

[0054] Step 102: After the detection of each frame of ultrasonic dynamic image data is completed, the characteristic structure is located by GPS navigation according to the detection result, and the detected features are marked with real-time coordinates;

[0055] Step 103. Record and establish a frame index of the structure feature of each frame of ultrasound dynamic image data section, each structure feature frame index corresponds to an ultrasound dynamic image data section, and the frame index file records the time sequence, coordinates and confidence information of the inspected section and the section structure feature frame;

[0056] Step 104. When the system is started, all standard sections are displayed by default, and the section status is displayed as unchecked. After each frame of detection, the sections with standard sections and section structure feature status that have been detected are displayed as checked, and the user is guided to use the probe to scan the unchecked standard sections, which plays a role in guiding the ultrasound standard sections;

[0057] Step 105. Detect and analyze the undetected sections based on deep learning technology;

[0058] In step 105, ultrasonic features are detected in adjacent grids to obtain contour vector sets of adjacent grids, and a union of the current grid and the contour vector sets of adjacent grids is output to determine whether the union satisfies a vector closure condition. If all vector elements are closed, the closure condition is satisfied, and the union is output as an ultrasonic feature vector set. Otherwise, the detected ultrasonic features are discarded.

[0059] According to the contour formed by the vectors in the ultrasonic feature vector set, the area to be inspected is marked on the frame image, and the frame before and after the frame where the movement occurs is selected as the auxiliary frame. According to the frame rate and movement speed of the probe when the auxiliary frame is taken, the movement distance between the two frames is obtained;

[0060] According to the following formula, the contour similarity of the ultrasonic features in the current frame and the auxiliary frame is determined:

[0061] ;

[0062] Among them, SIM represents the contour similarity of ultrasound features, represents the jth vector in the ultrasonic feature vector set of the frame, represents the h+jth vector in the auxiliary frame ultrasound feature vector set, k is the number of elements in the frame ultrasound feature vector set, h is the assumed point number, and h∈{1, 2, ..., k};

[0063] Traverse all values ​​of h and find the maximum value of SIM as the confidence of the ultrasonic section and cross-sectional structural features;

[0064] Marking ultrasound features in each frame of the image, and restoring the marked frames of the image to ultrasound dynamic image data output;

[0065] Perform stereo mapping inversion based on the ultrasonic feature vector set of the current frame and the auxiliary frame, the probe height, the section height, and the moving distance between the two frames to obtain a three-dimensional model of the ultrasonic feature, extract the surface shape of the three-dimensional model, and output the surface vector flow of the ultrasonic feature;

[0066] The similarity between the surface vector flow of the feature area and various typical feature vector flows is detected through a large database, and all typical feature vector flow types with similarities higher than a threshold are output.

[0067] Step 106. After the ultrasound imaging case detection is completed, the ultrasound imaging detection results are analyzed, the system summarizes the detected standard sections and the cross-sectional structural features they contain, and automatically generates the best standard section based on the standard section classification and confidence.

[0068] After the ultrasound doctor finishes the ultrasound examination, the system will summarize all the inspected standard sections and cross-sectional structures in the ultrasound guidance list, record the names and confidence levels of the inspected standard sections and cross-sectional structures, and select the section with the highest confidence level as the best standard section.

[0069] Embodiment 2: An ultrasound guidance system based on deep learning, the guidance system comprising the following contents: an ultrasound dynamic image data acquisition terminal, an image frame management module, an ultrasound dynamic image data real-time detection module, a GPS navigation positioning feature module, an ultrasound dynamic image data section structure feature indexing module, an ultrasound standard section guidance module and a digital analysis module;

[0070] The ultrasonic dynamic image data acquisition terminal is used to acquire ultrasonic dynamic image data; the image frame management module is used to perform frame processing on the acquired ultrasonic dynamic image data;

[0071] The ultrasonic dynamic image data real-time detection module is used to detect the collected ultrasonic dynamic image data;

[0072] The ultrasound doctor starts ultrasound case analysis, and the ultrasound dynamic image data real-time detection module obtains ultrasound dynamic images and divides the dynamic images into frames;

[0073] The real-time detection module of ultrasonic dynamic image data sends the ultrasonic framed images to the algorithm module for section structure feature detection;

[0074] Algorithm module target detection process:

[0075] a. receiving the preprocessed ultrasound image, the ultrasound image is cut into a number of grids, each grid is responsible for detecting the ultrasound section and cross-sectional structure of the area where it is located;

[0076] b. The entire image is taken as input and several objects are detected simultaneously in one forward pass; the detection module processes all grids in the image in parallel, and each grid generates a set of predictions, including the object category, the coordinates of the bounding box, the width and height of the box, and the confidence of each category. The object category is set to ultrasound section and cross-sectional structure;

[0077] c. Extract features from the ultrasound image in the convolution layer, including the edge, color distribution and shape of the ultrasound section and cross-sectional structure. The detection module determines the specific information of the ultrasound section and cross-sectional structure through the features;

[0078] d. The detection module identifies the shape of the ultrasonic section and the cross-sectional structure by analyzing the features within each grid, and identifies the shape of the ultrasonic section and the cross-sectional structure based on edge detection and shape features;

[0079] e. Based on the shape features of the ultrasonic feature contour frame, the detection module calculates the contour frame of the ultrasonic section and cross-sectional structure. Based on the ultrasonic feature contour frame, the contour frame includes the position coordinates and size of the ultrasonic section and cross-sectional structure in the image, which can identify the specific position of the ultrasonic section and cross-sectional structure.

[0080] The algorithm module returns the detection result, and the real-time detection module of ultrasonic dynamic image data calls the frame index to record the frame number, structural feature classification name, two-dimensional coordinates, and confidence information of the detected section structure feature;

[0081] At the same time, the real-time detection module of ultrasonic dynamic imaging data obtains the frame number, structural feature classification name, two-dimensional coordinates, and confidence information of the inspected section structure features in the returned results, and uses rectangular, polygonal, linear, and irregular frames to accurately locate and identify the inspected section structure in the current frame, and uses different colors to identify according to the confidence of the inspected features.

[0082] The GPS navigation positioning feature module locates and marks the ultrasound section and cross-sectional structure on the dynamic image according to the target width, height coordinates and confidence of the measured and calculated framed images, so as to help ultrasound doctors quickly diagnose the section and cross-sectional structure;

[0083] The real-time detection module of ultrasound dynamic image data sends instructions to the algorithm server to obtain the detection results of ultrasound frame image pictures. The detection results include the classification name of the detected structural features, two-dimensional coordinates (width, height), confidence, and the section to which they belong;

[0084] The real-time detection module of ultrasonic dynamic image data identifies the detected feature frame timestamp, serial number and two-dimensional coordinate information, just like GPS accurately locates and identifies the feature in the frame image, and then fills the identified static ultrasonic dynamic image data into the ultrasonic video analysis queue;

[0085] Play the ultrasound video analysis queue video stream, and the display interface displays the ultrasound dynamic image in real time, with the image accurately marking the structural features of each inspected section and cross section;

[0086] The ultrasound doctor selects a certain inspected section structure, and the system queries and displays the name, two-dimensional coordinates (width, height), and confidence information of the feature based on the serial number of the framed image and the feature classification name.

[0087] The ultrasonic image structure feature indexing module is used to split the ultrasonic video dynamic image into multiple frames according to the frame parameters, and then perform target detection on each frame image and establish an index frame;

[0088] The ultrasound standard section guidance module is used to determine whether the real-time ultrasound dynamic image data has been detected, thereby guiding the ultrasound doctor to perform targeted scanning;

[0089] The real-time detection module of ultrasound dynamic image data obtains the classification name, two-dimensional coordinates (width, height), confidence level, and section information of the inspected section structure features returned by the algorithm module;

[0090] The real-time detection module of ultrasound dynamic imaging data creates a standard section guide list when the case analysis is started. The standard section guide list contains all standard sections and cross-sectional structure information, which are sorted in the order of guiding standard section operations. Each standard section contains all cross-sectional structures of this section. The default status of the section and cross-sectional structure is undetected. When the algorithm module returns the detected section and cross-sectional structure, the standard section guide list will update the section cross-sectional structure status to detected, suspected or unknown according to the information of the detected section and cross-sectional structure characteristics.

[0091] The real-time detection module of ultrasonic dynamic imaging data calls the standard section structure diagram and data list set by the algorithm module according to the section classification name and section structure classification name information returned by the algorithm module. The standard section structure list contains the section classification name, the section structure features to which the section belongs (the status of all section structure features of the section is undetected by default), and the two-dimensional coordinates of the section features; the intelligent detection module matches the sections and features in the list according to the section to which the detected features belong and the feature classification name information. When the section structure feature status is detected, suspected, and unknown, the section structure feature status in the standard section structure class table is updated to detected, and other section structure features without detection results remain in the undetected state.

[0092] The real-time detection module of ultrasonic dynamic image data obtains the section structure name and two-dimensional coordinate information of the current standard section structure data list, marks the section structure feature name and its status on the standard structure diagram, and then renders the marked standard structure diagram into the current detected ultrasonic image for easy front-end display.

[0093] The ultrasound standard section guidance module displays the standard section guidance list and standard section structure diagram in real time on the front end. The standard section guidance list displays all the standard sections that should be inspected for this test item in a tree-like manner, with uninspected sections displayed in the front order, to guide ultrasound doctors to scan uninspected standard sections. The inspected features of the standard section structure are displayed with static green feature names, and the uninspected section structures are displayed with dynamic jumping gray feature names to prompt overdue doctors to scan and test as soon as possible.

[0094] The standard section feature guidance module displays undetected sections or cross-sectional structures, guiding ultrasound doctors to operate the probe for detection and preventing doctors from missing detection.

[0095] The standard section feature guidance module provides GPS navigation to locate the inspected, suspected or unknown standard sections or cross-sectional structures, guiding ultrasound doctors to confirm the inspected section structures, re-inspect the suspected or unknown section structures, and supplement the uninspected section structures.

[0096] The digital analysis module detects and analyzes uninspected surfaces based on deep learning technology.

[0097] The ultrasound guidance system also includes a real-time case playback module and a historical case review module;

[0098] The real-time case playback module is used to select playback guidance and view playback cases during the real-time guidance process;

[0099] During the real-time guidance process, the ultrasound doctor selects playback guidance, and the system records the current detection image frame sequence number as the playback end sequence number T, and sets the start playback time point of the current detection case video to 0 and the end playback time point to T;

[0100] During the video playback process of the system, the intelligent detection module loads the frame index document, queries the ultrasound dynamic image data frame index document according to the current frame sequence number, obtains the classification name, two-dimensional coordinates, and confidence information of all inspected section structures in the current frame, and then uses rectangular, polygonal, linear, and irregular frames to accurately locate and identify the names of the inspected features in the current frame, and uses different colors to identify them according to the confidence of the inspected features.

[0101] The video playback interface of the ultrasound standard section guidance module plays the video with identified features, and the statistical summary list queries the ultrasound dynamic image data frame index document records with frame sequence numbers from 0 to the playback end sequence number T, summarizes the section and cross-sectional structure feature information of the record set, and displays the section and cross-sectional structure detection status in a tree form.

[0102] When the ultrasound doctor selects a section to detect the cross-sectional structural features, the system pauses the video playback, locates and displays the ultrasound dynamic image data frame according to the sequence number and name of the cross-sectional structural features, and boldly marks the cross-sectional structural features in the image frame.

[0103] The ultrasound doctor double-clicks the video playback interface, and the system displays the playback interface in maximized or original page format.

[0104] Ultrasound doctors can choose to play historical videos at multiple speeds, such as 1x speed, 2x speed, etc.

[0105] Ultrasound doctors can select the screenshot function, and the system will take a screenshot of the current test page for test review.

[0106] The historical case review module is used to select a historical case after the real-time detection is completed, and the system opens the historical case video;

[0107] When the ultrasound doctor finishes the test, he selects a historical case, and the system opens the historical case video and plays the entire historical case video starting from time point 0.

[0108] During the video playback process of the system, the real-time detection module of ultrasonic dynamic image data loads the frame index document, queries the ultrasonic dynamic image data frame index document according to the current frame sequence number, obtains the classification name, two-dimensional coordinates, and confidence information of the current frame section structure feature, and then uses rectangular, polygonal, linear, and irregular frames to accurately locate and identify the name of the detected feature in the current frame, and uses different color identification according to the confidence of the detected section structure feature.

[0109] The video playback interface of the ultrasound standard section guidance module plays videos of marked sections and cross-sectional structures, and uses a statistical summary list to query the ultrasound dynamic image data frame index document records of historical case videos, summarizes the section and cross-sectional structure information of the record set, and displays the section and cross-sectional structure detection status in a tree format.

[0110] When the ultrasound doctor selects the inspected section structure feature of the trailer, the system pauses the video playback, locates and displays the ultrasound dynamic image data frame according to the frame number and coordinates of the section structure feature, and also boldly marks the section structure in the image frame.

[0111] The ultrasound doctor double-clicks the video playback interface, and the system displays the playback interface in maximized or original page format.

[0112] Ultrasound doctors can choose to play historical videos at multiple speeds, such as 1x speed, 2x speed, etc.

[0113] Ultrasound doctors can select the screenshot function, and the system will take a screenshot of the current test page for test review.

[0114] The image framing management module includes: an image framing unit, an image screening unit and a grayscale extraction unit;

[0115] The image framing unit is used to convert the ultrasound video into frame images and extract the corresponding frame images according to the preset framing parameters;

[0116] The image screening unit is used to clean the framed images and remove blurred images, completely black images and transparent images in each frame of the image;

[0117] The grayscale extraction unit is used to divide the image into a plurality of grids and identify the grayscale of each pixel in the grid.

[0118] Embodiment 3: Ultrasound guidance system based on deep learning, wherein the ultrasound image sections and cross-sectional structural features involved in the guidance system include the following:

[0119] It is suitable for the longitudinal section of the left lobe of the liver through the long axis of the abdominal aorta under the xiphoid process. As an anatomical organ, its cross-sectional structural characteristics include the splenic artery, pancreatic body, splenic vein, celiac artery, superior mesenteric artery, left atrium, esophagus, aorta and other characteristic structures; the longitudinal section of the liver through the long axis of the inferior vena cava under the xiphoid process. As an anatomical organ, its cross-sectional structural characteristics include the middle hepatic vein, common bile duct, portal vein, inferior vena cava, caudate lobe and other characteristics; the oblique sections of the left and right livers under the xiphoid process. As anatomical organs, their cross-sectional structural characteristics include the left lobe of the liver, the quadrate lobe (left inner lobe), the common hepatic duct, the left hepatic duct, the right hepatic duct, the right branch of the portal vein, the left branch of the portal vein, the caudate lobe, the inferior vena cava and other characteristic structures; the high oblique section of the first porta hepatis. As an anatomical organ, its cross-sectional structural characteristics include the right hepatic duct, the right branch of the portal vein, the left lobe of the liver, the left hepatic duct, the left branch of the portal vein and other characteristic structures.

[0120] Applicable to the cross-section of the thyroid gland. As an anatomical organ, its cross-sectional structural characteristics include the trachea, common carotid artery, internal jugular vein, esophagus, thyroid gland, spine, sternocleidomastoid muscle, left lobe of thyroid gland, right lobe of thyroid gland, isthmus of thyroid gland, sternohyoid muscle, sternothyroid muscle, thyroid nodule and other characteristic structures; the longitudinal section of the thyroid gland, as an anatomical organ, its cross-sectional structural characteristics include the trachea, common carotid artery, internal jugular vein, esophagus, thyroid gland, spine, left lobe of thyroid gland, right lobe of thyroid gland, isthmus of thyroid gland, thyroid nodule and other characteristic structures.

[0121] Applicable to breast sections. As an anatomical organ, its cross-sectional structural characteristics include the epidermis, subcutaneous tissue, glandular layer, muscle layer, breast nodules and other characteristics; the milk duct section, as an anatomical organ, its cross-sectional structural characteristics include the epidermis, subcutaneous fat, glandular layer, mammary ducts, mammary nodules and other characteristic structures.

[0122] Applicable to the posterior cross-section of the shoulder joint. As an anatomical organ, its cross-sectional structural characteristics include low-echo cartilage, scapular glenoid, joint capsule and other characteristic structures; the anterior long-axis section of the elbow joint, as an anatomical organ, its cross-sectional structural characteristics include the distal humerus, brachialis, pulley, coronoid process and other characteristic structures; the dorsal long-axis section of the wrist joint, as an anatomical organ, its cross-sectional structural characteristics include the radius, lunate, capitate and other characteristic structures; the anterior oblique sagittal section of the hip joint, as an anatomical organ, its cross-sectional structural characteristics include the femoral head, femoral neck, joint capsule and other characteristic structures.

[0123] It is applicable to the standard biparietal diameter section of the fetus, also known as the thalamic plane. As an anatomical organ, its cross-sectional structural characteristics include the thalamus, cavum septum pellucidum, lateral ventricle and other characteristic structures; the cerebellar section, as an anatomical organ, its cross-sectional structural characteristics include the cerebellum, posterior cranial fossa, posterior occipital cortex, cavum septum pellucidum and other characteristic structures; the midsagittal facial section, as an anatomical organ, its cross-sectional structural characteristics include the nasal bone, forehead, chin and other characteristic structures; the four-chamber heart section, as an anatomical organ, its cross-sectional structural characteristics include the left atrium, left ventricle, right atrium, right ventricle, mitral valve, tricuspid valve, endocardial cushion, descending aorta and other characteristic structures; the abdominal circumference section structure, as an anatomical organ, its cross-sectional structural characteristics include the gastric alveolar, umbilical vein, liver, descending aorta, ribs, inferior vena cava and other characteristic structures.

[0124] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. Ultrasound-guided method and system based on deep learning, characterized in that: The boot method comprises the following steps: Step 101. Using an acquisition terminal to acquire ultrasound dynamic image data, performing frame processing on the acquired ultrasound dynamic image data, and performing real-time detection on the frame-divided ultrasound image data; Step 102: After the detection of each frame of ultrasonic dynamic image data is completed, the characteristic structure is located by GPS navigation according to the detection result, and the detected features are marked with real-time coordinates; Step 103. Record and establish a frame index of the structural feature of each frame of ultrasound dynamic image data, each structural feature frame index corresponds to a section of ultrasound dynamic image data, and the frame index file records the time sequence, coordinates and confidence information of the inspected section and the structural feature of the section; Step 104. When the system is started, all standard sections are displayed by default, and the section status is displayed as unchecked. After each frame of detection, the sections with standard sections and section structure feature status that have been detected are displayed as checked, and the user is guided to use the probe to scan the unchecked standard sections, which plays a role in guiding the ultrasound standard sections; Step 105: Detect and analyze the undetected sections based on deep learning technology; Step 106. After the ultrasound imaging case detection is completed, the ultrasound imaging detection results are analyzed, the system summarizes the detected standard sections and the cross-sectional structural features they contain, and automatically generates the best standard section based on the standard section classification and confidence.

2. The ultrasound-guided method based on deep learning according to claim 1, characterized in that: The guide detection method also includes the following contents: During ultrasound-guided testing, by clicking the playback function key, the system loads the previously recorded video and frame index document section and cross-sectional structural feature information, plays the case video with marked section and cross-sectional structural features, and realizes the real-time case playback function; At the end of the ultrasound-guided test, the user can choose to replay a historical case. The system will load the recorded video and frame index document sections and cross-sectional structural features of the case, and play the historical case video with annotated sections and cross-sectional structural features.

3. The ultrasound-guided method based on deep learning according to claim 1, characterized in that: In step 101, the ultrasound doctor installs a sensor on the ultrasound probe to obtain the positioning coordinates of the probe, and obtains the movement speed and movement direction of the probe by relying on the dynamic sensing function of the sensor; The adjustment coefficient p is determined according to the angle between the ultrasonic scanning plane and the horizontal plane and the motion vector of the probe, where p=| |·cos(θ+α), where is the motion vector of the probe, θ is the angle between the motion vector of the probe and the scanning plane of the sound beam, and α is the angle between the ultrasonic scanning plane and the horizontal plane; Before the equipment is working, the shooting frame rate r1 when the probe is stationary is set, and the real-time frame rate r2 during the probe movement is determined in combination with the adjustment coefficient of the probe movement, so that r2 satisfies r2=r1+r1·p, and the ultrasound dynamic image data is shot at the real-time frame rate; Frame the ultrasonic dynamic image data, split all frame images, and analyze the quality of the framed images to discard unqualified images, including pure white, pure black, and images with blurred and irregular edges; Cut the ultrasound image into grids, the number of which is preset by the operator according to the corresponding parameters in the measurement feature list, and each grid contains an independently detected ultrasound section and its cross-sectional structural features; The convolutional neural network is trained using historical ultrasound image data with annotated standard sections and cross-sectional structural features as a corpus. The entire framed image is used as input, multiple objects are detected simultaneously in one forward pass, all grids in the image are processed in parallel, and a set of predictions is generated for each grid.

4. The ultrasound-guided method based on deep learning according to claim 3, characterized in that: In step 105, ultrasonic features are detected in adjacent grids to obtain contour vector sets of adjacent grids, and a union of the current grid and the contour vector sets of adjacent grids is output to determine whether the union satisfies a vector closure condition. If all vector elements are closed, the closure condition is satisfied, and the union is output as an ultrasonic feature vector set. Otherwise, the detected ultrasonic features are discarded. According to the contour formed by the vectors in the ultrasonic feature vector set, the area to be inspected is marked on the frame image, and the frame before and after the frame where the movement occurs is selected as the auxiliary frame. According to the frame rate and movement speed of the probe when the auxiliary frame is taken, the movement distance between the two frames is obtained; According to the following formula, the contour similarity of the ultrasonic features in the current frame and the auxiliary frame is determined: ； Among them, SIM represents the contour similarity of ultrasound features, represents the jth vector in the ultrasonic feature vector set of the frame, represents the h+jth vector in the auxiliary frame ultrasound feature vector set, k is the number of elements in the frame ultrasound feature vector set, h is the assumed point number, and h∈{1, 2, ..., k}; Traverse all values ​​of h and find the maximum value of SIM as the confidence of the ultrasonic feature; Marking ultrasound features in each frame of the image, and restoring the marked frames of the image to ultrasound dynamic image data output; Perform stereo mapping inversion based on the ultrasonic feature vector set of the current frame and the auxiliary frame, the probe height, the section height, and the moving distance between the two frames to obtain a three-dimensional model of the ultrasonic feature, extract the surface shape of the three-dimensional model, and output the surface vector flow of the ultrasonic feature; The similarity between the surface vector flow of the feature area and various typical feature vector flows is detected through a large database, and all typical feature vector flow types with similarities higher than a threshold are output.

5. An ultrasound-guided system based on deep learning, applied to the detection method according to any one of claims 1 to 4, characterized in that: The guidance system includes the following contents: an ultrasonic dynamic image data acquisition terminal, a dynamic image frame management module, an ultrasonic image data real-time detection module, a GPS navigation positioning feature module, an ultrasonic dynamic image section structure feature index module, an ultrasonic standard section guidance module and a digital analysis module; The ultrasonic dynamic image data acquisition terminal is used to acquire ultrasonic dynamic image data; the dynamic image frame management module is used to perform frame processing on the acquired ultrasonic dynamic image data; the ultrasonic image data real-time detection module is used to detect the acquired ultrasonic image data; the GPS navigation positioning feature module locates and identifies the characteristic structure on the dynamic image according to the measured and calculated target width, height coordinates and confidence of the framed image, so as to help the ultrasound doctor to quickly diagnose the section and cross-sectional structural features; the ultrasonic image section structural feature index module is used to divide the ultrasonic video dynamic image into multiple frames according to the frame parameters, and then perform target detection on each frame image, and establish a frame index document, which saves the frame number, section structural feature name, coordinates, detection confidence and other information; the ultrasonic standard section guidance module is used to determine whether the real-time detection of ultrasonic dynamic image data has been detected, so as to guide the ultrasound doctor to perform targeted scanning; the digital analysis module is used to perform digital analysis on the ultrasonic dynamic image data; the summary statistics module is used to summarize all the section structural features and their confidences of the detected section after the ultrasonic guided detection is completed, and automatically generate the best standard section and save it in the detection report.

6. The deep learning-based ultrasound guidance system according to claim 5, characterized in that: The guidance system also includes a real-time case playback module and a historical case review module; The real-time case playback module is used to select case playback during the real-time guidance process, and the system loads the previously recorded video and frame index section and section structure feature information, and plays the case video with the section and section structure features marked; The historical case review module is used to select a historical case after the real-time guidance is completed. The system will load the recorded video and frame index document sections and cross-sectional structural features of the case, and display the historical case video with marked sections and cross-sectional structural features.

7. The deep learning-based ultrasound guidance system according to claim 6, characterized in that: The image framing management module includes: an image framing unit, an image screening unit and a grayscale extraction unit; The image framing unit is used to convert the ultrasound video into frame images and extract the corresponding frame images according to the preset framing parameters; The image screening unit is used to clean the framed images and remove blurred images, completely black images and transparent images in each frame of the image; The grayscale extraction unit is used to divide the image into a plurality of grids and identify the grayscale of each pixel in the grid.

Images