A method and system for automatic bubble detection and classification based on ultrasonic video

Abstract

The present invention provides a method and system for automatic bubble detection and grading based on ultrasonic video, the system comprising an ultrasonic device for collecting cardiac ultrasonic video; a target tracking module for tracking the area to be detected in each frame of the ultrasonic video; an ROI area detection module for calculating the ROI area in the area to be detected, that is, the area where bubbles may exist; and a trained 3D convolutional neural network for automatically detecting the input ultrasonic video to obtain an automatic grading result. The method and system for automatic bubble detection and grading based on ultrasonic video of the present invention can realize automatic bubble detection and grading of ultrasonic video, thereby assisting in judging the severity of decompression sickness, and largely avoiding relying on the operator's experience and subjective judgment to grade decompression sickness.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an automatic bubble detection and grading method and system based on ultrasonic video. Background Art

[0002] In diving, especially deep diving, even if the decompression plan is fully followed, bubbles may be generated in the diver's body, causing venous gas embolism (VGE). In many cases, due to the small number of bubbles, divers will not have obvious symptoms of decompression illness (DCI) in most cases, so venous gas embolism is difficult to detect.

[0003] Studies have shown that the number of bubbles in the body is positively correlated with the risk of decompression sickness, so it is particularly important to develop a method for detecting bubbles in blood vessels that cause decompression sickness. In recent years, visual monitoring using two-dimensional ultrasound imaging has been used in the field of decompression sickness. Compared with Doppler ultrasound measurement, this method can detect bubbles in the left and right hearts at the same time, is easy to monitor, and has lower requirements for operators. At present, the EB (Eftedal-Brubakk) grading system developed in 1997 is the most commonly used two-dimensional ultrasound image grading system. It is divided into six levels from 0 to 5. The higher the level, the higher the risk of decompression sickness. However, this method still needs to rely on the operator's experience and subjective judgment to grade decompression sickness, and has disadvantages such as strong subjectivity and inability to automatically identify bubbles. Summary of the invention

[0004] In view of the above-mentioned defects of the prior art, the object of the present invention is to provide a method and system for automatic bubble detection and grading based on ultrasonic video, which can realize automatic bubble detection and grading of ultrasonic video, thereby assisting in judging the severity of decompression sickness, and largely avoiding relying on the operator's experience and subjective judgment to grade decompression sickness.

[0005] To achieve the above object, on the one hand, the present invention provides a method for automatic bubble detection and classification based on ultrasonic video, comprising the following steps:

[0006] Step S1: reading the ultrasound video of the heart and selecting the area where bubbles are to be identified, i.e., the area to be detected;

[0007] Step S2: using a target tracking algorithm to track the area to be detected in each frame of the ultrasound video;

[0008] Step S3: in a frame of video image, calculating the ROI area (Region of Interest) in the area to be detected, that is, the area where bubbles may exist;

[0009] Step S4: Repeat step S3 until all frames of the ultrasound video are traversed;

[0010] Step S5: constructing a data set using the ROI regions and corresponding decompression sickness grades of the plurality of ultrasound videos obtained in steps S1 to S4;

[0011] Step S6: training a 3D convolutional neural network (CNN) using the data set constructed in step S5;

[0012] Step S7: Use the 3D convolutional neural network trained in step S6 to automatically detect the input ultrasound video to be detected, and obtain an automatic classification result.

[0013] Furthermore, the ultrasound video described in step S1 is a cardiac ultrasound video collected by an ultrasound device, and the video needs to be converted into a grayscale image after being read.

[0014] Furthermore, the selection of the area where bubbles are to be identified in step S1 is that the operator manually selects the area where bubbles are to be identified in the first frame of video image.

[0015] Furthermore, the use of a target tracking algorithm to track the area to be detected in the ultrasound video in step S2 is to track the area where bubbles are to be detected in each frame of the video image. The target tracking algorithm may be, for example, a KCF (Kernel Correlation Filter) algorithm.

[0016] Furthermore, the step S3 of calculating the ROI area in the area to be detected includes the following steps:

[0017] i) The difference method between adjacent frames of N1 consecutive frames is used to obtain the difference image of frame N1-1. The variance of the corresponding pixel points in the difference image of frame N1-1 is judged. If the variance is greater than the threshold θ1, it is considered to be the ROI area, that is, the area where bubbles may exist. Specifically, if the frame numbers of the two subtracted images are the kth frame and the (k+1)th frame, they are represented by f respectively. k (x,y),f k+1 (x, y), the difference image is D k (x, y) is expressed as:

[0018] D k (x, y) = f k+1 (x, y)-f k (x,y),k=1,2,3,…,N1-1

[0019] The ROI region R′ of the i-th frame i (x, y) is expressed as:

[0020]

[0021] When there are not enough frames before the i-th frame, the first frame is used to supplement it, and N1 is an odd number.

[0022] ii) Due to the interference of factors such as valves, the ROI area obtained above needs to be further de-interfered to obtain the ROI area after de-interference. It can be observed that the valve shows periodic motion within a certain range, and bubbles are constantly generated and disappeared. Therefore, it is first necessary to find similar images in consecutive N2 frames through the normalized cross-correlation method, and the formula for determining whether they are related is expressed as:

[0023]

[0024] Among them, f represents the image, represents the mean of the template image to be matched. is the mean value of f(x, y) under the image template. In this method, t is the current i-th frame image, and f is an image frame in the continuous N2 frames. The maximum value of the obtained correlation coefficient matrix γ(u, u) is the correlation coefficient between the i-th frame image and another frame. By finding the peak value of the correlation coefficient in the continuous N2 frames, the corresponding number of frames with the greatest similarity can be obtained:

[0025]

[0026] The N2-frame ultrasound image is binarized by threshold θ2, and the intersection of corresponding pixel points of the continuous N2-frame images is used to obtain the seed point of the ultrasound image valve. The result of binarization of the k-th frame image is:

[0027]

[0028] The seed point S of the valve of consecutive N2 frames i (x, y) is expressed as:

[0029] S i (x, y) = ΠB k (x, y, θ2)

[0030] For each specific valve in each frame, it is first binarized by threshold θ3, and then morphological operations are performed to eliminate small areas, separate the connection between valves and bubbles in thin areas, and smooth the boundaries of larger valves. Its structural element is diamond, and the distance from the origin to the diamond point is a. The region growing method needs to process the grayscale image, so the binary image obtained above is multiplied by the original image of this frame to obtain the ultrasound image L of the limited area i (x, y):

[0031]

[0032] Then, the ultrasound image L i (x, y) using the obtained seed point S i (x, y), using the region growing method to find the exact size V of the valve in each frame i (x, y), where the threshold θ of the region growing method v .

[0033] iii) Finally, the ROI area R′ in step i) i (x, y) minus the valve area V from step ii) i (x, y) is the ROI area R after removing interference. i (x, y).

[0034] Furthermore, the method of finding similar images in continuous N2 frames by the normalized cross-correlation method described in the above step ii) is due to the periodic beating of the heart. Therefore, the related images found by this method can correspond to ultrasound images collected at the same time in different cardiac cycles. Theoretically, two frames of related images are one cardiac cycle apart. Today's cardiac ultrasound equipment is often equipped with an ECG interface. Therefore, when using cardiac ultrasound data collected with an ECG interface, each image can be phase-marked according to the cardiac cycle through the electrocardiogram signal, so as to directly obtain the ultrasound image corresponding to the frame to be processed at the same time in different cardiac cycles, avoiding errors caused by inaccurate related frames calculated by the normalized cross-correlation method.

[0035] Furthermore, the step S3 described in step S4 is repeated until all frames of the ultrasound video are traversed, and the video images in the ultrasound video are processed frame by frame until the bubble ROI areas of all frames of the ultrasound video are identified.

[0036] Furthermore, the decompression sickness levels described in step S5 include six levels from 0 to 5.

[0037] Furthermore, when training the 3D convolutional neural network in step S6, the input of the network is the ROI region of the ultrasound video frame sequence, and the output is the corresponding decompression sickness level 0 to 5.

[0038] Furthermore, step S7 first obtains the bubble ROI area of ​​the ultrasound video through steps S1 to S4, and then inputs the data into the trained 3D convolutional neural network to obtain the final classification result.

[0039] On the other hand, the present invention also provides an automatic bubble detection and grading system based on ultrasonic video, comprising:

[0040] Ultrasound equipment for collecting cardiac ultrasound video;

[0041] A target tracking module, used for tracking the area to be detected in each frame of the ultrasound image of the ultrasound video;

[0042] A ROI region detection module is used to calculate the ROI region in the area to be detected, that is, the area where bubbles may exist;

[0043] The trained 3D convolutional neural network is used to automatically detect the input ultrasound video to be detected and obtain automatic classification results.

[0044] Furthermore, the ultrasound device has an ECG interface.

[0045] Furthermore, the target tracking module adopts a target tracking algorithm, such as a KCF (Kernel Correlation Filter) algorithm.

[0046] Furthermore, the 3D convolutional neural network is trained by constructing a data set based on the ROI region and the corresponding decompression sickness level, the input is the ROI region of the ultrasound video frame sequence, and the output is the corresponding decompression sickness level 0 to 5.

[0047] Furthermore, the target tracking module, the ROI area detection module and the 3D convolutional neural network are integrated into an intelligent computer device, such as a desktop computer, a laptop computer, a tablet computer or a smart phone.

[0048] The present invention has the following beneficial effects:

[0049] The present invention proposes an automatic bubble detection and grading method and system based on ultrasonic video, which can realize automatic bubble detection and grading of ultrasonic video, thereby assisting in judging the severity of decompression sickness, and largely avoiding relying on the operator's experience and subjective judgment to grade decompression sickness. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flow chart of an automatic bubble detection and classification method based on ultrasonic video according to a preferred embodiment of the present invention;

[0051] Figure 2 is a schematic diagram of a frame from a pig's cardiac ultrasound video;

[0052] Figure 3 After identification by the present invention Figure 2 Schematic diagram of the ROI area in this frame. DETAILED DESCRIPTION

[0053] The embodiments of the present invention are described in detail below. The following embodiments are implemented on the premise of the technical solution of the present invention, and detailed implementation methods and specific operation processes are given, but the protection scope of the present invention is not limited to the following embodiments.

[0054] like Figure 1 As shown, in a preferred embodiment, a method for automatic bubble detection and classification based on ultrasonic video of the present invention comprises the following steps:

[0055] Step S1: In this embodiment, the ultrasound video to be read is a pig heart ultrasound video collected by ultrasound equipment. After reading the video, each frame of the video image needs to be converted into a grayscale image. The operator manually selects the area in the first frame of the video image where the bubble is to be identified, that is, the area to be detected. The area is a rectangular area, represented by (x col , x row , d col , d row ). Among them, x col Indicates that the upper left corner of the area is the xth col Column, x row Indicates that the upper left corner of the area is the xth row Line, d col and d row Respectively represent the number of columns and rows occupied by the area.

[0056] Step S2: Use a target tracking algorithm to track the area, and track the desired area to be detected in each frame of video image. In this embodiment, the target tracking algorithm uses a KCF (Kernel Correlation Filter) algorithm, which uses HOG features and Gaussian kernel functions to track the area to be detected in each frame of video image.

[0057] Step S3: Calculate the ROI area in the area to be detected, that is, the area where bubbles may exist, which specifically includes the following steps:

[0058] i) The difference method between adjacent frames of N1 consecutive frames is used to obtain the difference image of frame N1-1. The variance of the corresponding pixel points in the difference image of frame N1-1 is judged. If the variance is greater than the threshold θ1, it is considered to be the ROI area, that is, the area where bubbles may exist. Specifically, if the frame numbers of the two subtracted images are the kth frame and the (k+1)th frame, they are represented by f respectively. k (x,y),f k+1 (x, y), the difference image is D k (x, y) is expressed as:

[0059] D k (x, y) = f k+1 (x, y)-f k(x,y),k=1,2,3,…,N1-1

[0060] The ROI region R′ of the i-th frame i (x, y) is expressed as:

[0061]

[0062] When there are not enough frames before the i-th frame, the first frame is used to supplement it, and N1 is an odd number.

[0063] In this embodiment, the value of N1 is 15, and θ1 is 0.05 (max (R′ i (x, y))-min(R′ i (x, y))).

[0064] ii) Due to the interference of factors such as valves, the ROI area obtained above needs to be further de-interfered to obtain the ROI area after de-interference. It can be observed that the valve shows periodic motion within a certain range, and bubbles are constantly generated and disappeared. Therefore, it is first necessary to find similar images in consecutive N2 frames through the normalized cross-correlation method, and the formula for determining whether they are related is expressed as:

[0065]

[0066] Among them, f represents the image, represents the mean of the template image to be matched. is the mean value of f(x, y) under the image template. In this method, t is the current i-th frame image, and f is an image frame in the continuous N2 frames. The maximum value of the obtained correlation coefficient matrix γ(u, v) is the correlation coefficient between the i-th frame image and another frame. By finding the peak value of the correlation coefficient in the continuous N2 frames, the corresponding number of frames with the greatest similarity can be obtained:

[0067]

[0068] The heart beats periodically, so the related images found by the normalized cross-correlation method can correspond to ultrasound images collected at the same time in different cardiac cycles. Theoretically, two frames of related images are one cardiac cycle apart. Today's cardiac ultrasound equipment is often equipped with an ECG interface. Therefore, when using cardiac ultrasound data collected with an ECG interface, each image can be phase-marked according to the cardiac cycle through the electrocardiogram signal, so as to directly obtain the ultrasound image corresponding to the frame to be processed at the same time in different cardiac cycles, avoiding errors caused by inaccurate related frames calculated by the normalized cross-correlation method.

[0069] The N2-frame ultrasound image is binarized by threshold θ2, and the intersection of corresponding pixel points of the continuous N2-frame images is used to obtain the seed point of the ultrasound image valve. The result of binarization of the k-th frame image is:

[0070]

[0071] The seed point S of the valve of consecutive N2 frames i (x, y) is expressed as:

[0072] S i (x, y) = ΠB k (x, y, θ2)

[0073] For each specific valve in each frame, it is first binarized by threshold θ3, and then morphological operations are performed to eliminate small areas, separate the connection between valves and bubbles in thin areas, and smooth the boundaries of larger valves. Its structural element is diamond, and the distance from the origin to the diamond point is a. The region growing method needs to process the grayscale image, so the binary image obtained above is multiplied by the original image of this frame to obtain the ultrasound image L of the limited area i (x, y):

[0074]

[0075] Then, the ultrasound image L i (x, y) using the obtained seed point S i (x, y), using the region growing method to find the exact valve size V in each frame i (x, y), where the threshold θ of the region growing method v .

[0076] In this embodiment, the value of N2 is 15, θ2 is 0.5*255, θ3 is 0.35*255, a is 4, θ v It is 0.22.

[0077] iii) Finally, the ROI area R′ in step i) i (x, y) minus the valve area V from step ii) i (x, y) is the ROI area R after removing interference. i (x, y), such as Figure 2 shown.

[0078] Step S4: Repeat step S3 until all frames of the video image are traversed, that is, the ROI areas where bubbles may exist in all frames of the ultrasound video are identified.

[0079] Step S5: construct a data set with the ROI area of ​​the ultrasound video obtained in steps S3 to S4 and the corresponding decompression sickness level to train the neural network. There are six levels of decompression sickness, 0 to 5.

[0080] Step S6: Use the constructed data set to train a 3D convolutional neural network, the input of the 3D convolutional neural network is the bubble ROI area of ​​the ultrasound video frame sequence, and the output is the corresponding decompression sickness level (0-5).

[0081] Step S7: Use the trained 3D convolutional neural network to automatically classify the ultrasound video to be detected. Specifically, first obtain the ROI area of ​​the ultrasound video to be detected through steps S1 to S4, and then input the data into the trained 3D convolutional neural network to obtain the automatic classification result. In this embodiment, the automatic classification result obtained is level 5.

[0082] In a preferred embodiment, an automatic bubble detection and grading system based on ultrasonic video of the present invention includes: an ultrasonic device for collecting cardiac ultrasonic video; a target tracking module for tracking the area to be detected in each frame of the ultrasonic image of the ultrasonic video; an ROI area detection module for calculating the ROI area in the area to be detected, that is, the area where bubbles may exist; and a trained 3D convolutional neural network for automatically detecting the input ultrasonic video to be detected and obtaining an automatic grading result.

[0083] The ultrasound device has an ECG interface.

[0084] The target tracking module adopts a target tracking algorithm, such as a KCF (Kernel Correlation Filter) algorithm.

[0085] The 3D convolutional neural network is trained by constructing a data set through ROI regions and corresponding decompression sickness levels, the input is the ROI region of the ultrasound video frame sequence, and the output is the corresponding decompression sickness level 0 to 5.

[0086] The target tracking module, the ROI area detection module and the 3D convolutional neural network are integrated into an intelligent computer device, such as a desktop computer, a laptop computer, a tablet computer or a smart phone.

[0087] The preferred specific embodiments of the present invention are described in detail above. It should be understood that ordinary technicians in the field can make many modifications and changes based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by technicians in the technical field based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for automatic bubble detection and classification based on ultrasonic video, characterized in that: The following steps are involved: Step S1: reading the ultrasound video of the heart and selecting the area where bubbles are to be identified, i.e., the area to be detected; Step S2: using a target tracking algorithm to track the area to be detected in each frame of the ultrasound video; Step S3: in a frame of video image, calculating the ROI area in the area to be detected, that is, the area where bubbles may exist; Step S4: Repeat step S3 until all frames of the ultrasound video are traversed; Step S5: constructing a data set using the ROI regions and corresponding decompression sickness grades of the plurality of ultrasound videos obtained in steps S1 to S4; Step S6: training a 3D convolutional neural network using the data set constructed in step S5; Step S7: Use the 3D convolutional neural network trained in step S6 to automatically detect the input ultrasound video to be detected, and obtain an automatic classification result; wherein, The step S3 of calculating the ROI area in the area to be detected comprises the following steps: i) The difference method between adjacent frames of N1 consecutive frames is used to obtain the difference image of frame N1-1. The variance of the corresponding pixel points in the difference image of frame N1-1 is judged. If the variance is greater than the threshold θ1, it is considered to be the ROI area, that is, the area where bubbles may exist. Specifically, if the frame numbers of the two subtracted images are the kth frame and the (k+1)th frame, they are represented by f respectively. k (x,y),f k+1 (x, y), the difference image is D k (x, y) is expressed as: D k (x,y)=f k+1 (x,y)-f k (x,y),k=1,2,3,…,N1-1 The ROI region R′ of the i-th frame i (x, y) is expressed as: When there are not enough frames before the i-th frame, the first frame is used to supplement it, and N1 is an odd number; ii) To further remove the interference of the valve, firstly, similar images in consecutive N2 frames are found by the normalized cross-correlation method, and the formula for determining whether they are related is expressed as: Among them, f represents the image, represents the mean of the template image to be matched. is the mean value of f(x, y) under the image template; t in this method is the current i-th frame image, and f represents an image frame in the continuous N2 frames; the maximum value of the obtained correlation coefficient matrix γ(u, v) is the correlation coefficient between the i-th frame image and another frame; by finding the peak value of the correlation coefficient in the continuous N2 frames, the corresponding number of frames with the largest similarity can be obtained: The N2-frame ultrasound image is binarized by threshold θ2, and the intersection of corresponding pixel points of the continuous N2-frame images is used to obtain the seed point of the ultrasound image valve; the result of binarization of the k-th frame image is: The seed point S of the valve of consecutive N2 frames i (x, y) is expressed as: For each specific valve in each frame, it is first binarized by threshold θ3, and then morphological operations are performed to eliminate small areas, separate the connection between valves and bubbles in thin areas, and smooth the boundaries of larger valves. Its structural element is diamond, and the distance from the origin to the diamond point is a; the binary image obtained above is multiplied by the original image of this frame to obtain the ultrasound image L of the limited area i (x, y): Then, the ultrasound image L i (x, y) using the obtained seed point S i (x, y), using the region growing method to find the exact size V of the valve in each frame i (x, y), where the threshold θ of the region growing method v ; iii) Finally, the ROI area R′ in step i) i (x, y) minus the valve area V from step ii) i (x, y) is the ROI area R after removing interference. i (x, y); In step S7, the bubble ROI area of ​​the ultrasound video is first obtained through steps S1 to S4, and then the data is input into the trained 3D convolutional neural network to obtain the final classification result.

2. The method for automatic bubble detection and classification based on ultrasonic video according to claim 1, characterized in that: The ultrasound video described in step S1 is a cardiac ultrasound video collected by an ultrasound device, and needs to be converted into a grayscale image after reading the video; the selection of the area where bubbles are to be identified is performed by the operator manually selecting the area where bubbles are to be identified in the first frame of the video image.

3. The method for automatic bubble detection and classification based on ultrasonic video according to claim 1, characterized in that: The target tracking algorithm described in step S2 is the KCF algorithm.

4. The method for automatic bubble detection and classification based on ultrasonic video according to claim 1, characterized in that: The decompression sickness level described in step S5 has six levels from 0 to 5.

5. The method for automatic bubble detection and classification based on ultrasonic video according to claim 1, characterized in that: The training 3D convolutional neural network described in step S6 has an input of the ROI region of the ultrasound video frame sequence, and an output of the corresponding decompression sickness grade 0 to 5.

Images