Method for rapidly and accurately extracting ROI (Region of Interest) of ultrasonic cardiac video frame based on time-space complementary information

Abstract

A time-space complementary information-based ultrasonic cardiac video frame ROI rapid and accurate extraction method comprises the following steps of inputting an ultrasonic cardiac video, and extracting an initial mask in combination with time-space complementary information; correcting the initial mask; and drawing the corrected initial mask by using a Hough transform straight line detection method to obtain a standard fan-shaped mask. According to the method, after an ROI initial mask is obtained by combining an input echocardiography video with time-space complementary information, geometric information for drawing a standard mask is obtained from the improved initial mask, and thus a standard ROI mask is obtained. The method does not need a large amount of training data, does not have the problem of algorithm overfitting, and has good accuracy and stability.

Description

Technical Field

[0001] The invention relates to the technical field of automatic diagnosis of ultrasonic medical images, and in particular to a method for quickly and accurately extracting ROI of ultrasonic cardiography video frames based on time-space complementary information. Background Art

[0002] With the rapid development of deep learning technology, automatic diagnosis of cardiac ultrasound medical images based on deep learning has gradually become a hot topic. These technologies can be trained based on a large amount of image data and learn based on the results of physician annotation, so that the physician's judgment experience can be stored in the form of weights to achieve the purpose of automated and efficient diagnosis. In addition to the influence of model selection and parameter regulation, another key core for the final result of deep learning is the quality of the data. It can be clearly seen from Figure 1 (a) that in addition to the region of interest (ROI) shown in the red fan-shaped box of the output of the hospital ultrasound equipment, the entire image also contains information such as equipment parameters, resolution, afterglow, thermal index, mechanical precision, etc. in the blue box shown in Figure 1 (b).

[0003] For the automatic interpretation and analysis of ultrasound cardiac images, in order to effectively evaluate the motion and morphological characteristics of the heart area and reduce the interference of other irrelevant information, it is necessary to remove the information of non-interested areas, otherwise it will affect the reasoning results after model learning. However, there is still little research on the automatic extraction of image data areas in the above-mentioned ultrasound video frames. The main reason is that the automatic analysis of ultrasound cardiac images is almost all based on public data sets. Currently, the widely used ultrasound cardiac public data sets include EchoNet-LVH, CAMUS, HMC-QU, TED and other data sets. Figure 2(a) to Figure 2(d) As shown in the figure, the ultrasound video frames in these data sets have been preprocessed by manual classification, filtering out interference information, and removing equipment information such as time resolution, detection depth, afterglow, etc. This protects patient privacy on the one hand, and greatly saves researchers' preparation time for ultrasound data sets on the other.

[0004] However, in practical applications, the data types in public datasets are limited. There are certain differences in ultrasound images of different regions and ages. If the ultrasound public dataset is used to train the model, it is far from enough, which will limit the final training and reasoning results. In addition, when the trained model is used for the actual data of different hospitals using the ultrasound public dataset, there will be problems of inapplicability. This is mainly because there is a significant difference between the data distribution of the ultrasound public dataset and the output images of the equipment used in each hospital. Therefore, in order to obtain better reasoning results in practical applications, it is necessary to retrain in the private ultrasound public dataset. When training the model, for private data, it is necessary to remove the interference information outside the red sector area in Figure 1 (a) and extract the ultrasound information area. If manual extraction is used, it is not only time-consuming and labor-intensive, but also seriously affects the degree of automation of the entire process. Therefore, it is necessary to use image processing algorithms to automatically extract the ultrasound information area, that is, extract ROI.

[0005] Since the ROI of images in different application scenarios is different, specific analysis is required according to specific problems, so there is no unified paradigm. In the field of image region recognition and extraction, Hough transform detection and threshold segmentation methods based on the maximum connected region are widely used. Hough transform detection is often used to detect regular regions.

[0006] For example, the method of using Hough transform to detect circles is used to realize fruit recognition and thus estimate yield. The method of using Hough transform to detect straight lines is used to detect and identify the four sides of a rectangular original and then segment it. Reference [2]: Wang Ying, Wang Pengfei, Yang Yuwang. Target size feature measurement based on image segmentation. Computer Technology and Development, 2018, 28(02): 191-195.

[0007] Reference [3]: He Yibin, Ma Dong, Li Jingwei, et al. A method and system for image recognition and detection of key components of high-speed trains [P]. Hubei Province: CN202210627624.0, 2022-08-23. It is recorded in that after denoising the images of key parts of the train, the standard circular parts are detected using the improved Hough transform to obtain the parameters of non-standard circular parts, and then the non-standard parts are detected by the spatial projection segmentation method. However, for echocardiography, as shown in Figures 3(a) and 3(b), although the information area in the echocardiogram is a regular fan-shaped area, the obtained video display area is irregular, and there are often serious edge loss and large internal cavities.

[0008] The threshold segmentation method based on the maximum connected area can be used to segment the target subject in the image when the target subject is partially intact and the edges are clear. For example, in the study of the segmentation algorithm of overlapping elliptical leaves under complex background, the maximum connected area is selected to determine the main body of the overlapping leaves, thereby achieving the subsequent segmentation purpose. Reference [5]: Yang Chaoqiang, Yan Hao, Luo Chun. Medical image processing method, device, electronic device and storage medium [P]. Shaanxi Province: CN202110514772.7, 2022-12-02. After obtaining the image segmentation result output by the pre-trained segmentation model after the medical image is input, the corresponding maximum connected area in the image segmentation result is used as the target segmentation area of ​​the category label, thereby avoiding the misclassification of the image main target segmentation. Reference [6]: Jiao Liangbao, Cao Xuehong, Ye Qiling, et al. A method for identifying the type of GPR image disease based on shape features [P]. Jiangsu Province: CN201811376241.0, 2019-03-29. In order to realize the method of identifying the type of GPR image disease based on shape features, the pre-processed GPR image is processed using the maximum connected area to determine the shape of the GPR image disease and thus determine the disease type. However, as shown in FIG4( a ), in an ultrasound video, the maximum connected region of a single frame cannot completely cover the ultrasound information region, which is insufficient for extracting the standard fan-shaped region of the ROI.

[0009] In order to obtain a standard mask, it is necessary to segment the image area in the ultrasound video frame so as to use parameters to calculate the geometric parameters of the corresponding fan-shaped area. However, due to the presence of some special noise in the ultrasound video data itself, the widely used threshold segmentation and Hough transform straight line estimation algorithm cannot solve this problem well. As shown in Figure 4(a), if the ultrasound image area is extracted by threshold segmentation technology and the maximum connected region search strategy, due to the problem of angle change during ultrasound image scanning, the ultrasound information area is often seriously missing after the single frame of the ultrasound image is binarized and threshold segmented. Even if a relatively complete image area as shown in Figure 4(b) can be segmented in special cases, the accompanying electrocardiogram part (the undulating curve in the figure, different colors) will often be extracted at the same time, and most of all ultrasound video frequencies do not have such a complete image area.

[0010] If the ROI is extracted by combining the information of multiple frames in the video, as shown in Figure 4(c), since the mask needs to be binarized, the grayscale area of ​​the edge will be set to black (set to 0) below the threshold or white (set to 255) above the threshold after binarization, which will cause serious aliasing in the final image area, as shown in Figure 6(a). If Hough transform detection is used at this time, multiple difficult-to-distinguish line segments will often be detected, as shown in Figure 6(b), and the extracted ROI area is not a regular image, which is not very suitable for Hough transform detection.

[0011] In addition, since the heart itself has multiple chambers and is a three-dimensional target, similar to the size of a fist, there are often large areas of holes in the ultrasound information area of ​​the ultrasound cardiogram in different sections, as shown in Figure 4(d), resulting in the grayscale value base of the image area in the lower fan-shaped area being even 0, as shown in Figure 7(a). For the method that directly detects edges and combines Hough, the surrounding interference information and large and unavoidable holes will cause detection failure, as shown in Figure 7(b).

[0012] The mask extraction of the ROI area of ​​the ultrasound video based on the machine learning method requires training a classifier to distinguish the ROI area from the non-ROI area at the pixel level. The core is to use a large number of labeled training samples to fit the parameters of the classifier, and the performance of the algorithm is highly dependent on the quality and representativeness of the training samples, and it is also prone to overfitting. For traditional threshold segmentation and Hough transform detection, due to the particularity of ultrasound video, some ROI areas will be seriously missing in a single frame image, and the initial mask obtained by combining the image information in multiple frames will have jagged phenomena. For these reasons, the extracted ROI mask is incomplete and irregular. Summary of the invention

[0013] In order to solve the above technical problems, the present invention utilizes the temporal continuity of the video image frames and the spatial difference of the image information area, and the information complementarity between frames in the cardiac ultrasound video, and proposes a method for fast and accurate extraction of ultrasound video frame ROI based on spatiotemporal complementary information. After obtaining the ROI initial mask by combining the input ultrasound video with the spatiotemporal complementary information, the geometric information for drawing the standard mask is obtained from the improved initial mask, thereby obtaining a standard ROI mask. This method does not require a large amount of training data, nor does it have the problem of algorithm overfitting, and has good accuracy and stability.

[0014] The technical solution adopted by the present invention is:

[0015] The method for quickly and accurately extracting ROI from ultrasound video frames based on spatiotemporal complementary information includes the following steps:

[0016] Step 1: Input the echocardiographic video and extract the initial mask by combining the spatiotemporal complementary information;

[0017] Step 2: Correct the initial mask;

[0018] Step 3: Draw the corrected initial mask using the Hough transform method to detect straight lines to obtain a standard fan-shaped mask.

[0019] The step 1 comprises the following steps:

[0020] S1.1: Taking advantage of the overlapping of interference information between single video frames and the continuous movement of ultrasound information areas, the read ultrasound cardiography video is divided into frames and subtracted after several frames. Since the interference information is static, after subtracting after several frames, the offset part is the interference information, which is the background; the ultrasound information area with differences is successfully retained and is the foreground area;

[0021] S1.2: Traverse all video frames and perform subtraction at intervals of several frames, retain and accumulate the average value, at which time the foreground area information retained is the most complete;

[0022] S1.3: After obtaining the complete foreground area, the ultrasound information area is retained using the maximum connected region algorithm, and the retained ultrasound information area is the initial mask.

[0023] When the heart rate curve is overlaid on the ultrasound cardiogram, the green channel in the pixel points of the ultrasound cardiogram is thresholded so that the pixel points that are displayed as green as a whole are blackened, and the gray area in the ultrasound cardiogram is retained to the greatest extent.

[0024] The step 2 comprises the following steps:

[0025] S2.1: First, dilation and erosion operations are performed on the initial mask;

[0026] S2.2: Use binarization to fill the holes in the initial mask. Binarization fills the holes by traversing the image. If a white area is encountered, it is skipped; if an area containing a black part is encountered, the dilation operation is performed again to fill the hole.

[0027] In step 3, the Hough transform method for detecting straight lines is used to obtain the angles of both sides and the coordinates of the vertices of the drawn sector mask, thereby obtaining information such as the sector radius to obtain a standard sector mask.

[0028] In step 3, the Hough transform method for detecting straight lines obtains the geometric information of the fan-shaped ROI, including the vertices and the angles of the two sides, by emitting a number of rays from the intersection to the initial mask area, counting the number of pixels on each ray and taking the average value, which is the radius of the fan; through the fan center coordinates, the angles of the two sides, and the fan radius obtained from the initial mask, a standard fan-shaped image can be drawn.

[0029] In the step 3, in some cardiac ultrasound videos, the ROI region is not a standard sector region, but a partial sector with truncated vertex. At this time, the vertex coordinates obtained will be outside the original rectangular area, so that a standard sector mask cannot be output. For this reason, the original rectangle needs to be drawn into an enlarged rectangle to ensure that the vertex coordinates fall therein, and the standard sector is drawn after the sector radius is obtained in the same way, and finally the original rectangle is clipped to the enlarged rectangle and the mask is retained, so that the final mask image is obtained.

[0030] The step 3 comprises the following steps:

[0031] S3.1: Using the Hough transform straight line detection algorithm to detect straight lines on the corrected initial mask, also called the perfected initial mask, retains the intersection position information;

[0032] S3.2: Determine whether the intersection point is within the original rectangular area;

[0033] If so, obtain the angles of both sides of the sector and the length of the sector radius, and draw a standard sector mask;

[0034] If not, perform the following steps:

[0035] S3.3: save the original rectangular area;

[0036] S3.4: Get the angles of both sides of the sector and the length of the sector radius, and draw a larger rectangular mask;

[0037] S3.5: Draw a standard sector;

[0038] S3.6: Use the original rectangular area rectangular mask in S3.3 to extract the fan-shaped mask.

[0039] Through the above steps, the standard mask is obtained.

[0040] The present invention provides a method for quickly and accurately extracting ROI from ultrasound video frames based on spatiotemporal complementary information, and the technical effects are as follows:

[0041] 1) Aiming at the problem of extracting ROI area in cardiac ultrasound, the present invention proposes a method that does not require deep learning, but analyzes the ROI area of ​​each frame in the video through spatiotemporal complementarity and extracts the initial mask. The method mainly includes three parts: obtaining the initial mask, improving the initial mask, and drawing the standard mask. The core of the initial mask acquisition part is to use the maximum connected area of ​​the mask obtained by subtracting the frames and accumulating the mean in the cardiac ultrasound video to obtain the initial mask; the standard mask drawing part uses the Hough transform method to detect the straight line to obtain the angles of both sides and the vertex coordinates of the fan-shaped mask, so as to obtain the fan radius and other information to obtain the standard fan-shaped mask. This method uses traditional image processing methods to obtain a regular mask image, which is different from the commonly used machine learning method, omitting the steps of preparing training sets and training data sets, and there is no overfitting problem.

[0042] 2) The method proposed in the present invention uses traditional image processing methods, combined with the spatiotemporal complementary information of ultrasound video to obtain the initial mask and then draw the standard mask. Therefore, the method proposed in the present invention can be used to obtain the standard mask for ultrasound videos of different angles, while the commonly used machine learning methods need to train and learn different types of cardiac ultrasound videos. By testing 145 two-chamber ultrasound videos and 148 four-chamber ultrasound videos provided by a central hospital of a certain city, the success rates were 96.55% and 98.64%, respectively, and the time taken was 143.15s and 145.32s, respectively. The average time for obtaining the ROI standard mask for each video was 0.97s. Therefore, compared with the machine learning method, the method proposed in the present invention is more applicable, takes less time, and is more efficient.

[0043] 3) The method proposed in the present invention is to utilize traditional image processing methods, combined with the characteristics of ultrasonic cardiology videos, to extract the standard mask of ROI from the two directions of time and space. From the perspective of time, by subtracting the ultrasonic cardiology images after video framing at intervals of several frames, and combining all the sequence images, the information of ROI is fully extracted, the interference information is removed, and the initial mask is extracted; from the perspective of space, for the information of each area in the ultrasonic cardiology video (including interference information and ROI information), the characteristics of each type of information are fully analyzed: the motion area of ​​the interference information is fixed without displacement and the state is constant, and the motion area of ​​the ROI information is within a fixed range but the state is variable. In combination with the characteristics of ultrasonic cardiology videos in time and space, the present invention proposes a fast and accurate extraction algorithm for ROI of ultrasonic cardiology videos based on time-space complementary information, which fully extracts the information of ROI in the ultrasonic cardiology video to obtain the initial mask, and then obtains the standard mask of ROI. This method is applicable to the extraction of standard ROI masks for all ultrasound cardiology videos. It can directly extract the standard mask without being affected by the data type. After obtaining the standard mask, the interference information such as equipment parameters, resolution, afterglow, thermal index, mechanical precision, etc. can be removed through operations to retain the ROI. This saves a lot of manpower and time costs for the preparation of data sets commonly used for deep learning today.

[0044] 4) For the traditional segmentation of ROI area based on Hough transform detection and threshold segmentation method based on maximum connected area, it is difficult to segment the ROI of ultrasound video, mainly because the ROI area of ​​ultrasound video is irregular in video display, the main part is often missing and the edge is relatively blurred. In order to extract the standard mask of ROI area, the present invention fully combines the spatiotemporal characteristics of ultrasound video, maximizes the information in the initial mask of ROI extraction, and uses Hough transform detection to obtain the geometric information of drawing the standard mask after correcting the initial mask. Combined with the characteristics that the initial mask is fan-shaped and the two sides are clear and easy to detect, the angle and position information of the two sides of the fan-shaped ROI are obtained by Hough transform detection, so as to obtain the geometric information such as the coordinates and radius of the ROI fan-shaped vertex, and then the standard mask of ROI is obtained by drawing after the geometric information is obtained. Due to the particularity of ultrasound video, the present invention does not directly use the widely used segmentation method to extract ROI, but fully extracts the information of ROI to obtain the initial mask based on the idea of ​​spatiotemporal complementarity, and then draws the ROI standard mask after extracting the geometric information from the initial mask. This method combines the idea of ​​spatiotemporal complementarity with traditional image processing technology to achieve fast and accurate extraction of ROI areas, bringing new ideas and methods to the field of medical image analysis.

[0045] 5) In order to make the method proposed in the present invention more adaptable, the present invention also handles the special cases existing in the data accordingly. There is a situation where the heart rate curve (displayed in green color) covers the top of the ROI and affects the initial mask extraction. The present invention proposes a method for removing the green channel based on pixel level. By setting a threshold value for the green channel, the pixels whose final display color is green are set to black. This method can not only effectively remove the heart rate curve, but also retain the information of the gray area in the ROI area to the maximum extent, so that the initial mask obtained is not affected by the heart rate curve line and the ROI information is more completely retained. For the situation where the standard sector ROI part appears in the display area, the present invention solves this problem by drawing a rectangular mask of a larger area and then drawing a standard sector and then intercepting it with the original area rectangular mask. Through the processing of special cases, the standard mask for extracting ROI from ultrasound cardiography video has stronger universality, and after considering the special cases, the obtained standard mask result is more complete and accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0047] FIG1( a ) is a schematic diagram of cardiac ultrasound image information (ultrasound information area);

[0048] FIG1( b ) is a schematic diagram of cardiac ultrasound image information (interference information).

[0049] Figure 2(a) is a schematic diagram of a cardiac ultrasound public dataset (EchoNet-LVH public dataset);

[0050] FIG2( b ) is a schematic diagram of a cardiac ultrasound public dataset (CAMUS public dataset);

[0051] Figure 2(c) is a schematic diagram of a cardiac ultrasound public dataset (HEMC-QU public dataset);

[0052] Figure 2(d) is a schematic diagram of the cardiac ultrasound public dataset (TED public dataset).

[0053] FIG3( a ) is a schematic diagram of an ultrasound information region with a missing portion (partially missing);

[0054] FIG3( b ) is a schematic diagram of an ultrasound information region with missing parts (missing vertices).

[0055] Figure 4(a) shows the threshold segmentation result of a single frame with a large number of missing parts of an ultrasound cardiac image;

[0056] Figure 4(b) shows the single frame result after the heart rate of the ultrasound cardiogram is segmented simultaneously;

[0057] Figure 4(c) shows the segmentation result of the ultrasound cardiac image after combining multiple frames;

[0058] FIG4( d ) is a schematic diagram of an ultrasound cardiac image.

[0059] Figure 5 Flowchart for extracting ROI using standard mask through AND operation.

[0060] Figure 6(a) is a schematic diagram of the initial mask with aliasing phenomenon;

[0061] Figure 6(b) is the Hough transform line detection result of Figure 6(a).

[0062] FIG7( a ) is a schematic diagram of an initial mask with severe missing areas in an ultrasound image;

[0063] Figure 7(b) is the Hough transform line detection result of Figure 7(a).

[0064] Figure 8 This is a flow chart of a method for quickly and accurately extracting ROI regions from ultrasound video frames based on spatiotemporal complementary information.

[0065] Figure 9(a) shows the initial mask result (retained information after subtraction of a single frame);

[0066] Figure 9(b) shows the initial mask result (initial mask based on spatiotemporal complementation);

[0067] Figure 9(c) shows the initial mask result (the initial mask obtained by the maximum connected component algorithm);

[0068] Figure 10(a) is an example of an experiment to remove the green channel (the heart rate curve is overlaid on the image area);

[0069] Figure 10(b) is an example of the green channel removal experiment (mask before removing the green channel);

[0070] Figure 10(c) is an example of a green channel removal experiment (the mask after removing the green channel from the entire image);

[0071] FIG10( d ) is an example of an experiment to remove the green channel (the mask after setting a threshold to remove the green channel).

[0072] Fig.11 is the initial mask with aliasing.

[0073] Fig.12 To complete the initial mask flow chart.

[0074] Fig.13 To obtain the standard mask flow chart.

[0075] Figure 14(a) is a schematic diagram of Hough transform detection of straight lines (the intersection point is within the original rectangular area);

[0076] FIG14( b ) is a schematic diagram of a straight line detected by Hough transform (the intersection point is outside the original rectangular area).

[0077] Figure 15(a) is the final standard mask image (the intersection is within the original rectangular area);

[0078] Figure 15(b) is the final standard mask image (the intersection point is outside the original rectangular area).

[0079] FIG16( a ) is a result of performing AND operation on a two-chamber cardiac ultrasound image and a standard mask image;

[0080] FIG16( b ) is the result of performing AND operation on the four-chamber cardiac ultrasound image and the standard mask image;

[0081] FIG16( c ) shows the result of performing an AND operation on the special ultrasound image and the standard mask image. DETAILED DESCRIPTION

[0082] Principle analysis:

[0083] The core of obtaining ROI in ultrasound video is to determine the geometric parameters of the fan-shaped area containing ultrasound image information, including the vertex position coordinates, as shown in the green box in Figure 1(a); the fan-shaped area angle, as shown in the orange straight line in Figure 1(a); and the fan-shaped radius length, as shown in the pink dotted line in Figure 1(a). After obtaining these geometric information, a mask image of the same size as the ultrasound video frame and containing a binary fan-shaped area can be constructed. This is called a standard mask in the present invention. Using this standard mask, the ROI area in the ultrasound video frame can be extracted through AND operations, as shown in Figure 1(a). Figure 5 shown.

[0084] The method for fast and accurate extraction of ROI from ultrasound video frames based on spatiotemporal complementary information specifically includes: obtaining an initial mask based on spatiotemporal complementary information, correcting the initial mask, estimating the vertex of the fan-shaped ROI, estimating the angle and radius of the fan-shaped ROI, and finally obtaining a regular fan-shaped mask. The overall process is as follows Figure 8 The specific contents are as follows:

[0085] (I): Extraction of initial mask:

[0086] In order to obtain the standard mask, the extraction of the initial mask is the basis. As shown in Figure 1(a) and Figure 1(b), in the cardiac ultrasound video, the interference information in the blue area is some parameter information displayed in the fixed area of ​​the video, while the ultrasound information area is constantly moving in the orange fan frame. Through this feature, the present invention proposes an ultrasound area extraction algorithm based on spatiotemporal complementary information. By utilizing the characteristics of overlapping interference information between single frames and continuous movement of ultrasound information areas, the read ultrasound video is divided into frames and subtracted after several frames, such as subtracting the 1st frame from the 5th frame, subtracting the 2nd frame from the 6th frame, subtracting the 3rd frame from the 7th frame... At this time, since the interference information is static, after subtracting after several frames, the offset part is the interference information, which is the background; the ultrasound information area with differences is successfully retained as the foreground. As shown in Figure 9(a), the foreground area retained after subtracting two frames is limited. In order to retain the complete area of ​​ultrasound information, combined with the fact that several frames of ultrasound cardiology video contain information, all video frames are traversed and the results of subtracting several frames are retained and accumulated to calculate the average. At this time, the retained foreground area information is the most complete, as shown in Figure 9(b).

[0087] Since the target is an ultrasound cardiology video, which contains interference information of the heart rate image, after obtaining the complete foreground area, the heart rate image will be mistakenly saved as the foreground because the heart rate changes with the video time. In order to solve this problem, the present invention uses the maximum connected region algorithm to retain the ultrasound information area, combining the characteristics that the ultrasound information area occupies a large area and is concentrated, and the heart rate part occupies a small area and is dispersed. FIG9(c) is the retained ultrasound information area, which is the initial mask.

[0088] In addition, there is a special case where the heart rate curve is covered on the ultrasound image as shown in Figure 10 (a). In this case, the mask obtained is shown in Figure 10 (b). The heart rate curve is adhered to the ROI area. At this time, the maximum connected area algorithm cannot remove the curve part. By observing the ultrasound image, it is found that the heart rate curve is green. It is assumed that before performing spatiotemporal complementary extraction of the ultrasound area, the green channel of the ultrasound image is removed so that the ultrasound image participating in the complementary algorithm does not contain the heart rate curve. Then the final mask naturally does not have the heart rate curve. At this time, the problem of heart rate curve interference can be well solved. Through experiments, the mask result after removing the green channel of the ultrasound image and then performing the complementary algorithm is shown in Figure 10 (c). At this time, the experimental object for removing the green channel is the entire ultrasound image. All green channels in the image will be removed without distinction. At this time, a small part of the red box in Figure 10 (c) will be missing. The reason is that after the green channel is removed as a whole, some gray areas do not meet the threshold conditions during binarization and are set to black, and the green channel is set to 0. In order to reduce the pressure of subsequent hole filling operations, the present invention improves the above-mentioned method of removing the heart rate curve, sets a threshold for the green channel in the pixel points, and sets the pixel points that are displayed as green as a whole to black. Comparing the red frame areas in Figure 10 (c) and Figure 10 (d), the improved method can retain the gray area in the ultrasound cardiogram to the greatest extent, making the final initial mask more complete. At the same time, in order to prevent the ROI area in the ultrasound cardiogram from being disturbed by the heart rate curve during the subsequent extraction, the same operation is performed on the original ultrasound cardiogram video through this method to remove the heart rate curve and retain the result.

[0089] (II): Correction of the initial mask:

[0090] The initial mask obtained is not perfect. The quality of cardiac ultrasound videos varies due to the proficiency of the doctors and the different testing equipment. Therefore, the experimental results of a single frame cannot produce a good initial mask. It is necessary to combine the entire video, traverse the video frames, subtract the results after several frames, and then accumulate and average the results to obtain the maximum connected area, which is the initial mask of the ROI area in the video. As mentioned above, different testers and different testing equipment will result in uneven ultrasound image quality. There are many cases where the edges of the initial mask have jagged edges. At the same time, due to the special structure of the heart, there is a probability that there will be holes in the initial mask, such as Fig.11 As shown. Fig.11 The initial mask shown in FIG. 1 has jagged edges, which is very unfavorable for subsequent straight lines. At the same time, the holes therein have a very serious impact on the acquisition of the standard fan radius.

[0091] In order to obtain a standard sector area later, the initial mask needs to be corrected. The corrected initial mask, also called the flowchart of the improved initial mask, is as follows: Fig.12 shown.

[0092] First, the dilation and erosion algorithms are applied to the image. This not only effectively removes the jagged edges and makes the surface smooth, but also effectively fills the small holes in the initial mask. The large holes in the initial mask are filled using the binary filling algorithm. Binarization fills the holes by traversing the image. If a white area is encountered, it is skipped; if a black area is encountered, the dilation algorithm is used to fill the hole. After these two traditional image processing algorithms, the initial mask can be more regular as a whole, and the edges are smoother, which is convenient for subsequent further processing.

[0093] After the closing operation of the expansion and erosion algorithms, the edge of the initial mask can become smooth and the small holes in the mask can be effectively filled. However, the size of the operator and the number of convolutions have an impact on the improved initial mask. The following is a comparison of the time used for the same initial mask under different operator sizes and the number of convolutions per time and the final initial mask results, as shown in Table 1.

[0094] Table 1 Comparison of initial mask results under different operator sizes and the same number of rolls

[0095]

[0096] Through the comparison in Table 1 above, we can find that the time used is basically the same. The larger the operator, the less time it takes. Among them, when the expansion operator is larger than the erosion operator, the result is better. Based on the above analysis, the expansion operator is set to 5*5 and the erosion operator is set to 3*3, which can shorten the operation time and get good results. It is worth noting that the effect achieved under the condition of 1 convolution is good enough, and there is no need to repeat the convolution, which increases the operation time.

[0097] (III): Get the standard sector mask:

[0098] In cardiac ultrasound video, the desired ROI area is a standard sector, but due to reasons such as video angle or video length, the initial mask obtained is not standard, so it is necessary to use this initial mask to further draw a standard mask. Fig.13 This is the flowchart for obtaining the standard mask image.

[0099] Since the ROI area in the cardiac ultrasound video is a standard fan-shaped area, although there are cases where the vertex is cut off and the fan-shaped area is incomplete, the final standard mask needs to draw a standard fan-shaped area. For drawing a standard fan-shaped area, the required information includes vertex coordinates, angles on both sides, and radius length. By analyzing the ultrasound cardiology video, from a single frame to the whole, the ROI area is finally gathered in the initial mask, and the information for drawing the standard fan-shaped area is naturally obtained from it.

[0100] For drawing a standard sector, the vertex coordinates are the key. Combined with the analysis of the initial mask obtained, it is found that although the initial mask is not a standard sector, the two side lines are relatively clear. Therefore, the present invention uses the Hough transform straight line detection algorithm to detect the straight line on the improved initial mask, so that not only the angle of the two sides can be recorded, but also the intersection of the two sides is the vertex position of the sector. Figure 14 (a) and Figure 14 (b) show schematic diagrams of Hough transform straight line detection.

[0101] The basic principle of Hough transform is to use the duality of points and lines, that is, a point in the original image coordinate system corresponds to a straight line in the parametric coordinate system, and a straight line in the parametric coordinate system corresponds to a point in the original coordinate system. All points that appear as straight lines in the original coordinate system have the same slope and intercept, so they correspond to the same point in the parametric coordinate system. In this way, after projecting each point in the original coordinate system to the parametric coordinate system, see if there are any clustered points in the parametric coordinate system. Such clustered points correspond to the straight line in the original coordinate system.

[0102] In terms of straight line detection, Hough transform detection of straight lines has good practicality and stability. Therefore, the present invention also uses Hough transform detection of straight lines to detect the straight lines on both sides and record information such as vertices and angles of both sides to provide data for subsequent drawing of standard masks. The following is an algorithm for obtaining geometric information of fan-shaped ROI by Hough transform detection of straight lines.

[0103]

[0104]

[0105] After obtaining the intersection (vertex) and the angles on both sides, it is necessary to obtain the fan radius. The present invention emits several rays from the intersection to the initial mask area, counts the number of pixels on each ray and takes the average value, which is the radius of the fan. A standard fan image can be drawn by the fan center coordinates, the angles on both sides, and the fan radius obtained from the initial mask. In addition, in some cardiac ultrasound videos, it will be found that the ROI area is not a standard fan area, but a partial fan with the vertex cut off, as shown in Figure 3 (b). At this time, the vertex coordinates obtained will be outside the original rectangular area, and thus a standard fan mask cannot be output. For this reason, the original rectangle needs to be drawn into a larger rectangle to ensure that the vertex coordinates fall within it, and the standard fan is drawn after obtaining the fan radius in the same way. Finally, the original rectangle is clipped to the enlarged rectangle and the mask is retained, and the final mask image is obtained. The final mask image is shown in Figures 15 (a) and 15 (b).

[0106] As shown in Fig. 16(a), Fig. 16(b) and Fig. 16(c), the results of extracting the ROI standard mask from the ultrasonic two-chamber and four-chamber cardiac images and the ultrasonic cardiac images whose vertices and corners are beyond the display area by the algorithm of the present invention are shown respectively. After obtaining the ROI standard mask, by performing AND operation on the ultrasonic cardiac video or performing AND operation on the ultrasonic cardiac image after frame division, the ultrasonic cardiac image containing only the ROI area as shown in (1-c1) to (1-c3) in Fig. 16(a), (2-c1) to (2-c3) in Fig. 16(b), and (3-c1) to (3-c3) in Fig. 16(c) can be obtained. The ultrasonic cardiac video after the AND operation has removed the rest of the interference information, and can be well used for in-depth research of ultrasonic medical treatment.

[0107] The present invention provides a method for quickly and accurately extracting ROI from ultrasound video frames based on spatiotemporal complementary information, which specifically includes the following features:

[0108] 1) The method of the present invention is to segment the ROI area based on the analysis of the cardiac ultrasound video based on the time-space complementary information, with good segmentation effect and high algorithm stability. As the heart beats, the pixel values ​​of each point within the fan-shaped ultrasound image range change frequently; in addition, since the heart cavity is hollow and the heart is a fist-shaped three-dimensional target, each frame of the image acquired by the probe will have different degrees of black information-free areas, especially in the peripheral area of ​​the fan-shaped area of ​​the ultrasound imaging. If the ROI area is extracted based on a single frame of image, it will often fail due to the above-mentioned influence. Therefore, the present invention proposes a premise assumption that any point in the video in an ultrasound video, as long as there is a change in pixel value at two time points once, the point can be considered as a pixel point in the ROI. In order to improve the robustness of the algorithm and reduce the sawtooth effect caused by the edge changes caused by the video compression algorithm, the present invention adopts a method of inter-frame subtraction, then binarization, and then a union operation separated by several frames to improve the complete capture degree of the ROI pixel points. Experimental results show that the ROI candidate area obtained by this time-space complementary method has a significant effect on improving the estimation accuracy of the standard mask parameters.

[0109] 2) The ROI candidate area can be extracted to the maximum extent through the time-space complementation method. Due to the scanning angle and depth of the probe used by the doctor, in some ultrasonic cardiology videos, the heart rate curve is covered on the top of the ROI area due to the limitation of the video area. Since the pixel value of the heart rate curve area changes with time, the ROI candidate area obtained by the time-space complementary information will mistakenly capture and retain the heart rate curve as the ROI at the same time, resulting in the ROI obtained not being a complete ultrasonic image area. In response to the above-mentioned problems, combined with the characteristic that the heart rate curve is green, the present invention sets a threshold value for the channel of the pixel before extracting the ROI area using the time-space complementary information algorithm. If the pixel is displayed as green, it is set to black. After this operation, the situation where the heart rate curve covers the top of the ROI area can be ignored for the final extracted ROI candidate area result. At this time, the ROI candidate area obtained is the initial mask.

[0110] 3) The initial mask obtained is not perfect, the initial mask only retains the information of ROI to the greatest extent, but the edge information of the partial mask is in motion due to ROI, after the initial mask is extracted by time-space complementary information, the edge grayscale area at this time combines the information of all frames of the ultrasound cardiography video, the connection between the edge information is weakened, and the sawtooth effect is revealed after binarization. And there are holes in the final ultrasound image determined by the specific function of the heart, these situations have a serious impact on the estimation of the subsequent detection straight line and the radius of the fan-shaped ROI, the sawtooth phenomenon often causes the two side lines of the standard fan-shaped mask to be unable to be effectively identified, and the existence of the hole causes the radius result of the standard fan-shaped mask obtained to be smaller. In order to avoid the influence of these controllable situations on the result, the present invention further obtains a more regular mask by expansion, corrosion algorithm, and the effective coordination of operations such as binarization filling holes, which is convenient for obtaining regular ROI fan-shaped masks in the future. By using the Hough algorithm to detect straight lines on the perfected mask, we can get the intersection of the two sides of the fan-shaped area (i.e., the center of the fan-shaped circle). We can then use the obtained intersection to emit several rays to the corrected initial mask, and count the total number of pixels in the initial mask area that the rays pass through and average them. The value is the radius of the fan-shaped area. After obtaining this geometric information, we can draw a regular fan-shaped mask.

[0111] 4) Due to the scanning angle or depth of the probe used by the doctor, there are two situations in the display area of ​​the standard sector ROI in the ultrasound cardiology video: the standard sector ROI appears completely in the display area of ​​the image and partially in the display area, as shown in Figure 3 (a) and Figure 3 (b). The same situation occurs in the initial mask obtained. For the incomplete display of the initial mask, the vertices and corners of the initial mask exceed the display area, resulting in incomplete display. For this situation where the initial mask is incompletely displayed, special processing is required to ensure that the final mask image is consistent with the actual situation. The specific steps are as follows: First, save the original display rectangular area mask, and draw a larger rectangular mask than the original display rectangle as the basis for subsequent drawing and cropping; then, draw a standard sector mask in the larger rectangular mask according to the acquired geometric parameters to ensure the integrity and accuracy of the mask; finally, intercept the original display rectangular mask with the larger rectangular mask to obtain the final mask image. By processing special cases, the problem of incomplete display of the initial mask can be effectively solved to ensure the quality and accuracy of the mask image.

Claims

1. A fast and accurate method for extracting ROI from ultrasound video frames based on spatiotemporal complementary information, characterized in that The following steps are involved: Step 1: Input the echocardiographic video and extract the initial mask by combining the spatiotemporal complementary information; Step 2: Correct the initial mask; Step 3: Draw the corrected initial mask using the Hough transform method to detect straight lines to obtain a standard fan-shaped mask.

2. According to claim 1, the method for quickly and accurately extracting ROI from ultrasound video frames based on spatiotemporal complementary information is characterized by: The step 1 comprises the following steps: S1.1: Taking advantage of the overlapping of interference information between single video frames and the continuous movement of ultrasound information areas, the read ultrasound cardiography video is divided into frames and subtracted after several frames. Since the interference information is static, after subtracting after several frames, the offset part is the interference information, which is the background; the ultrasound information area with differences is successfully retained and is the foreground area; S1.2: Traverse all video frames and perform subtraction at intervals of several frames, retain and accumulate the average value, at which time the foreground area information retained is the most complete; S1.3: After obtaining the complete foreground area, the ultrasound information area is retained using the maximum connected region algorithm, and the retained ultrasound information area is the initial mask.

3. The method for quickly and accurately extracting ROI from ultrasound video frames based on spatiotemporal complementary information according to claim 2, characterized in that: When the heart rate curve is overlaid on the ultrasonic cardiogram, the threshold value of the green channel in the pixel points of the ultrasonic cardiogram is set so that the pixel points that are displayed as green as a whole are set to black; and the gray area in the ultrasonic cardiogram is retained to the greatest extent.

4. The method for quickly and accurately extracting ROI from ultrasound video frames based on spatiotemporal complementary information according to claim 1, characterized in that: The step 2 comprises the following steps: S2.1: First, dilation and erosion operations are performed on the initial mask; S2.2: Use binarization to fill the holes in the initial mask. Binarization fills the holes by traversing the image. If a white area is encountered, it is skipped; if an area containing a black part is encountered, the dilation operation is performed again to fill the hole.

5. The method for quickly and accurately extracting ROI from ultrasound video frames based on spatiotemporal complementary information according to claim 1, characterized in that: In step 3, the Hough transform method for detecting straight lines is used to obtain the angles of both sides and the coordinates of the vertices of the drawn sector mask, thereby obtaining information such as the sector radius to obtain a standard sector mask.

6. The method for quickly and accurately extracting ROI from ultrasound video frames based on spatiotemporal complementary information according to claim 5, characterized in that: In step 3, the Hough transform method for detecting straight lines obtains the geometric information of the fan-shaped ROI, including the vertices and the angles of the two sides, by emitting a number of rays from the intersection to the initial mask area, counting the number of pixels on each ray and taking the average value, which is the radius of the fan; through the fan center coordinates, the angles of the two sides, and the fan radius obtained from the initial mask, a standard fan-shaped image can be drawn.

7. The method for quickly and accurately extracting ROI from ultrasound video frames based on spatiotemporal complementary information according to claim 6, characterized in that: In some cardiac ultrasound videos, the ROI area is not a standard sector-shaped area, but a partial sector with the top corners truncated. In this case, the vertex coordinates obtained will be outside the original rectangular area, and it is impossible to output a standard sector-shaped mask. To this end, the original rectangle is drawn as an enlarged rectangle to ensure that the vertex coordinates fall within it. After obtaining the sector radius, a standard sector is drawn. Finally, the original rectangle is cropped against the enlarged rectangle and the mask is retained to obtain the final mask image.

8. The method for quickly and accurately extracting ROI from ultrasound video frames based on spatiotemporal complementary information according to claim 6, characterized in that: The step 3 comprises the following steps: S3.1: Use the Hough transform straight line detection algorithm to detect straight lines on the corrected initial mask and retain the intersection position information; S3.2: Determine whether the intersection point is within the original rectangular area; If so, obtain the angles of both sides of the sector and the length of the sector radius, and draw a standard sector mask; If not, perform the following steps: S3.3: save the original rectangular area; S3.4: Get the angles of both sides of the sector and the length of the sector radius, and draw a larger rectangular mask; S3.5: Draw a standard sector; S3.6: Use the original rectangular area rectangular mask in S3.3 to intercept the fan-shaped mask; Through the above steps, the standard mask is obtained.

Images