Fetal ultrasound image label removing method and device based on image channel splitting

Abstract

The invention relates to a fetal ultrasound image label removing method and device based on image channel splitting, and the method comprises the steps: obtaining a plurality of fetal ultrasound images, constructing a label feature extraction data set according to the plurality of fetal ultrasound images, and expanding the data set according to a channel splitting technology, and training a pre-constructed mark feature extraction model and a mark feature supplementary drawing model by using the data set expanded by the channel splitting technology, so as to perform multi-dimensional feature extraction and supplementary drawing by using the channel splitting technology based on two deep learning models of mark feature extraction and mark feature supplementary drawing, thereby realizing end-to-end mark removal operation. Therefore, the problems that in the prior art, it is difficult to unify the resolution, the mark style, the mark color and the image quality of the ultrasonic images between different devices, and marks in the fetal ultrasonic images cannot be effectively recognized and removed are solved.

Description

Technical Field

[0001] The present application relates to the technical field of medical image analysis and processing, and in particular to a method and device for de-labeling fetal ultrasound images based on image channel splitting. Background Art

[0002] The traditional de-marking method is to identify the color of the marker in the ultrasound image, that is, to extract the color and sample the area around the marker to achieve the effect of inpainting. This traditional inpainting method can achieve the effect of de-marking, but the processing of details is often rough, which will cause interference in subsequent image analysis. In particular, if you continue to use these processed ultrasound images as training sets for other deep learning tasks, if you want to obtain more reliable training results, a de-marking method that is more in line with the characteristic distribution of medical images is very necessary.

[0003] Deep neural networks are the most popular image processing technology today. They have good processing effects on images with unclear features and complex structures. In addition, traditional deep learning methods often perform better on natural image processing tasks with large amounts of data and manual labeling. When faced with medical images, especially fetal ultrasound images with high noise, complex image structures, and unclear features, these traditional deep learning methods often fail to achieve the desired results.

[0004] In addition, the collection and labeling of medical images also requires a lot of manpower and medical knowledge. When the amount of data is limited, it is very challenging to let the deep neural network learn key knowledge more accurately and achieve end-to-end label removal. At the same time, the diversity of ultrasound equipment has also put forward new requirements and challenges for ultrasound image label removal. Different labeling forms require the model to have high versatility. Existing deep learning methods often have good labeling effects on the labeling forms visible in the training set, but lack corresponding countermeasures for labeling types other than those covered by the training set.

[0005] In summary, it is difficult for existing technologies to unify the resolution, marking style, marking color and image quality of ultrasound images between different devices, and it is impossible to effectively identify and remove marks in fetal ultrasound images, which needs to be solved urgently. Summary of the invention

[0006] The present application provides a fetal ultrasound image de-marking method and device based on image channel splitting to solve the problems that the prior art is difficult to unify the resolution, marking style, marking color and image quality of ultrasound images between different devices, and is unable to effectively identify and remove marks in fetal ultrasound images.

[0007] The first aspect of the present application provides a fetal ultrasound image de-labeling method based on image channel splitting, comprising the following steps: acquiring multiple fetal ultrasound images, and constructing a label feature extraction data set based on the multiple fetal ultrasound images, and expanding the label feature extraction data set using a preset image channel splitting strategy, and training a pre-constructed label feature extraction model based on the expanded label feature extraction data set to obtain a prediction model corresponding to the label feature extraction model; constructing a label removal data set based on the multiple fetal ultrasound images, and expanding the label removal data set using the image channel splitting strategy, and training a pre-constructed label removal model based on the expanded label removal data set to obtain a redrawing model corresponding to the label removal model; and performing an end-to-end label removal operation on the target fetal ultrasound image with labels based on the prediction model and the redrawing model.

[0008] Optionally, in one embodiment of the present application, the method of acquiring multiple fetal ultrasound images, constructing a marker feature extraction data set based on the multiple fetal ultrasound images, and expanding the marker feature extraction data set using a preset image channel splitting strategy, and training a pre-constructed marker feature extraction model based on the expanded marker feature extraction data set to obtain a prediction model corresponding to the marker feature extraction model, includes: acquiring multiple marked fetal ultrasound images from the multiple fetal ultrasound images, and constructing the marker feature extraction data set through the multiple marked fetal ultrasound images; extracting border text information corresponding to each marked fetal ultrasound image in the marker feature extraction data set, and filling in the missing parts of each marked fetal ultrasound image after extracting the border text information based on a preset image repair strategy. , to obtain a marked filled image; adjust the image size of each marked filled image to obtain a marked filled image of a target size; randomly add multiple types of marks of the same color to the marked filled image, and convert the marked filled image to the HSV space to extract the mark mask corresponding to the marked filled image; perform channel splitting on the marked filled image, and copy the mark mask to obtain a standard extended marked feature extraction dataset corresponding to the marked feature extraction dataset; based on a preset encoder, decoder and U-Net framework, construct the marked feature extraction model, and determine the first objective function corresponding to the marked feature extraction model; train the marked feature extraction model through the standard extended marked feature extraction dataset and the first objective function to obtain the prediction model.

[0009] Optionally, in one embodiment of the present application, constructing a mark removal dataset according to the multiple fetal ultrasound images, expanding the mark removal dataset using the image channel splitting strategy, and training a pre-constructed mark removal model based on the expanded mark removal dataset to obtain a redrawing model corresponding to the mark removal model, includes: acquiring multiple marked fetal ultrasound images from the multiple fetal ultrasound images, and acquiring multiple unmarked fetal ultrasound images corresponding to the multiple marked fetal ultrasound images from the multiple fetal ultrasound images, so as to construct multiple fetal ultrasound image sample pairs according to the multiple marked fetal ultrasound images and the multiple unmarked fetal ultrasound images; establishing the mark removal dataset based on the multiple fetal ultrasound image sample pairs, and performing preprocessing and channel splitting operations on the marked fetal ultrasound images and the unmarked fetal ultrasound images in each fetal ultrasound image sample pair in the mark removal dataset, so as to obtain a standard expanded mark removal dataset corresponding to the mark removal dataset; determining a second objective function corresponding to the mark removal model, so as to train the mark removal model according to the standard expanded mark removal dataset and the second objective function to obtain the redrawing model.

[0010] Optionally, in one embodiment of the present application, the end-to-end label removal operation is performed on the target fetal ultrasound image with labels based on the prediction model and the inpainting model, including: combining the prediction model and the inpainting model to construct a final label removal network model; using the final label removal network model to perform a label removal operation on the target fetal ultrasound image with labels to obtain the target fetal ultrasound image after the labels are removed.

[0011] The second aspect of the present application provides a fetal ultrasound image de-marking device based on image channel splitting, including: a marker feature extraction module, used to acquire a plurality of fetal ultrasound images, and construct a marker feature extraction data set based on the plurality of fetal ultrasound images, and expand the marker feature extraction data set using a preset image channel splitting strategy, and train a pre-constructed marker feature extraction model based on the expanded marker feature extraction data set to obtain a prediction model corresponding to the marker feature extraction model; a marker feature in-painting module, used to construct a marker removal data set based on the plurality of fetal ultrasound images, and expand the marker removal data set using the image channel splitting strategy, and train a pre-constructed marker removal model based on the expanded marker removal data set to obtain a in-painting model corresponding to the marker removal model; an integration processing module, used to perform an end-to-end marker removal operation on the target fetal ultrasound image with markers based on the prediction model and the in-painting model.

[0012] Optionally, in one embodiment of the present application, the marker feature extraction module includes: a first acquisition unit, used to acquire multiple marked fetal ultrasound images from the multiple fetal ultrasound images, and construct the marker feature extraction data set through the multiple marked fetal ultrasound images; an extraction unit, used to extract the border text information corresponding to each marked fetal ultrasound image in the marker feature extraction data set, and based on a preset image restoration strategy, fill the missing parts of each marked fetal ultrasound image after extracting the border text information to obtain a marked filled image; an adjustment unit, used to adjust the image size of each marked filled image to obtain a marked filled image of a target size; a conversion unit, used to convert the marked filled image into a marked filled image; A plurality of marks of the same color are randomly added to the marked filled image, and the marked filled image is converted into the HSV space to extract the mark mask corresponding to the marked filled image; a channel splitting unit is used to perform channel splitting on the marked filled image and copy the mark mask to obtain a standard extended mark feature extraction data set corresponding to the mark feature extraction data set; a first determination unit is used to construct the mark feature extraction model based on a preset encoder, decoder and U-Net framework, and determine the first objective function corresponding to the mark feature extraction model; a training unit is used to train the mark feature extraction model through the standard extended mark feature extraction data set and the first objective function to obtain the prediction model.

[0013] Optionally, in one embodiment of the present application, the marker feature infilling module includes: a second acquisition unit, used to acquire multiple marked fetal ultrasound images from the multiple fetal ultrasound images, and acquire multiple unmarked fetal ultrasound images corresponding to the multiple marked fetal ultrasound images from the multiple fetal ultrasound images, so as to construct multiple fetal ultrasound image sample pairs according to the multiple marked fetal ultrasound images and the multiple unmarked fetal ultrasound images; an establishment unit, used to establish the marker removal data set based on the multiple fetal ultrasound image sample pairs, and preprocess and perform channel splitting operations on the marked fetal ultrasound images and the unmarked fetal ultrasound images in each fetal ultrasound image sample pair in the marker removal data set, respectively, so as to obtain a standard extended marker removal data set corresponding to the marker removal data set; a second determination unit, used to determine a second objective function corresponding to the marker removal model, so as to train the marker removal model according to the standard extended marker removal data set and the second objective function, so as to obtain the infilling model.

[0014] Optionally, in one embodiment of the present application, the integrated processing module includes: a combination unit, used to combine the prediction model and the infill model to construct a final label removal network model; a processing unit, used to use the final label removal network model to perform a label removal operation on the target fetal ultrasound image with labels to obtain the target fetal ultrasound image after the labels are removed.

[0015] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the fetal ultrasound image de-labeling method based on image channel splitting as described in the above embodiment.

[0016] The fourth aspect of the present application provides a computer-readable storage medium, which stores a computer program. When the program is executed by a processor, it implements the above-mentioned fetal ultrasound image de-labeling method based on image channel splitting.

[0017] A fifth aspect of the present application provides a computer program product, including a computer program, which is executed to implement the above-mentioned fetal ultrasound image de-labeling method based on image channel splitting.

[0018] Therefore, the embodiments of the present application have the following beneficial effects:

[0019] The embodiments of the present application can obtain multiple fetal ultrasound images, and construct a marker feature extraction data set based on the multiple fetal ultrasound images, and expand the data set based on the channel splitting technology, and use the data set expanded by the channel splitting technology to train the pre-constructed marker feature extraction model and marker feature infilling model, so as to use the channel splitting technology to perform multi-dimensional feature extraction and infilling based on two deep learning models of marker feature extraction and marker feature infilling, thereby realizing end-to-end marker removal operations. The present application realizes the style unification of ultrasound images through channel splitting, thereby improving the versatility and robustness of the marker removal technology, and realizes end-to-end ultrasound image marker extraction and infilling, thereby significantly improving the automation and intelligence level of marker removal. As a result, the problems that the prior art is difficult to unify the resolution, marker style, marker color and image quality of ultrasound images between different devices, and cannot effectively identify and remove markers in fetal ultrasound images are solved.

[0020] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0022] Figure 1 A flowchart of a fetal ultrasound image de-labeling method based on image channel splitting provided according to an embodiment of the present application;

[0023] Figure 2 A schematic diagram of a process for extracting marker features from a fetal ultrasound image provided by an embodiment of the present application;

[0024] Figure 3 A schematic diagram of a process for removing marks from a fetal ultrasound image and performing infill drawing provided in one embodiment of the present application;

[0025] Figure 4 This is an exemplary diagram of a fetal ultrasound image de-labeling device based on image channel splitting according to an embodiment of the present application;

[0026] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0027] Among them, 10-fetal ultrasound image de-marking device based on image channel splitting; 100-mark feature extraction module, 200-mark feature infilling module, 300-integration processing module; 501-memory, 502-processor, 503-communication interface. DETAILED DESCRIPTION

[0028] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0029] The following describes the fetal ultrasound image de-marking method and device based on image channel splitting of the embodiment of the present application with reference to the accompanying drawings. In view of the problems mentioned in the above background technology, the present application provides a fetal ultrasound image de-marking method based on image channel splitting, in which a plurality of fetal ultrasound images are obtained, and a marker feature extraction data set is constructed according to the plurality of fetal ultrasound images, and the data set is expanded according to the channel splitting technology, and the pre-constructed marker feature extraction model and marker feature infilling model are trained using the data set expanded by the channel splitting technology, so as to perform multi-dimensional feature extraction and infilling based on the two deep learning models of marker feature extraction and marker feature infilling, thereby realizing end-to-end marker removal operation. The present application realizes the style unification of ultrasound images through channel splitting, thereby improving the versatility and robustness of the marker removal technology, realizing end-to-end ultrasound image marker extraction and infilling, thereby significantly improving the automation and intelligence level of marker removal. As a result, the problems that the prior art is difficult to unify the resolution, marker style, marker color and image quality of ultrasound images between different devices, and cannot effectively identify and remove markers in fetal ultrasound images are solved.

[0030] Specifically, Figure 1 A flowchart of a fetal ultrasound image de-labeling method based on image channel splitting provided in an embodiment of the present application.

[0031] like Figure 1 As shown, the fetal ultrasound image de-labeling method based on image channel splitting includes the following steps:

[0032] In step S101, a plurality of fetal ultrasound images are acquired, and a marker feature extraction data set is constructed based on the plurality of fetal ultrasound images. The marker feature extraction data set is expanded using a preset image channel splitting strategy, and a pre-constructed marker feature extraction model is trained based on the expanded marker feature extraction data set to obtain a prediction model corresponding to the marker feature extraction model.

[0033] The embodiments of the present application can first acquire multiple fetal ultrasound images to construct a marker feature extraction data set, and perform data preprocessing and image channel splitting on them to expand the marker feature extraction data set; secondly, construct a convolutional neural network model based on the U-Net architecture for marker feature extraction (i.e., a marker feature extraction model) and an objective function corresponding to the network model; thereafter, the embodiments of the present application need to train the marker feature extraction model based on the expanded marker feature extraction data set and the corresponding objective function, and use the trained marker feature extraction model as a prediction model.

[0034] Optionally, in one embodiment of the present application, a plurality of fetal ultrasound images are acquired, and a marker feature extraction data set is constructed based on the plurality of fetal ultrasound images, and the marker feature extraction data set is expanded using a preset image channel splitting strategy, and based on the expanded marker feature extraction data set, a pre-constructed marker feature extraction model is trained to obtain a prediction model corresponding to the marker feature extraction model, including: acquiring a plurality of marked fetal ultrasound images from a plurality of fetal ultrasound images, and constructing a marker feature extraction data set using the plurality of marked fetal ultrasound images; extracting border text information corresponding to each marked fetal ultrasound image in the marker feature extraction data set, and based on a preset image restoration strategy, repairing the missing parts of each marked fetal ultrasound image after extracting the border text information. Filling to obtain a marked filled image; adjusting the image size of each marked filled image to obtain a marked filled image of a target size; randomly adding multiple types of marks of the same color to the marked filled image, and converting the marked filled image to the HSV space to extract the marking mask corresponding to the marked filled image; performing channel splitting on the marked filled image and copying the marking mask to obtain a standard extended marked feature extraction dataset corresponding to the marked feature extraction dataset; based on the preset encoder, decoder and U-Net framework, constructing a marked feature extraction model, and determining the first objective function corresponding to the marked feature extraction model; training the marked feature extraction model through the standard extended marked feature extraction dataset and the first objective function to obtain a prediction model.

[0035] It should be noted that the embodiments of the present application can select a certain proportion of multiple fetal ultrasound images with labels from multiple fetal ultrasound images, and preprocess the fetal ultrasound images with labels. Among them, data preprocessing requires data processing and enhancement of the fetal ultrasound images with labels, and channel splitting of the images, so that the training data can be expanded to three times the number of samples. The specific data preprocessing process is as follows:

[0036] 1. Perform OCR recognition on the text information around the ultrasound image and fill in the missing parts using an image restoration technology based on a fast marching algorithm to obtain a fetal ultrasound image dataset that removes the text information but still contains the marking information (i.e., a fetal ultrasound image dataset corresponding to the filled image with markings);

[0037] 2. Adjust the image size in the data set. If the image size is larger than 960×720 (i.e., the target size), the image is segmented using the sliding window method to obtain several 960×720 image slices; if the image size is smaller than 960×720, the image is expanded so that the size of all images in the training set is unified to 960×720;

[0038] 3. Perform data enhancement on the fetal ultrasound images with labels, and add 100 examples of each of the three label styles: dotted line, letter, and crosshair on the basis of the original labels, with the same color as the original label;

[0039] 4. Convert the marked fetal ultrasound image to HSV space, extract the mark mask by setting the range of the mark color, and use it as the target of the training data;

[0040] 5. Split the channels of the marked fetal ultrasound image and copy the marked mask to achieve a one-to-one correspondence effect, such as Figure 2 As shown, the training data is expanded to three times the number of samples to obtain a preprocessed training set (i.e., a standard expanded labeled feature extraction data set) and a training target;

[0041] Secondly, based on the preset encoder, decoder and U-Net framework, a marker feature extraction model is constructed, and the first objective function corresponding to the marker feature extraction model is determined; the marker feature extraction model is trained using the standard expanded marker feature extraction data set and the first objective function to obtain a prediction model.

[0042] Again, the embodiments of the present application can construct a marker feature extraction model. The overall network structure of the marker feature extraction model continues the classic U-Net framework, including an encoder (downsampling), a decoder (upsampling) and jump connections. In addition, the embodiments of the present application adopt a combination of Dice loss and cross entropy loss to construct the objective function (i.e., the first objective function) of the marker feature extraction model.

[0043] Afterwards, the embodiments of the present application can train the marker feature extraction model through the standard expanded marker feature extraction data set and the first objective function. The training process aims to minimize the value of the objective function. The trained network model is obtained according to the final weight parameters of the discriminator and the generator. The trained network model is used as the final marker feature extraction model, that is, the prediction model.

[0044] It can be understood that the embodiments of the present application realize the automatic extraction of marker features, and also realize the infill technology based on the extracted marker features that conforms to the original features of the ultrasound image, thereby achieving end-to-end realization of the entire process of marker removal, and significantly improving the automation and intelligence of marker removal; in addition, the embodiments of the present application also take into account the problems of different resolutions, marker styles, marker colors and image qualities of ultrasound images between different devices, thereby achieving a unified style of ultrasound images through channel splitting, thereby improving the versatility and robustness of the marker removal technology.

[0045] In step S102, a marker removal dataset is constructed based on multiple fetal ultrasound images, and the marker removal dataset is expanded using an image channel splitting strategy. Based on the expanded marker removal dataset, a pre-constructed marker removal model is trained to obtain a redrawing model corresponding to the marker removal model.

[0046] Furthermore, the embodiments of the present application also need to construct a mark removal dataset through the acquisition of multiple fetal ultrasound images, and perform dataset preprocessing and image channel splitting on it to expand the mark removal dataset; secondly, the embodiments of the present application need to construct a U-Net architecture convolutional neural network model based on partial convolution for mark removal (i.e., a mark removal model), and determine the objective function corresponding to the mark removal model (i.e., the second objective function); thereafter, the embodiments of the present application can train the mark removal model based on the expanded mark removal dataset and the second objective function, and use the trained mark removal model as the infill model.

[0047] Optionally, in one embodiment of the present application, a mark removal dataset is constructed based on multiple fetal ultrasound images, and the mark removal dataset is expanded using an image channel splitting strategy, and a pre-constructed mark removal model is trained based on the expanded mark removal dataset to obtain a redrawing model corresponding to the mark removal model, including: obtaining multiple marked fetal ultrasound images from multiple fetal ultrasound images, and obtaining multiple unmarked fetal ultrasound images corresponding to the multiple marked fetal ultrasound images from the multiple fetal ultrasound images, so as to construct multiple fetal ultrasound image sample pairs based on the multiple marked fetal ultrasound images and the multiple unmarked fetal ultrasound images; establishing a mark removal dataset based on the multiple fetal ultrasound image sample pairs, and preprocessing and channel splitting operations are performed on the marked fetal ultrasound images and the unmarked fetal ultrasound images in each fetal ultrasound image sample pair in the mark removal dataset, so as to obtain a standard expanded mark removal dataset corresponding to the mark removal dataset; determining a second objective function corresponding to the mark removal model, so as to train the mark removal model based on the standard expanded mark removal dataset and the second objective function to obtain the redrawing model.

[0048] In the actual implementation process, the embodiment of the present application selects a certain proportion of image samples from the fetal ultrasound images containing labels from multiple fetal ultrasound images, and finds the corresponding images from the fetal ultrasound images without labels, constructs fetal ultrasound image sample pairs, so as to establish a label removal data set (i.e., training data of the label removal model), and performs data enhancement on the label removal data set, i.e., randomly adding various labels of the same color to the fetal ultrasound images containing labels; further, the embodiment of the present application can adopt a label feature extraction operation, and at the same time, use the ultrasound image without label information as the target image, which corresponds one-to-one with the ultrasound image containing labels; finally, the embodiment of the present application can perform channel splitting on the target image and the training image to convert a three-channel image into three single-channel images, such as Figure 3 As shown, single-channel labeled fetal ultrasound data and corresponding label masks, which are three times the number of samples, are obtained, namely, a standard expanded label removal data set to achieve the purpose of training data expansion.

[0049] Afterwards, the embodiments of the present application can construct a marker removal model, the overall structure of which is generally similar to U-Net or other convolutional neural network architectures, which include encoder and decoder parts. The difference is that each layer of convolution operation is a partial convolution, that is, the convolution operation is limited to the valid image area. The second objective function is in the form of a combination of multiple losses, including pixel-level reconstruction loss, perceptual loss, style loss, and total variational loss. These losses work together to enable the network to not only generate correct images at the pixel level when repairing images, but also maintain consistency in high-level features and textures, and the generated repair results are more realistic and natural.

[0050] Finally, the embodiments of the present application can train the mark removal model through the standard expanded mark removal data set and the second objective function. The training process aims to minimize the value of the objective function. The trained mark removal model is obtained according to the final weight parameters of the discriminator and the generator, and the trained mark removal model is used as the inpainting model.

[0051] In step S103, based on the prediction model and the inpainting model, an end-to-end label removal operation is performed on the target fetal ultrasound image with labels.

[0052] Furthermore, the embodiments of the present application can combine two deep neural network models to achieve end-to-end fetal ultrasound image label removal.

[0053] In the actual implementation process, the embodiment of the present application can use computer languages ​​such as Python and utilize the Pytorch development framework to implement the above-mentioned fetal ultrasound image de-labeling operation based on image channel splitting deep learning.

[0054] Therefore, the embodiments of the present application have the characteristics of high adaptability, etc., can effectively solve the problem that marks in fetal ultrasound images are difficult to identify and remove, and can be easily extended to the task of mark removal of other medical ultrasound images.

[0055] Optionally, in one embodiment of the present application, based on the prediction model and the inpainting model, an end-to-end label removal operation is performed on the target fetal ultrasound image with labels, including: combining the prediction model and the inpainting model to construct a final label removal network model; using the final label removal network model to perform an end-to-end label removal operation on the target fetal ultrasound image with labels to obtain the target fetal ultrasound image after the labels are removed.

[0056] It should be noted that the embodiment of the present application combines the two networks of the prediction model and the infill model to obtain the final label removal network model, so that when a fetal ultrasound image to be labeled is input (i.e., a target fetal ultrasound image with labels) is input, a label removal operation is performed on it, thereby obtaining a fetal ultrasound image after the labels are removed.

[0057] In the process of specifically combining the two models, the mask image used by the infill model of each channel in the embodiment of the present application is a combination of mask images generated by the marker feature extraction models of the three channels, so as to more comprehensively cover the various marker features included.

[0058] Therefore, the embodiments of the present application take into account the problems of different resolutions, marking styles, marking colors and image qualities of ultrasound images between different devices, and achieve a unified style of ultrasound images through channel splitting, thereby improving the versatility and robustness of the marking removal technology, thereby realizing end-to-end ultrasound image marker extraction and infilling, and significantly improving the automation and intelligence of marker removal.

[0059] According to the fetal ultrasound image de-labeling method based on image channel splitting proposed in the embodiment of the present application, a plurality of fetal ultrasound images are obtained, and a marker feature extraction data set is constructed based on the plurality of fetal ultrasound images, and the data set is expanded according to the channel splitting technology, and the pre-constructed marker feature extraction model and marker feature infilling model are trained using the data set expanded by the channel splitting technology, so as to perform multi-dimensional feature extraction and infilling based on two deep learning models of marker feature extraction and marker feature infilling, thereby realizing end-to-end marker removal operation by using channel splitting technology. The present application realizes the style unification of ultrasound images through channel splitting, thereby improving the versatility and robustness of the marker removal technology, realizing end-to-end ultrasound image marker extraction and infilling, thereby significantly improving the automation and intelligence level of marker removal.

[0060] Next, a fetal ultrasound image de-labeling device based on image channel splitting proposed in an embodiment of the present application is described with reference to the accompanying drawings.

[0061] Figure 4 It is a block diagram of a fetal ultrasound image de-labeling device based on image channel splitting according to an embodiment of the present application.

[0062] like Figure 4 As shown, the fetal ultrasound image de-marking device 10 based on image channel splitting includes: a marker feature extraction module 100 , a marker feature infilling module 200 and an integration processing module 300 .

[0063] Among them, the marker feature extraction module 100 is used to obtain multiple fetal ultrasound images, and construct a marker feature extraction data set based on the multiple fetal ultrasound images, and expand the marker feature extraction data set using a preset image channel splitting strategy, and based on the expanded marker feature extraction data set, train a pre-constructed marker feature extraction model to obtain a prediction model corresponding to the marker feature extraction model.

[0064] The marker feature inpainting module 200 is used to construct a marker removal dataset based on multiple fetal ultrasound images, expand the marker removal dataset using an image channel splitting strategy, and train a pre-constructed marker removal model based on the expanded marker removal dataset to obtain an inpainting model corresponding to the marker removal model.

[0065] The integrated processing module 300 is used to perform an end-to-end label removal operation on the target fetal ultrasound image with labels based on the prediction model and the inpainting model.

[0066] Optionally, in one embodiment of the present application, the marker feature extraction module 100 includes: a first acquisition unit, an extraction unit, an adjustment unit, a conversion unit, a channel splitting unit, a first determination unit and a training unit.

[0067] The first acquisition unit is used to acquire a plurality of fetal ultrasound images with labels from a plurality of fetal ultrasound images, and construct a label feature extraction data set through the plurality of fetal ultrasound images with labels.

[0068] The extraction unit is used to extract the border text information corresponding to each marked fetal ultrasound image in the mark feature extraction data set, and fill the missing part of each marked fetal ultrasound image after extracting the border text information based on a preset image restoration strategy to obtain a marked filled image.

[0069] The adjusting unit is used to adjust the image size of each marked filling image to obtain a marked filling image of a target size.

[0070] The conversion unit is used for randomly adding multiple types of marks of the same color to the filled image with the mark, and converting the filled image with the mark to the HSV space to extract the mark mask corresponding to the filled image with the mark.

[0071] The channel splitting unit is used to perform channel splitting on the filled image with the mark and copy the mark mask to obtain a standard extended mark feature extraction data set corresponding to the mark feature extraction data set.

[0072] The first determination unit is used to construct a marker feature extraction model based on a preset encoder, decoder and U-Net framework, and determine a first objective function corresponding to the marker feature extraction model.

[0073] The training unit is used to train the marker feature extraction model through the standard expanded marker feature extraction data set and the first objective function to obtain a prediction model.

[0074] Optionally, in one embodiment of the present application, the marking feature in-painting module 200 includes: a second acquiring unit, a establishing unit and a second determining unit.

[0075] Among them, the second acquisition unit is used to acquire multiple marked fetal ultrasound images from multiple fetal ultrasound images, and acquire multiple unmarked fetal ultrasound images corresponding to the multiple marked fetal ultrasound images from the multiple fetal ultrasound images, so as to construct multiple fetal ultrasound image sample pairs according to the multiple marked fetal ultrasound images and the multiple unmarked fetal ultrasound images.

[0076] An establishing unit is used to establish a label removal data set based on multiple fetal ultrasound image sample pairs, and preprocess and perform channel splitting operations on the labeled fetal ultrasound images and the unlabeled fetal ultrasound images in each fetal ultrasound image sample pair in the label removal data set, so as to obtain a standard extended label removal data set corresponding to the label removal data set.

[0077] The second determining unit is used to determine a second objective function corresponding to the mark removal model, so as to train the mark removal model according to the standard expanded mark removal data set and the second objective function to obtain the inpainting model.

[0078] Optionally, in one embodiment of the present application, the integration processing module 300 includes: a combination unit and a processing unit, wherein the combination unit is used to combine the prediction model and the inpainting model to construct a final mark removal network model.

[0079] The processing unit is used to perform an end-to-end label removal operation on the target fetal ultrasound image with labels by using the final label removal network model to obtain the target fetal ultrasound image after the labels are removed.

[0080] It should be noted that the above explanation of the embodiment of the fetal ultrasound image de-marking method based on image channel splitting is also applicable to the fetal ultrasound image de-marking device based on image channel splitting of this embodiment, and will not be repeated here.

[0081] According to the fetal ultrasound image de-marking device based on image channel splitting proposed in the embodiment of the present application, a plurality of fetal ultrasound images are acquired, and a marker feature extraction data set is constructed based on the plurality of fetal ultrasound images, and the data set is expanded according to the channel splitting technology, and the pre-constructed marker feature extraction model and marker feature infilling model are trained using the data set expanded by the channel splitting technology, so as to perform multi-dimensional feature extraction and infilling based on two deep learning models of marker feature extraction and marker feature infilling, and thus realize end-to-end marker removal operation by using channel splitting technology. The present application realizes the style unification of ultrasound images through channel splitting, thereby improving the versatility and robustness of the marker removal technology, and realizes end-to-end ultrasound image marker extraction and infilling, thereby significantly improving the automation and intelligence level of marker removal.

[0082] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0083] A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0084] When the processor 502 executes the program, the fetal ultrasound image de-labeling method based on image channel splitting provided in the above embodiment is implemented.

[0085] Furthermore, the electronic device further comprises:

[0086] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0087] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0088] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0089] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0090] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0091] The processor 502 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0092] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned fetal ultrasound image de-labeling method based on image channel splitting.

[0093] An embodiment of the present application further provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned fetal ultrasound image de-labeling method based on image channel splitting.

[0094] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0095] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0096] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0097] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0098] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0099] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0100] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0101] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A fetal ultrasound image de-labeling method based on image channel splitting, characterized in that: The following steps are involved: Acquire multiple fetal ultrasound images, and construct a marker feature extraction data set according to the multiple fetal ultrasound images, and expand the marker feature extraction data set using a preset image channel splitting strategy, and train a pre-constructed marker feature extraction model based on the expanded marker feature extraction data set to obtain a prediction model corresponding to the marker feature extraction model; Constructing a marker removal dataset according to the multiple fetal ultrasound images, expanding the marker removal dataset using the image channel splitting strategy, and training a pre-constructed marker removal model based on the expanded marker removal dataset to obtain an inpainting model corresponding to the marker removal model; Based on the prediction model and the inpainting model, an end-to-end label removal operation is performed on the fetal ultrasound image with the target label.

2. The method according to claim 1, characterized in that The method comprises acquiring a plurality of fetal ultrasound images, constructing a marker feature extraction data set according to the plurality of fetal ultrasound images, expanding the marker feature extraction data set by using a preset image channel splitting strategy, and training a pre-constructed marker feature extraction model based on the expanded marker feature extraction data set to obtain a prediction model corresponding to the marker feature extraction model, including: Acquire a plurality of fetal ultrasound images with labels from the plurality of fetal ultrasound images, and construct the label feature extraction data set through the plurality of fetal ultrasound images with labels; Extracting the frame text information corresponding to each marked fetal ultrasound image in the mark feature extraction data set, and filling the missing part of each marked fetal ultrasound image after extracting the frame text information based on a preset image restoration strategy to obtain a marked filled image; Adjusting the image size of each marked padded image to obtain a marked padded image of a target size; Randomly adding multiple types of marks of the same color to the marked filled image, and converting the marked filled image to the HSV space to extract a mark mask corresponding to the marked filled image; Performing channel splitting on the marked filled image and copying the marked mask to obtain a standard extended marked feature extraction data set corresponding to the marked feature extraction data set; Based on the preset encoder, decoder and U-Net framework, construct the marker feature extraction model, and determine the first objective function corresponding to the marker feature extraction model; The marker feature extraction model is trained using the standard expanded marker feature extraction data set and the first objective function to obtain the prediction model.

3. The method according to claim 2, characterized in that The step of constructing a mark removal data set according to the plurality of fetal ultrasound images, expanding the mark removal data set by using the image channel splitting strategy, and training a pre-constructed mark removal model based on the expanded mark removal data set to obtain a redrawing model corresponding to the mark removal model includes: Acquire a plurality of fetal ultrasound images with labels from among the plurality of fetal ultrasound images, and acquire a plurality of fetal ultrasound images without labels corresponding to the plurality of fetal ultrasound images with labels from among the plurality of fetal ultrasound images, so as to construct a plurality of fetal ultrasound image sample pairs according to the plurality of fetal ultrasound images with labels and the plurality of fetal ultrasound images without labels; Based on the multiple fetal ultrasound image sample pairs, the label-removed dataset is established, and the labeled fetal ultrasound image and the unlabeled fetal ultrasound image in each fetal ultrasound image sample pair in the label-removed dataset are preprocessed and channel split, respectively, to obtain a standard extended label-removed dataset corresponding to the label-removed dataset; A second objective function corresponding to the mark removal model is determined, so as to train the mark removal model according to the standard expanded mark removal data set and the second objective function to obtain the inpainting model.

4. The method according to claim 3, characterized in that: The end-to-end label removal operation is performed on the target fetal ultrasound image with labels based on the prediction model and the inpainting model, including: Combining the prediction model and the inpainting model to construct a final label removal network model; The final label removal network model is used to perform a label removal operation on the target fetal ultrasound image with labels to obtain the target fetal ultrasound image after the labels are removed.

5. A fetal ultrasound image de-marking device based on image channel splitting, characterized in that: include: A marker feature extraction module is used to acquire a plurality of fetal ultrasound images, construct a marker feature extraction data set according to the plurality of fetal ultrasound images, expand the marker feature extraction data set using a preset image channel splitting strategy, and train a pre-constructed marker feature extraction model based on the expanded marker feature extraction data set to obtain a prediction model corresponding to the marker feature extraction model; a marker feature inpainting module, configured to construct a marker removal dataset according to the plurality of fetal ultrasound images, expand the marker removal dataset using the image channel splitting strategy, and train a pre-constructed marker removal model based on the expanded marker removal dataset to obtain an inpainting model corresponding to the marker removal model; The integrated processing module is used to perform an end-to-end label removal operation on the fetal ultrasound image with the target label based on the prediction model and the inpainting model.

6. The device according to claim 5, characterized in that The marker feature extraction module comprises: a first acquisition unit, configured to acquire a plurality of fetal ultrasound images with labels from the plurality of fetal ultrasound images, and construct the label feature extraction data set through the plurality of fetal ultrasound images with labels; an extraction unit, configured to extract the frame text information corresponding to each fetal ultrasound image with a mark in the mark feature extraction data set, and fill the missing part of each fetal ultrasound image with a mark after the frame text information is extracted based on a preset image restoration strategy, so as to obtain a filled image with a mark; An adjusting unit, used for adjusting the image size of each marked filling image to obtain a marked filling image of a target size; A conversion unit, configured to randomly add multiple types of marks of the same color to the marked filling image, and convert the marked filling image into an HSV space to extract a mark mask corresponding to the marked filling image; A channel splitting unit, used for performing channel splitting on the marked filled image and copying the marked mask to obtain a standard extended marked feature extraction data set corresponding to the marked feature extraction data set; A first determining unit, configured to construct the marker feature extraction model based on a preset encoder, decoder and U-Net framework, and determine a first objective function corresponding to the marker feature extraction model; A training unit is used to train the marker feature extraction model through the standard expanded marker feature extraction data set and the first objective function to obtain the prediction model.

7. The device according to claim 6, characterized in that The marking feature infill module comprises: a second acquisition unit, configured to acquire a plurality of fetal ultrasound images with labels from among the plurality of fetal ultrasound images, and acquire a plurality of fetal ultrasound images without labels corresponding to the plurality of fetal ultrasound images with labels from among the plurality of fetal ultrasound images, so as to construct a plurality of fetal ultrasound image sample pairs according to the plurality of fetal ultrasound images with labels and the plurality of fetal ultrasound images without labels; an establishing unit, configured to establish the label-removed data set based on the plurality of fetal ultrasound image sample pairs, and to perform preprocessing and channel splitting operations on the labeled fetal ultrasound image and the unlabeled fetal ultrasound image in each fetal ultrasound image sample pair in the label-removed data set, respectively, so as to obtain a standard extended label-removed data set corresponding to the label-removed data set; The second determining unit is used to determine a second objective function corresponding to the mark removal model, so as to train the mark removal model according to the standard expanded mark removal data set and the second objective function to obtain the inpainting model.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the fetal ultrasound image de-labeling method based on image channel splitting as described in any one of claims 1 to 4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the fetal ultrasound image de-labeling method based on image channel splitting as described in any one of claims 1 to 4.

10. A computer program product, comprising a computer program, characterized in that The computer program is executed to implement the fetal ultrasound image de-labeling method based on image channel splitting as described in any one of claims 1 to 4.

Images