Fetal ultrasound image de-identification method and device based on image channel splitting

Abstract

The application relates to a fetal ultrasound image label removing method and device based on image channel splitting, wherein the method comprises the following steps: acquiring a plurality of fetal ultrasound images, constructing a label feature extraction dataset according to the plurality of fetal ultrasound images, expanding the dataset according to a channel splitting technology, training a pre-constructed label feature extraction model and a label feature filling model by using the dataset expanded by the channel splitting technology, and removing labels in an end-to-end mode by using the two deep learning models of label feature extraction and label feature filling and the channel splitting technology for multi-dimensional feature extraction and filling. Therefore, the problems that the resolution, label style, label color and image quality of ultrasound images of different devices cannot be unified in the prior art, and labels in fetal ultrasound images cannot be effectively recognized and removed are solved.

Description

Technical Field

[0001] The present application relates to the technical field of medical image analysis and processing, and in particular to a fetal ultrasound image de-labeling method and device based on image channel splitting. Background Art

[0002] Traditional de-labeling methods rely on identifying the color of markers in ultrasound images. This involves extracting the color and sampling the area surrounding the markers to achieve inpainting. While this traditional inpainting method can achieve de-labeling results, it often suffers from a rough treatment of details, which can interfere with subsequent image analysis. In particular, if these processed ultrasound images are subsequently used as training sets for other deep learning tasks, a de-labeling method that better reflects the characteristic distribution of medical images is essential for more reliable training results.

[0003] Deep neural networks are currently the most popular image processing technology, and they are highly effective for images with complex structures and unclear features. Furthermore, traditional deep learning methods tend to perform well on natural image processing tasks requiring large amounts of data and manual labeling. However, these traditional deep learning methods often fail to achieve the desired results when applied to medical images, particularly fetal ultrasound images, which are characterized by high noise levels, complex structures, and unclear features.

[0004] Furthermore, the collection and labeling of medical images requires significant manpower and medical knowledge. Given limited data, enabling deep neural networks to more accurately learn key knowledge and achieve end-to-end label removal is challenging. Furthermore, the diversity of ultrasound equipment presents new requirements and challenges for ultrasound image label removal. Different labeling formats require models with high versatility. Existing deep learning methods often achieve good delabeling results for labeling formats visible in the training set, but lack appropriate responses for labeling types beyond those covered by the training set.

[0005] In summary, existing technologies have difficulty in unifying the resolution, marking style, marking color, and image quality of ultrasound images between different devices, and are unable to effectively identify and remove markings in fetal ultrasound images, which urgently needs to be solved. Summary of the Invention

[0006] The present application provides a fetal ultrasound image de-marking method and device based on image channel splitting to solve the problems of the existing technology that it is difficult to unify the resolution, marking style, marking color and image quality of ultrasound images between different devices, and it is impossible to effectively identify and remove marks in fetal ultrasound images.

[0007] The first aspect of the present application provides a fetal ultrasound image de-labeling method based on image channel splitting, comprising the following steps: acquiring multiple fetal ultrasound images, and constructing a marker feature extraction dataset based on the multiple fetal ultrasound images, and expanding the marker feature extraction dataset using a preset image channel splitting strategy, and training a pre-constructed marker feature extraction model based on the expanded marker feature extraction dataset to obtain a prediction model corresponding to the marker feature extraction model; constructing a label removal dataset based on the multiple fetal ultrasound images, and expanding the label removal dataset using the image channel splitting strategy, and training a pre-constructed marker removal model based on the expanded marker removal dataset to obtain a redrawing model corresponding to the label removal model; and performing an end-to-end label removal operation on the target fetal ultrasound image with labels based on the prediction model and the redrawing model.

[0008] Optionally, in one embodiment of the present application, the method of acquiring multiple fetal ultrasound images, constructing a marker feature extraction dataset based on the multiple fetal ultrasound images, and expanding the marker feature extraction dataset using a preset image channel splitting strategy, and training a pre-constructed marker feature extraction model based on the expanded marker feature extraction dataset to obtain a prediction model corresponding to the marker feature extraction model, includes: acquiring multiple marked fetal ultrasound images from the multiple fetal ultrasound images, and constructing the marker feature extraction dataset through the multiple marked fetal ultrasound images; extracting border text information corresponding to each marked fetal ultrasound image in the marker feature extraction dataset, and filling in the missing parts of each marked fetal ultrasound image after extracting the border text information based on a preset image repair strategy. , to obtain a marked filled image; adjust the image size of each marked filled image to obtain a marked filled image of a target size; randomly add multiple types of marks of the same color to the marked filled image, and convert the marked filled image to the HSV space to extract the mark mask corresponding to the marked filled image; perform channel splitting on the marked filled image and copy the mark mask to obtain a standard extended marked feature extraction dataset corresponding to the marked feature extraction dataset; based on a preset encoder, decoder and U-Net framework, construct the marked feature extraction model, and determine the first objective function corresponding to the marked feature extraction model; train the marked feature extraction model through the standard extended marked feature extraction dataset and the first objective function to obtain the prediction model.

[0009] Optionally, in one embodiment of the present application, constructing a label removal dataset based on the multiple fetal ultrasound images, expanding the label removal dataset using the image channel splitting strategy, and training a pre-constructed label removal model based on the expanded label removal dataset to obtain a redrawing model corresponding to the label removal model, includes: obtaining multiple labeled fetal ultrasound images from the multiple fetal ultrasound images, and obtaining multiple unlabeled fetal ultrasound images corresponding to the multiple labeled fetal ultrasound images from the multiple fetal ultrasound images, so as to construct multiple fetal ultrasound image sample pairs based on the multiple labeled fetal ultrasound images and the multiple unlabeled fetal ultrasound images; establishing the label removal dataset based on the multiple fetal ultrasound image sample pairs, and performing preprocessing and channel splitting operations on the labeled fetal ultrasound images and the unlabeled fetal ultrasound images in each fetal ultrasound image sample pair in the label removal dataset, so as to obtain a standard expanded label removal dataset corresponding to the label removal dataset; and determining a second objective function corresponding to the label removal model, so as to train the label removal model based on the standard expanded label removal dataset and the second objective function to obtain the redrawing model.

[0010] Optionally, in one embodiment of the present application, the end-to-end label removal operation is performed on the target fetal ultrasound image with labels based on the prediction model and the inpainting model, including: combining the prediction model and the inpainting model to construct a final label removal network model; using the final label removal network model to perform a label removal operation on the target fetal ultrasound image with labels to obtain the target fetal ultrasound image after the labels are removed.

[0011] The second aspect of the present application provides a fetal ultrasound image de-labeling device based on image channel splitting, including: a marker feature extraction module, used to acquire multiple fetal ultrasound images, and construct a marker feature extraction data set based on the multiple fetal ultrasound images, and expand the marker feature extraction data set using a preset image channel splitting strategy, and train a pre-constructed marker feature extraction model based on the expanded marker feature extraction data set to obtain a prediction model corresponding to the marker feature extraction model; a marker feature inpainting module, used to construct a marker removal data set based on the multiple fetal ultrasound images, and expand the marker removal data set using the image channel splitting strategy, and train a pre-constructed marker removal model based on the expanded marker removal data set to obtain a inpainting model corresponding to the marker removal model; an integration processing module, used to perform an end-to-end marker removal operation on the target fetal ultrasound image with markers based on the prediction model and the inpainting model.

[0012] Optionally, in one embodiment of the present application, the marker feature extraction module includes: a first acquisition unit, used to acquire multiple marked fetal ultrasound images from the multiple fetal ultrasound images, and construct the marker feature extraction data set through the multiple marked fetal ultrasound images; an extraction unit, used to extract the border text information corresponding to each marked fetal ultrasound image in the marker feature extraction data set, and based on a preset image repair strategy, fill the missing part of each marked fetal ultrasound image after extracting the border text information to obtain a marked filled image; an adjustment unit, used to adjust the image size of each marked filled image to obtain a marked filled image of a target size; a conversion unit, used to convert the marked filled image into a marked filled image. Multiple types of marks of the same color are randomly added to the marked filled image, and the marked filled image is converted into HSV space to extract the mark mask corresponding to the marked filled image; a channel splitting unit is used to perform channel splitting on the marked filled image and copy the mark mask to obtain a standard extended mark feature extraction dataset corresponding to the mark feature extraction dataset; a first determination unit is used to construct the mark feature extraction model based on a preset encoder, decoder and U-Net framework, and determine the first objective function corresponding to the mark feature extraction model; a training unit is used to train the mark feature extraction model through the standard extended mark feature extraction dataset and the first objective function to obtain the prediction model.

[0013] Optionally, in one embodiment of the present application, the marker feature infilling module includes: a second acquisition unit, used to acquire multiple marked fetal ultrasound images from the multiple fetal ultrasound images, and acquire multiple unmarked fetal ultrasound images corresponding to the multiple marked fetal ultrasound images from the multiple fetal ultrasound images, so as to construct multiple fetal ultrasound image sample pairs based on the multiple marked fetal ultrasound images and the multiple unmarked fetal ultrasound images; an establishment unit, used to establish the marker removal dataset based on the multiple fetal ultrasound image sample pairs, and perform preprocessing and channel splitting operations on the marked fetal ultrasound images and the unmarked fetal ultrasound images in each fetal ultrasound image sample pair in the marker removal dataset, so as to obtain a standard extended marker removal dataset corresponding to the marker removal dataset; a second determination unit, used to determine a second objective function corresponding to the marker removal model, so as to train the marker removal model according to the standard extended marker removal dataset and the second objective function, so as to obtain the infilling model.

[0014] Optionally, in an embodiment of the present application, the integration processing module comprises a combination unit configured to combine the prediction model and the inpainting model to construct a final label removal network model; and a processing unit configured to perform label removal operation on the target labeled fetal ultrasound image by using the final label removal network model to obtain a label-removed target fetal ultrasound image.

[0015] In a third aspect, an electronic device is provided, which comprises a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The processor executes the program to implement the fetal ultrasound image label removal method based on image channel splitting as described in the above embodiments.

[0016] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program. The program is executed by a processor to implement the fetal ultrasound image label removal method based on image channel splitting as described above.

[0017] In a fifth aspect, a computer program product is provided, which comprises a computer program. The computer program is executed to implement the fetal ultrasound image label removal method based on image channel splitting as described above.

[0018] Therefore, the embodiments of the present application have the following beneficial effects:

[0019] The embodiments of the present application can obtain a plurality of fetal ultrasound images, construct a label feature extraction dataset according to the plurality of fetal ultrasound images, expand the dataset according to a channel splitting technology, train a pre-constructed label feature extraction model and a label feature inpainting model by using the dataset expanded by the channel splitting technology, and perform multi-dimensional feature extraction and inpainting by using the channel splitting technology based on the two deep learning models of label feature extraction and label feature inpainting, so as to realize end-to-end label removal operation. The present application realizes style unification of ultrasound images by using the channel splitting technology, thereby improving the generality and robustness of the label removal technology, and realizing end-to-end ultrasound image label extraction and inpainting, thereby significantly improving the automation and intelligence level of label removal. Thus, the problems that the prior art cannot unify the resolution, label style, label color and image quality of ultrasound images between different devices, and cannot effectively identify and remove the labels in fetal ultrasound images are solved.

[0020] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0021] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description of embodiments, taken in conjunction with the accompanying drawings, in which:

[0022] Figure 1 A flow chart of a fetal ultrasound image delabeling method based on image channel splitting according to an embodiment of the present application;

[0023] Figure 2 A flow chart of extracting fetal ultrasound image label features according to an embodiment of the present application;

[0024] Figure 3 A flow chart of removing fetal ultrasound image labels and performing redrawing according to an embodiment of the present application;

[0025] Figure 4 An example diagram of a fetal ultrasound image delabeling device based on image channel splitting according to an embodiment of the present application;

[0026] Figure 5 A structural diagram of an electronic device according to an embodiment of the present application.

[0027] Wherein, 10 - fetal ultrasound image delabeling device based on image channel splitting; 100 - label feature extraction module, 200 - label feature redrawing module, 300 - integration processing module; 501 - memory, 502 - processor, 503 - communication interface. DETAILED DESCRIPTION

[0028] Embodiments of the present application are described in detail below with reference to the attached drawings, which are examples of embodiments of the present application. The embodiments described below are examples and are intended to explain the present application, and should not be understood as limiting the present application.

[0029] The following describes a fetal ultrasound image de-labeling method and apparatus based on image channel splitting according to an embodiment of the present application with reference to the accompanying drawings. To address the issues mentioned in the background art, the present application provides a fetal ultrasound image de-labeling method based on image channel splitting. In this method, multiple fetal ultrasound images are acquired, and a marker feature extraction dataset is constructed based on the multiple fetal ultrasound images. The dataset is expanded using the channel splitting technique, and a pre-built marker feature extraction model and a marker feature infilling model are trained using the dataset expanded using the channel splitting technique. Based on two deep learning models for marker feature extraction and marker feature infilling, multi-dimensional feature extraction and infilling are performed using the channel splitting technique, thereby achieving end-to-end marker removal. This application achieves a unified style for ultrasound images through channel splitting, thereby improving the versatility and robustness of the marker removal technique, achieving end-to-end ultrasound image marker extraction and infilling, and significantly improving the automation and intelligence level of marker removal. This solves the problems of the prior art in being unable to unify the resolution, marker style, marker color, and image quality of ultrasound images between different devices, and ineffectively identifying and removing markers from fetal ultrasound images.

[0030] Specifically, Figure 1 This is a flowchart of a fetal ultrasound image delabeling method based on image channel splitting provided in an embodiment of the present application.

[0031] like Figure 1 As shown, the fetal ultrasound image de-labeling method based on image channel splitting includes the following steps:

[0032] In step S101, a plurality of fetal ultrasound images are acquired, and a marker feature extraction dataset is constructed based on the plurality of fetal ultrasound images. The marker feature extraction dataset is expanded using a preset image channel splitting strategy, and a pre-constructed marker feature extraction model is trained based on the expanded marker feature extraction dataset to obtain a prediction model corresponding to the marker feature extraction model.

[0033] The embodiments of the present application can first acquire multiple fetal ultrasound images to construct a marker feature extraction dataset, and perform data preprocessing and image channel splitting on them to expand the marker feature extraction dataset; secondly, construct a convolutional neural network model based on the U-Net architecture for marker feature extraction (i.e., a marker feature extraction model) and an objective function corresponding to the network model; thereafter, the embodiments of the present application need to train the marker feature extraction model based on the expanded marker feature extraction dataset and the corresponding objective function, and use the trained marker feature extraction model as a prediction model.

[0034] Optionally, in one embodiment of the present application, a plurality of fetal ultrasound images are obtained, and a marker feature extraction data set is constructed based on the plurality of fetal ultrasound images, and the marker feature extraction data set is expanded using a preset image channel splitting strategy, and based on the expanded marker feature extraction data set, a pre-constructed marker feature extraction model is trained to obtain a prediction model corresponding to the marker feature extraction model, including: obtaining a plurality of marked fetal ultrasound images from a plurality of fetal ultrasound images, and constructing a marker feature extraction data set through the plurality of marked fetal ultrasound images; extracting border text information corresponding to each marked fetal ultrasound image in the marker feature extraction data set, and based on a preset image repair strategy, repairing the missing part of each marked fetal ultrasound image after extracting the border text information. Filling to obtain a marked filled image; adjusting the image size of each marked filled image to obtain a marked filled image of a target size; randomly adding multiple types of marks of the same color to the marked filled image, and converting the marked filled image to the HSV space to extract the marking mask corresponding to the marked filled image; performing channel splitting on the marked filled image and copying the marking mask to obtain a standard extended marked feature extraction dataset corresponding to the marked feature extraction dataset; constructing a marked feature extraction model based on a preset encoder, decoder and U-Net framework, and determining the first objective function corresponding to the marked feature extraction model; training the marked feature extraction model through the standard extended marked feature extraction dataset and the first objective function to obtain a prediction model.

[0035] It should be noted that the embodiments of the present application can select a certain proportion of multiple fetal ultrasound images with labels from multiple fetal ultrasound images and preprocess the labeled fetal ultrasound images. The data preprocessing requires data processing and enhancement of the labeled fetal ultrasound images and image channel splitting so that the training data can be expanded to three times the number of samples. The specific data preprocessing process is as follows:

[0036] 1. Perform OCR on the text around the ultrasound image and fill in the missing parts using an image inpainting technique based on a fast marching algorithm. This yields a fetal ultrasound image dataset that removes the text but still contains the labeled information (i.e., the fetal ultrasound image dataset corresponding to the filled-in image with the labels).

[0037] 2. Adjust the image size in the dataset. If the image size is larger than 960×720 (i.e., the target size), the image is segmented using the sliding window method to obtain several 960×720 image slices; if the image size is smaller than 960×720, the image is expanded so that the size of all images in the training set is unified to 960×720;

[0038] 3. Data augmentation was performed on the fetal ultrasound images with markers. 100 examples of each of the three marker styles, dashed lines, letters, and crosshairs, were added to the original markers. The colors were the same as the original markers.

[0039] 4. Convert the marked fetal ultrasound image to HSV space, set the range of the marked color, and extract the marked mask as the target of the training data;

[0040] 5. Split the channels of the fetal ultrasound image with markers and copy the marker masks to achieve a one-to-one correspondence effect, such as Figure 2 As shown, the training data is expanded to three times the number of samples to obtain the preprocessed training set (i.e., the standard expanded labeled feature extraction data set) and the training target;

[0041] Secondly, based on the preset encoder, decoder and U-Net framework, a marker feature extraction model is constructed, and the first objective function corresponding to the marker feature extraction model is determined; the marker feature extraction model is trained using the standard expanded marker feature extraction dataset and the first objective function to obtain a prediction model.

[0042] Again, the embodiments of the present application can construct a marker feature extraction model. The overall network structure of the marker feature extraction model continues the classic U-Net framework, including an encoder (downsampling), a decoder (upsampling) and jump connections. In addition, the embodiments of the present application adopt a combination of Dice loss and cross entropy loss to construct the objective function (i.e., the first objective function) of the marker feature extraction model.

[0043] Afterwards, the embodiment of the present application can train the marker feature extraction model through the standard expanded marker feature extraction data set and the first objective function. The training process aims to minimize the value of the objective function. The trained network model is obtained according to the final weight parameters of the discriminator and the generator, and the trained network model is used as the final marker feature extraction model, that is, the prediction model.

[0044] It can be understood that the embodiments of the present application realize the automatic extraction of marker features, and also realize the inpainting technology based on the extracted marker features that conforms to the original features of the ultrasound image, thereby achieving end-to-end realization of the entire process of marker removal, and significantly improving the automation and intelligence of marker removal; in addition, the embodiments of the present application also take into account the problems of different resolutions, marker styles, marker colors and image qualities of ultrasound images between different devices, thereby achieving a unified style of ultrasound images through channel splitting, thereby improving the versatility and robustness of the marker removal technology.

[0045] In step S102, a marker removal dataset is constructed based on multiple fetal ultrasound images, and the marker removal dataset is expanded using an image channel splitting strategy. Based on the expanded marker removal dataset, a pre-constructed marker removal model is trained to obtain an inpainting model corresponding to the marker removal model.

[0046] Furthermore, the embodiments of the present application also need to construct a marker removal dataset through the acquisition of multiple fetal ultrasound images, and perform dataset preprocessing and image channel splitting on it to expand the marker removal dataset; secondly, the embodiments of the present application need to construct a U-Net architecture convolutional neural network model based on partial convolution for marker removal (i.e., a marker removal model), and determine the objective function corresponding to the marker removal model (i.e., the second objective function); thereafter, the embodiments of the present application can train the marker removal model based on the expanded marker removal dataset and the second objective function, and use the trained marker removal model as the infill model.

[0047] Optionally, in one embodiment of the present application, a label removal dataset is constructed based on multiple fetal ultrasound images, and the label removal dataset is expanded using an image channel splitting strategy, and a pre-constructed label removal model is trained based on the expanded label removal dataset to obtain a redrawing model corresponding to the label removal model, including: obtaining multiple labeled fetal ultrasound images from multiple fetal ultrasound images, and obtaining multiple unlabeled fetal ultrasound images corresponding to the multiple labeled fetal ultrasound images from the multiple fetal ultrasound images, so as to construct multiple fetal ultrasound image sample pairs based on the multiple labeled fetal ultrasound images and the multiple unlabeled fetal ultrasound images; establishing a label removal dataset based on the multiple fetal ultrasound image sample pairs, and performing preprocessing and channel splitting operations on the labeled fetal ultrasound images and the unlabeled fetal ultrasound images in each fetal ultrasound image sample pair in the label removal dataset, so as to obtain a standard expanded label removal dataset corresponding to the label removal dataset; determining a second objective function corresponding to the label removal model, so as to train the label removal model based on the standard expanded label removal dataset and the second objective function to obtain the redrawing model.

[0048] In the actual implementation process, the embodiment of the present application selects a certain proportion of image samples from the fetal ultrasound images containing labels from multiple fetal ultrasound images, and finds their corresponding images from the fetal ultrasound images without labels, constructs fetal ultrasound image sample pairs, and establishes a label removal data set (i.e., training data for the label removal model), and performs data enhancement on the label removal data set, i.e., randomly adding various labels of the same color to the fetal ultrasound images containing labels; furthermore, the embodiment of the present application can adopt a label feature extraction operation, and at the same time, use the ultrasound image without label information as the target image, which corresponds one-to-one with the ultrasound image containing labels; finally, the embodiment of the present application can perform channel splitting on the target image and the training image to convert a three-channel image into three single-channel images, such as Figure 3 As shown in FIG, single-channel labeled fetal ultrasound data and corresponding label masks, which are three times the number of samples, are obtained, namely, the standard expanded label removal dataset, so as to achieve the purpose of training data expansion.

[0049] Afterwards, the embodiments of the present application can construct a marker removal model, the overall structure of which is generally similar to U-Net or other convolutional neural network architectures, which include encoder and decoder parts. The difference is that each layer of convolution operation is a partial convolution, that is, the convolution operation is limited to the valid image area. The second objective function is in the form of a combination of multiple losses, including pixel-level reconstruction loss, perceptual loss, style loss and total variation loss. These losses work together to enable the network to not only generate correct images at the pixel level when repairing images, but also maintain consistency in high-level features and textures, and the generated repair results are more realistic and natural.

[0050] Finally, the embodiments of the present application can train the marker removal model through the standard expanded marker removal data set and the second objective function. The training process aims to minimize the value of the objective function. The trained marker removal model is obtained according to the final weight parameters of the discriminator and the generator, and the trained marker removal model is used as the inpainting model.

[0051] In step S103, an end-to-end label removal operation is performed on the target fetal ultrasound image with labels based on the prediction model and the inpainting model.

[0052] Furthermore, the embodiments of the present application can combine two deep neural network models to achieve end-to-end fetal ultrasound image label removal.

[0053] During the actual implementation process, the embodiment of the present application can use computer languages ​​such as Python and utilize the Pytorch development framework to implement the above-mentioned fetal ultrasound image de-labeling operation based on deep learning of image channel splitting.

[0054] Therefore, the embodiments of the present application have the characteristics of high adaptability, can effectively solve the problem of difficult identification and removal of markers in fetal ultrasound images, and can be easily expanded to the task of marker removal of other medical ultrasound images.

[0055] Optionally, in one embodiment of the present application, an end-to-end label removal operation is performed on the target fetal ultrasound image with labels based on the prediction model and the inpainting model, including: combining the prediction model and the inpainting model to construct a final label removal network model; using the final label removal network model to perform an end-to-end label removal operation on the target fetal ultrasound image with labels to obtain the target fetal ultrasound image after the labels are removed.

[0056] It should be noted that the embodiment of the present application combines the two networks of the prediction model and the inpainting model to obtain the final label removal network model, so that when the fetal ultrasound image to be labeled is input (i.e., the target fetal ultrasound image with labels), a label removal operation is performed on it, thereby obtaining the fetal ultrasound image after the labels are removed.

[0057] In the process of specifically combining the two models, the mask image used by the infill model of each channel in the embodiment of the present application is a combination of the mask images generated by the marker feature extraction models of the three channels, so as to more comprehensively cover the various marker features included.

[0058] Therefore, the embodiments of the present application take into account the problems of different resolutions, marking styles, marking colors and image quality of ultrasound images among different devices, and achieve the unified style of ultrasound images through channel splitting, thereby improving the versatility and robustness of the marking removal technology, thereby realizing end-to-end ultrasound image marking extraction and inpainting, and significantly improving the automation and intelligence of marking removal.

[0059] According to the fetal ultrasound image de-labeling method based on image channel splitting proposed in the embodiment of the present application, a plurality of fetal ultrasound images are acquired, and a marker feature extraction dataset is constructed based on the plurality of fetal ultrasound images, and the dataset is expanded according to the channel splitting technology, and the pre-constructed marker feature extraction model and marker feature infilling model are trained using the dataset expanded by the channel splitting technology. Based on the two deep learning models of marker feature extraction and marker feature infilling, the channel splitting technology is used to perform multi-dimensional feature extraction and infilling, thereby realizing end-to-end marker removal operation. The present application realizes the style unification of ultrasound images through channel splitting, thereby improving the versatility and robustness of the marker removal technology, realizing end-to-end ultrasound image marker extraction and infilling, thereby significantly improving the degree of automation and intelligence of marker removal.

[0060] Secondly, the image channel splitting based fetal ultrasound image de-labeling device according to the embodiment of the present application is described with reference to the accompanying drawings.

[0061] Figure 4 is a block schematic diagram of the image channel splitting based fetal ultrasound image de-labeling device according to the embodiment of the present application.

[0062] As shown in Figure 4 , the image channel splitting based fetal ultrasound image de-labeling device 10 comprises a label feature extraction module 100, a label feature filling module 200 and an integration processing module 300.

[0063] The label feature extraction module 100 is configured to acquire a plurality of fetal ultrasound images, construct a label feature extraction dataset according to the plurality of fetal ultrasound images, expand the label feature extraction dataset by using a preset image channel splitting strategy, and train a pre-constructed label feature extraction model based on the expanded label feature extraction dataset to obtain a prediction model corresponding to the label feature extraction model.

[0064] The label feature filling module 200 is configured to construct a label removal dataset according to the plurality of fetal ultrasound images, expand the label removal dataset by using the image channel splitting strategy, and train a pre-constructed label removal model based on the expanded label removal dataset to obtain a filling model corresponding to the label removal model.

[0065] The integration processing module 300 is configured to perform end-to-end label removal operation on a target fetal ultrasound image with label based on the prediction model and the filling model.

[0066] Optionally, in an embodiment of the present application, the label feature extraction module 100 comprises a first acquisition unit, an extraction unit, an adjustment unit, a conversion unit, a channel splitting unit, a first determination unit and a training unit.

[0067] The first acquisition unit is configured to acquire a plurality of fetal ultrasound images with label from the plurality of fetal ultrasound images, and construct a label feature extraction dataset by using the plurality of fetal ultrasound images with label.

[0068] The extraction unit is configured to extract the bounding box text information corresponding to each fetal ultrasound image with label in the label feature extraction dataset, and fill the missing part in each fetal ultrasound image with label after extracting the bounding box text information based on a preset image inpainting strategy to obtain a fetal ultrasound image with label filled.

[0069] The adjustment unit is configured to adjust the image size of each fetal ultrasound image with label filled to obtain a fetal ultrasound image with label filled with a target size.

[0070] The conversion unit is used to randomly add multiple types of marks of the same color to the filled image with the mark, and convert the filled image with the mark to the HSV space to extract the mark mask corresponding to the filled image with the mark.

[0071] The channel splitting unit is used to perform channel splitting on the padded image with markers and copy the marker mask to obtain a standard expanded marker feature extraction dataset corresponding to the marker feature extraction dataset.

[0072] The first determining unit is used to construct a marker feature extraction model based on a preset encoder, decoder and U-Net framework, and determine a first objective function corresponding to the marker feature extraction model.

[0073] The training unit is used to train the marker feature extraction model using a standard expanded marker feature extraction data set and a first objective function to obtain a prediction model.

[0074] Optionally, in one embodiment of the present application, the marking feature in-painting module 200 includes: a second acquiring unit, a establishing unit, and a second determining unit.

[0075] Among them, the second acquisition unit is used to acquire multiple marked fetal ultrasound images from multiple fetal ultrasound images, and acquire multiple unmarked fetal ultrasound images corresponding to the multiple marked fetal ultrasound images from the multiple fetal ultrasound images, so as to construct multiple fetal ultrasound image sample pairs based on the multiple marked fetal ultrasound images and the multiple unmarked fetal ultrasound images.

[0076] An establishing unit is used to establish a label removal dataset based on multiple fetal ultrasound image sample pairs, and perform preprocessing and channel splitting operations on the labeled fetal ultrasound image and the unlabeled fetal ultrasound image in each fetal ultrasound image sample pair in the label removal dataset to obtain a standard extended label removal dataset corresponding to the label removal dataset.

[0077] The second determining unit is configured to determine a second objective function corresponding to the marker removal model, so as to train the marker removal model according to the standard expanded marker removal dataset and the second objective function to obtain an inpainting model.

[0078] Optionally, in one embodiment of the present application, the integration processing module 300 includes: a combining unit and a processing unit, wherein the combining unit is configured to combine the prediction model and the inpainting model to construct a final label removal network model.

[0079] The processing unit is configured to perform an end-to-end label removal operation on the target fetal ultrasound image with labels using the final label removal network model to obtain the target fetal ultrasound image after label removal.

[0080] It should be noted that the above explanation of the embodiment of the fetal ultrasound image de-labeling method based on image channel splitting is also applicable to the fetal ultrasound image de-labeling device based on image channel splitting in this embodiment, and will not be repeated here.

[0081] According to the fetal ultrasound image de-labeling device based on image channel splitting proposed in the embodiment of the present application, a plurality of fetal ultrasound images are acquired, and a marker feature extraction data set is constructed based on the plurality of fetal ultrasound images, and the data set is expanded according to the channel splitting technology, and the pre-constructed marker feature extraction model and marker feature infilling model are trained using the data set expanded by the channel splitting technology. Based on the two deep learning models of marker feature extraction and marker feature infilling, the channel splitting technology is used to perform multi-dimensional feature extraction and infilling, thereby realizing end-to-end marker removal operation. The present application realizes the style unification of ultrasound images through channel splitting, thereby improving the versatility and robustness of the marker removal technology, realizing end-to-end ultrasound image marker extraction and infilling, thereby significantly improving the degree of automation and intelligence of marker removal.

[0082] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0083] Memory 501 , processor 502 , and computer programs stored in the memory 501 and executable on the processor 502 .

[0084] When the processor 502 executes the program, the fetal ultrasound image de-labeling method based on image channel splitting provided in the above embodiment is implemented.

[0085] Furthermore, the electronic device further includes:

[0086] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0087] The memory 501 is used to store computer programs that can be run on the processor 502 .

[0088] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0089] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected with each other through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 5 Only one thick line is used to represent the bus in the figure, but it does not mean that there is only one bus or only one type of bus.

[0090] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can complete communication between each other through an internal interface.

[0091] The processor 502 can be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0092] The embodiments of the present application further provide a computer readable storage medium, having stored thereon a computer program, which is executed by a processor to implement the above fetal ultrasound image unlabeling method based on image channel splitting.

[0093] The embodiments of the present application further provide a computer program product, comprising a computer program, which is executed to implement the above fetal ultrasound image unlabeling method based on image channel splitting.

[0094] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0095] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0096] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing a custom logical function or process step, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed in a different order than shown or discussed, including performing functions in a substantially simultaneous manner or in a reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0097] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or N wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program can be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing it in other suitable ways as necessary, and then storing it in a computer memory.

[0098] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0099] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0100] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0101] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A fetal ultrasound image de-labeling method based on image channel splitting, characterized in that: The following steps are involved: Acquiring a plurality of fetal ultrasound images, constructing a marker feature extraction dataset based on the plurality of fetal ultrasound images, expanding the marker feature extraction dataset using a preset image channel splitting strategy, and training a pre-constructed marker feature extraction model based on the expanded marker feature extraction dataset to obtain a prediction model corresponding to the marker feature extraction model; Constructing a marker removal dataset based on the multiple fetal ultrasound images, expanding the marker removal dataset using the image channel splitting strategy, and training a pre-constructed marker removal model based on the expanded marker removal dataset to obtain an inpainting model corresponding to the marker removal model; Based on the prediction model and the inpainting model, an end-to-end label removal operation is performed on the target fetal ultrasound image with the label; The method of acquiring a plurality of fetal ultrasound images, constructing a marker feature extraction dataset based on the plurality of fetal ultrasound images, expanding the marker feature extraction dataset using a preset image channel splitting strategy, and training a pre-constructed marker feature extraction model based on the expanded marker feature extraction dataset to obtain a prediction model corresponding to the marker feature extraction model includes: Acquire a plurality of fetal ultrasound images with labels from the plurality of fetal ultrasound images, and construct the label feature extraction dataset using the plurality of fetal ultrasound images with labels; Extracting the frame text information corresponding to each marked fetal ultrasound image in the mark feature extraction data set, and filling the missing part of each marked fetal ultrasound image after extracting the frame text information based on a preset image restoration strategy to obtain a marked filled image; Adjusting the image size of each marked padded image to obtain a marked padded image of a target size; Randomly adding multiple types of marks of the same color to the marked filled image, and converting the marked filled image to HSV space to extract a mark mask corresponding to the marked filled image; Performing channel splitting on the marked filled image and copying the mark mask to obtain a standard expanded mark feature extraction dataset corresponding to the mark feature extraction dataset; Based on the preset encoder, decoder and U-Net framework, construct the marker feature extraction model, and determine the first objective function corresponding to the marker feature extraction model; The marker feature extraction model is trained using the standard expanded marker feature extraction data set and the first objective function to obtain the prediction model.

2. The method according to claim 1, characterized in that The step of constructing a marker removal dataset based on the plurality of fetal ultrasound images, expanding the marker removal dataset using the image channel splitting strategy, and training a pre-constructed marker removal model based on the expanded marker removal dataset to obtain an inpainting model corresponding to the marker removal model includes: Acquire a plurality of fetal ultrasound images with markers from the plurality of fetal ultrasound images, and acquire a plurality of fetal ultrasound images without markers corresponding to the plurality of fetal ultrasound images with markers from the plurality of fetal ultrasound images, so as to construct a plurality of fetal ultrasound image sample pairs based on the plurality of fetal ultrasound images with markers and the plurality of fetal ultrasound images without markers; Establishing the label-removed dataset based on the multiple fetal ultrasound image sample pairs, and performing preprocessing and channel splitting operations on the fetal ultrasound image with labels and the fetal ultrasound image without labels in each fetal ultrasound image sample pair in the label-removed dataset to obtain a standard extended label-removed dataset corresponding to the label-removed dataset; A second objective function corresponding to the marker removal model is determined, and the marker removal model is trained according to the standard expanded marker removal dataset and the second objective function to obtain the inpainting model.

3. The method according to claim 2, characterized in that The end-to-end label removal operation is performed on the target fetal ultrasound image with labels based on the prediction model and the inpainting model, including: Combining the prediction model and the inpainting model to construct a final label removal network model; The final label removal network model is used to perform a label removal operation on the target fetal ultrasound image with labels to obtain the target fetal ultrasound image after label removal.

4. A fetal ultrasound image de-labeling device based on image channel splitting, characterized in that: include: a marker feature extraction module, configured to acquire a plurality of fetal ultrasound images, construct a marker feature extraction dataset based on the plurality of fetal ultrasound images, expand the marker feature extraction dataset using a preset image channel splitting strategy, and train a pre-constructed marker feature extraction model based on the expanded marker feature extraction dataset to obtain a prediction model corresponding to the marker feature extraction model; a marker feature inpainting module, configured to construct a marker-removed dataset based on the plurality of fetal ultrasound images, expand the marker-removed dataset using the image channel splitting strategy, and train a pre-constructed marker-removal model based on the expanded marker-removed dataset to obtain an inpainting model corresponding to the marker-removed model; an integrated processing module, configured to perform an end-to-end label removal operation on the target fetal ultrasound image with the label based on the prediction model and the inpainting model; Wherein, the marker feature extraction module includes: a first acquiring unit, configured to acquire a plurality of fetal ultrasound images with labels from the plurality of fetal ultrasound images, and construct the label feature extraction dataset using the plurality of fetal ultrasound images with labels; an extraction unit, configured to extract the frame text information corresponding to each marked fetal ultrasound image in the mark feature extraction data set, and fill in the missing portion of each marked fetal ultrasound image after the frame text information is extracted based on a preset image restoration strategy to obtain a marked filled image; an adjusting unit, configured to adjust the image size of each marked filling image to obtain a marked filling image of a target size; a conversion unit, configured to randomly add multiple types of marks of the same color to the marked fill image, and convert the marked fill image into an HSV space to extract a mark mask corresponding to the marked fill image; a channel splitting unit, configured to perform channel splitting on the marked filled image and copy the mark mask to obtain a standard expanded mark feature extraction dataset corresponding to the mark feature extraction dataset; A first determining unit is configured to construct the marker feature extraction model based on a preset encoder, decoder, and U-Net framework, and determine a first objective function corresponding to the marker feature extraction model; A training unit is used to train the marker feature extraction model using the standard expanded marker feature extraction data set and the first objective function to obtain the prediction model.

5. The device according to claim 4, characterized in that The marking feature inpainting module includes: a second acquiring unit, configured to acquire a plurality of fetal ultrasound images with markers from the plurality of fetal ultrasound images, and acquire a plurality of fetal ultrasound images without markers corresponding to the plurality of fetal ultrasound images with markers from the plurality of fetal ultrasound images, so as to construct a plurality of fetal ultrasound image sample pairs according to the plurality of fetal ultrasound images with markers and the plurality of fetal ultrasound images without markers; an establishing unit, configured to establish the label-removed dataset based on the plurality of fetal ultrasound image sample pairs, and perform preprocessing and channel splitting operations on the fetal ultrasound image with labels and the fetal ultrasound image without labels in each fetal ultrasound image sample pair in the label-removed dataset, respectively, to obtain a standard extended label-removed dataset corresponding to the label-removed dataset; The second determining unit is configured to determine a second objective function corresponding to the marker removal model, so as to train the marker removal model according to the standard expanded marker removal dataset and the second objective function to obtain the inpainting model.

6. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the fetal ultrasound image de-labeling method based on image channel splitting according to any one of claims 1 to 3.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the fetal ultrasound image de-labeling method based on image channel splitting as described in any one of claims 1 to 3.

8. A computer program product comprising a computer program, characterized in that The computer program is executed to implement the fetal ultrasound image de-labeling method based on image channel splitting according to any one of claims 1 to 3.

Images