Method and device for removing artificial mark in ultrasonic image and readable storage medium thereof

Abstract

The invention provides a method and device for removing an artificial mark in an ultrasonic image and a readable storage medium thereof.The method comprises the steps that thyroid ultrasonic images are collected from multiple hospitals, a data set is processed, then the data set is input into a Restormer model, the artificial mark is removed through an encoding-decoding architecture and MDTA and GDFN modules of the Restormer model, a progressive learning strategy is adopted for training and optimization, and the artificial mark is obtained. And finally, carrying out classification, segmentation and target detection task testing on the marked data set. The method does not depend on manual marking of the mask, can adaptively learn, recognize and remove the mark, and reserve image details and structure information, effectively solves many problems when an existing method processes the manual mark of the ultrasonic image, and remarkably improves the performance of a deep learning model in related tasks.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for removing artificial marks in ultrasound images based on a Restormer model, and a readable storage medium thereof. Background Art

[0002] Ultrasound examinations are widely used in medical diagnosis. In order to accurately indicate the location of the lesion, doctors will add artificial marks (such as crosshairs, numbers, and letters) to ultrasound images, which is very helpful for traditional diagnosis. However, with the in-depth application of deep learning in the field of medical image processing, these artificial marks have become a new problem. Deep learning models rely on a large amount of labeled data training to identify lesion features, and artificial marks in ultrasound images are often mistaken by the model as part of the lesion, resulting in deviations in feature extraction and learning, affecting the accuracy and robustness of the model in classification, segmentation, and detection tasks. Therefore, it is urgent to remove artificial marks from ultrasound images.

[0003] At present, there are mainly the following methods for removing artificial labels from ultrasound images: 1. Traditional image restoration algorithms: such as the Criminisi algorithm and partial convolution technology, require manual annotation of masks to define the restoration area, which is cumbersome and time-consuming, and difficult to apply on a large scale. Moreover, when processing complex background images, it is easy to lose details and texture information, resulting in poor image quality after restoration.

[0004] 2. Deep learning methods based on convolutional neural networks (CNNs): such as the U-Net-based image restoration network, which can automatically remove labels, but the CNN model has a fixed receptive field and is difficult to capture long-distance dependency information. It is not effective when dealing with complex backgrounds or labels with variable shapes, and most of them still require manual annotation of masks, which is greatly affected by the quality of the masks.

[0005] 3. Blind repair method: This method does not require manual masking and can automatically detect and repair damaged areas. However, the existing blind repair algorithms have problems with inaccurate positioning and poor repair effects when dealing with diverse markers. They are easily disturbed in complex backgrounds and the repair results are unstable.

[0006] 4. Transformer-based image restoration method: The Transformer model can capture long-distance image dependencies through the self-attention mechanism, and demonstrates strong feature modeling capabilities in image restoration tasks. It has great potential for handling the task of removing artificial labels from ultrasound images.

[0007] Therefore, there is an urgent need for a method and device for removing artificial marks in ultrasound images based on a Restormer model and a readable storage medium thereof to solve the problems existing in the prior art. Summary of the invention

[0008] The embodiments of the present application provide a method, device and readable storage medium for removing artificial marks in ultrasound images based on a Restormer model. The methods for removing artificial marks in ultrasound images that exist in the current technology, including traditional image restoration algorithms, deep learning methods based on convolutional neural networks, blind restoration methods, etc., have the problems of relying on manually labeled masks, poor processing effects of complex marks, susceptibility to background interference, inaccurate positioning, difficulty in simultaneously maintaining image details and removing marks, and incomplete restoration, which affect the accuracy and performance of deep learning models in related tasks.

[0009] The core technology of this invention is mainly based on the Restormer model, combining convolutional neural networks with self-attention mechanisms, adopting an encoding-decoding architecture and multi-dimensional convolutional head transposed attention and gated deep convolutional feedforward network modules to automatically detect and repair artificial markers in ultrasound images and improve the performance of deep learning models.

[0010] In a first aspect, the present application provides a method for removing artificial marks in an ultrasound image, the method comprising the following steps: S1. Collect thyroid ultrasound image data as a data set and perform preprocessing; Wherein, the data set includes samples with artificial labels and samples without labels; S2. Input the preprocessed data set into the Restormer model, extract multi-scale features and reduce the resolution through the encoder of the Restormer model, and then restore the high-resolution features through the decoder of the Restormer model; S3, adjust the attention weight through the multi-dimensional convolutional head transposed attention module of the Restormer model, and filter and retain key features and remove artificial labels through the gated deep convolutional feedforward network module of the Restormer model; S4, using progressive learning strategy to train the Restormer model in stages; S5. Perform multi-task tests on the dataset after removing manual labels, including classification, segmentation, and object detection tasks, to comprehensively evaluate the model performance improvement.

[0011] Furthermore, in step S1, preprocessing includes cleaning and screening the collected thyroid ultrasound image data, removing blurry or low-quality samples, and then expanding the data set using data enhancement technology.

[0012] Furthermore, in step S2, the encoder extracts multi-scale features of the dataset through multi-layer convolution operations and self-attention mechanism, and reduces the image resolution. The decoder restores high-resolution features through pixel reconstruction and skip connection mechanism, retaining image detail information.

[0013] Furthermore, in step S3, the multi-dimensional convolutional head transposed attention module adjusts the attention weights between channels through dynamic convolution to adaptively capture the global and local features in the ultrasound image. The gated deep convolutional feedforward network module introduces a gating mechanism to filter and retain key feature information to effectively remove artificial labels.

[0014] Furthermore, in step S4, a progressive learning strategy is adopted to divide the training process into multiple stages, gradually adjust the patch size and batch size of the input image, and use the AdamW optimizer to update the weights. The L1 loss function is used as the optimization target during the training process, and the peak signal-to-noise ratio and structural similarity index are used to evaluate the quality of the de-labeled image.

[0015] Furthermore, in step S4, during the training process, the initial learning rate of the AdamW optimizer is set to 0.00125, and the weight decay coefficient is set to 0.001.

[0016] Furthermore, in step S5, the classification task uses a visual converter to classify benign and malignant thyroid nodules to evaluate the effect of improving the classification accuracy of the unlabeled data; The segmentation task uses the Unet model to accurately segment the nodule area and verify the improvement of the segmentation accuracy of the lesion area by the de-labeled image; The target detection task detects the nodule position based on the YOLOv8 architecture to evaluate the enhancement effect of the de-labeling process on the target detection performance.

[0017] In a second aspect, the present application provides a device for removing artificial marks in ultrasound images based on a Restormer model, comprising: An acquisition module collects thyroid ultrasound image data as a data set and performs preprocessing; Among them, the dataset includes samples with manual labels and unlabeled samples; The feature extraction module inputs the preprocessed data set into the Restormer model, extracts multi-scale features and reduces the resolution through the encoder of the Restormer model, and then restores the high-resolution features through the decoder of the Restormer model; Remove the manual labeling module, adjust the attention weight through the multi-dimensional convolutional head transposed attention module of the Restormer model, and filter and retain key features through the gated deep convolutional feedforward network module of the Restormer model to remove manual labels; The training module uses a progressive learning strategy to train the Restormer model in stages; The evaluation module conducts multi-task testing on the dataset after removing manual labels, including classification, segmentation, and object detection tasks, to comprehensively evaluate the model performance improvement.

[0018] In a third aspect, the present application provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the above-mentioned method for removing artificial markers in ultrasound images.

[0019] In a fourth aspect, the present application provides a readable storage medium, in which a computer program is stored. The computer program includes a program code for controlling a process to execute the process. The process includes the method for removing artificial markers in ultrasound images according to the above-mentioned method.

[0020] The main contributions and innovations of the present invention are as follows: 1. Maskless blind repair to improve efficiency and accuracy The present invention proposes a maskless blind repair method based on the Restormer model, which does not require manual mask annotation, and recognizes and removes manual marks through model adaptive learning. This avoids the tedious operation of manually drawing masks in traditional methods, saves a lot of time and labor costs, and effectively reduces the errors caused by manual annotation. For example, traditional methods such as the Criminisi algorithm, which manually annotates masks, are inefficient when processing a large number of ultrasound images, while the present invention can achieve automated processing and significantly improve the repair efficiency.

[0021] The model can accurately identify and process artificial markers with complex shapes and random positions, effectively overcoming the limitations of existing methods in dealing with diverse markers, improving the accuracy of marker removal, and making the restored image closer to the real unmarked image.

[0022] 2. Excellent image detail preservation and restoration effects The encoding-decoding structure of the Restormer model combined with the MDTA and GDFN modules can efficiently capture local and global features of the image. While removing artificial markers, it can well preserve the details and texture information of the image and effectively restore the image content of the marked area. Compared with traditional image restoration methods, some traditional algorithms are prone to losing details when processing complex background ultrasound images. The present invention can still maintain superior restoration effects under complex backgrounds and multiple markers, ensuring the quality of the restored image and providing a more reliable basis for subsequent diagnosis.

[0023] 3. Significantly improve the performance of deep learning models By removing artificial labels from ultrasound images, the interference of labels on deep learning models (such as classification, segmentation and target detection models) is effectively eliminated. Experiments show that the accuracy rate in classification tasks is improved by 27.38%, the IoU in segmentation tasks is improved by 45.33%, and the mAP50 and mAP50-95 in target detection tasks are improved by 0.644 and 0.451, respectively. This enables the model to focus more accurately on the characteristics of real lesions, thereby significantly improving the overall performance of the model in various tasks, improving the accuracy and reliability of disease diagnosis, and contributing to the intelligent development of medical image processing.

[0024] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 is a flow chart of a method for removing artificial marks in an ultrasound image according to an embodiment of the present application; Figure 2 It is a qualitative result diagram of removing four different types of artificial marks according to the algorithm proposed in the embodiment of the present application; Figure 3 It is a qualitative comparison diagram of the training and test results of the UDer and UDRer data sets using the unet segmentation model according to the embodiment of the present application; Figure 4 It is a qualitative comparison diagram of the training and test results of the UDer and UDRer data sets using the YOLOv8 target detection model according to the embodiment of the present application; Figure 5 It is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0027] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0028] Currently, a variety of methods have been used to remove artificial markers from ultrasound images, mainly including deep learning algorithms based on traditional image restoration technology and convolutional neural networks. However, these methods still have many limitations in practical applications.

[0029] Based on this, the present invention solves the problems existing in the prior art based on the Restormer model.

[0030] Embodiment 1 The present application aims to propose a method for removing artificial labels from ultrasound images based on the Restormer model. The Restormer model combines a convolutional neural network with a self-attention mechanism and adopts an encoding-decoding architecture, which can effectively remove artificial labels from ultrasound images while maintaining the details and structural information of the image. This method does not require manual annotation of masks, automatically detects and repairs marked areas, and has higher robustness and practicality. Compared with traditional methods, the present invention has better de-labeling effect in complex backgrounds, significantly improving the clarity of ultrasound images and the diagnostic accuracy of deep learning models.

[0031] For the convenience of description and analysis, the present invention abbreviates the dataset containing artificial markers in thyroid ultrasound images as "UDer" (Ultrasound Images Dataset with Artificial Marker), and the dataset with artificial markers removed after being processed by the algorithm of the present invention is abbreviated as "UDRer" (Ultrasound Images Dataset with Removed Artificial Marker). In the following analysis, the present invention will use these abbreviations to represent the corresponding datasets to simplify the description and ensure the coherence and readability of the content.

[0032] Specifically, the embodiment of the present application provides a method for removing artificial marks in an ultrasound image. Specifically, referring to Figure 1 , the method comprises the following steps: S1: A large amount of thyroid ultrasound image data was collected from multiple hospitals, including ultrasound image samples with manual labels and without labels. In order to ensure the consistency and high quality of the dataset, the collected image data was cleaned and screened to remove blurry or low-quality samples. Then, in order to increase the diversity of the data and improve the generalization ability of the model, data augmentation techniques such as rotation, scaling, and flipping were used to expand the dataset.

[0033] S2: The preprocessed ultrasound image is input into the Restormer model, which adopts an encoder-decoder structure, including a four-level symmetrical encoder and decoder. In the encoder part, multi-layer convolution operations and self-attention mechanisms are used to extract multi-scale features of ultrasound images, and the image resolution is gradually reduced to enhance the feature representation capability. The decoder part restores high-resolution features through pixel-shuffle and skip connections, while retaining the image's detailed information, ensuring that the unlabeled image maintains its original clarity and integrity.

[0034] S3: In the process of removing artificial labels, the Multi-Dimensional Convolutional Head Transposed Attention (MDTA) module of the Restormer model adjusts the attention weights between channels through dynamic convolution to adaptively capture global and local features in ultrasound images. This not only reduces the computational complexity, but also improves the accuracy and effect of label removal. In addition, the Gated Deep Convolutional Feedforward Network (GDFN) module introduces a gating mechanism to ensure that the model can maintain high fidelity when dealing with complex backgrounds by selectively screening and retaining key feature information, effectively removing interference without losing key details of the image.

[0035] S4: In order to improve the learning effect and stability of the model, a progressive learning strategy is used to divide the training process into multiple stages, and the patch size and batch size of the input image are gradually adjusted to enable the model to better learn the global and local features in ultrasound images. During the model training process, the AdamW optimizer is used to update the weights, and the initial learning rate is set to 0.00125 and the weight decay coefficient is 0.001 to prevent overfitting and improve the generalization ability of the model. To ensure the high efficiency of the model for image restoration, the L1 loss function (L1 Loss) is used as the optimization target during the training process, and the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are used to evaluate the quality of the de-labeled image to verify the restoration effect of the model.

[0036] Among them, the AdamW optimizer is an optimization algorithm widely used in deep learning, which is used to update the parameters of the model to minimize the loss function. It is an improvement on the Adam optimizer. The AdamW optimizer combines the adaptive learning rate method (AdaptiveLearningRateMethods) and the weight decay (WeightDecay) technology, and has good performance when training neural networks. The L1 loss function (L1Loss), also known as the mean absolute error (MeanAbsoluteError, MAE), is a loss function used in machine learning and deep learning to measure the difference between the model prediction value and the true value. Peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are two important indicators for evaluating image quality in the field of image processing. They measure the similarity between the processed image (in the present invention, the ultrasound image after removing artificial labels) and the original image or reference image from different angles.

[0037] Among them, setting the initial learning rate to 0.00125 means that at the beginning of training, the model parameters will be updated with a relatively small step size. A smaller initial learning rate helps the model converge more stably in the early stage of training, avoiding skipping the minimum point of the loss function due to too large a step size, which causes the model to fail to converge to the optimal solution. During the training process, the learning rate is usually adjusted according to certain strategies (such as learning rate decay) to more finely adjust the model parameters in the later stage of training. Setting the weight decay coefficient to 0.001 indicates that the penalty for the model weight is moderate. In the ultrasound image artificial label removal method based on the Restormer model, since the data set may have a certain complexity (such as differences in images collected by different hospitals, the characteristics of the ultrasound image itself, etc.), appropriate weight decay helps to improve the generalization ability of the model, so that the model can better handle unseen data and avoid overfitting problems that perform well on the training set but degrade in the test set or actual application.

[0038] Therefore, the reasonable setting of these two parameters of the AdamW optimizer is to balance the convergence speed, stability and generalization ability during the model training process, so that the model can more effectively learn to remove artificial labels in ultrasound images and show good performance in subsequent classification, segmentation and target detection tasks. These parameter settings cooperate with factors such as model structure, data characteristics and training strategy to jointly affect the training effect and final performance of the model.

[0039] S5: Multi-task testing is performed on the dataset after removing the manual labels to comprehensively evaluate the performance improvement of the model, including three major tasks: classification, segmentation, and target detection. In the classification task, Vision Transformer (ViT) is used to classify thyroid nodules into benign and malignant ones to evaluate the improvement of classification accuracy of the unlabeled data. In the segmentation task, the Unet model is used to accurately segment the nodule area to verify the improvement of the segmentation accuracy of the lesion area by the unlabeled image. In the target detection task, the nodule position is detected based on the YOLOv8 architecture to evaluate the enhancement effect of the unlabeling process on the target detection performance. This series of tests verifies that the processing method of removing manual labels can significantly improve the overall performance of deep learning models in various tasks.

[0040] Among them, Vision Transformer (ViT) is a neural network based on the Transformer architecture for computer vision tasks. It divides the image into small patches of fixed size, each patch is regarded as an independent input unit, converted into a vector through linear transformation, and then position encoding is added to introduce position information, and then these vectors are input into the Transformer encoder. Its core is to use the self-attention mechanism to learn the relationship between different patches in the image, so as to capture global features. Advantages include the ability to learn global features and greater scalability.

[0041] Unet model: A classic convolutional neural network architecture designed for biomedical image segmentation. It has an encoder-decoder symmetric structure. The encoder gradually extracts high-level features through a series of convolution and downsampling operations, and the decoder gradually restores the original resolution through upsampling. The corresponding features in the encoder are spliced ​​with the decoder using jump connections to combine low-level detail information and high-level semantic information to achieve accurate pixel-level segmentation.

[0042] YOLOv8 architecture: It is an advanced target detection algorithm network structure. It uses an end-to-end architecture to predict the category and location of the target directly from the input image. Its backbone network is used to extract image features, the neck network fuses and processes the features extracted by the backbone network, and the detection head predicts the target category and bounding box location based on the processed features. The innovations of YOLOv8 include using the composite scaling method of the EfficientNet architecture to improve feature extraction capabilities; integrating the Squeeze-and-Excitation (SE) module to enhance the representation of useful information; integrating CrossStagePartialNetwork (CSPNet) to reduce computational complexity; realizing multi-scale feature fusion through the Feature Pyramid Network (FPN) and adopting an adaptive feature fusion strategy; in addition, it may also use technologies such as deep separable convolution and new activation functions.

[0043] Preferably, in order to verify the feasibility of the present invention, the present invention is compared with the existing work: During the experiment, the present invention randomly selected 20% of the original data set as a test set and input it into the trained network for evaluation. Figure 2 The qualitative results presented in Figure 2 show that the proposed algorithm performs well in removing four different types of artificial labels. This shows that the algorithm is not only able to effectively identify and remove a variety of artificial labels, but also has strong robustness and versatility.

[0044] In order to further evaluate the performance of the algorithm of the present invention, the present invention compares and analyzes it with several existing algorithms for removing artificial labels. The quantitative evaluation indicators used include peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) to evaluate the effect of removing artificial labels. According to the results in Table 1: Table 1

[0045] Although the performance of each algorithm is relatively close in terms of SSIM between images with and without artificial labels, this is because artificial labels have little effect on the overall image structure, so the SSIM values ​​of each algorithm are not significantly different. However, in terms of PSNR, the method of the present invention is significantly better than the existing algorithms [1, 2, 3], reaching 64.65. This shows that the algorithm of the present invention can more effectively restore the details and texture information of the image, so that the restored image has a higher consistency with the original image.

[0046] It is worth noting that the significant improvement in PSNR suggests that the method of the present invention has obvious advantages in reducing noise and image distortion. This superiority stems from the algorithm's more sophisticated feature extraction and stronger denoising ability, which successfully removes the interference of artificial labels while retaining the real information of the image. This result not only verifies the effectiveness of the algorithm of the present invention, but also shows that it has a wider application potential in practical applications.

[0047] Among them, the algorithm 1 comes from YingX, ZhangY, YuM, et al. Cascade label removal algorithm for thyroid ultrasound images [J]. Journal of Medical and Biological Engineering and Computing, 2020, Vol. 58: 2641-2656.

[0048] The algorithm of 2 comes from SunM, MengQ, WangT, et al. Removal of artificially induced artifacts in ultrasound images of thyroid nodules based on edge connection and Criminisi image restoration algorithm [J]. Journal of Computer Methods and Biomedical Programs, 2021, Vol. 200.

[0049] The algorithm of 3 comes from GuoX, HuW, NiC, et al. Blind inpainting for removing artificial labels based on target-aware discrimination [C] / / Proceedings of the 2024 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). Institute of Electrical and Electronics Engineers, 2024: 1516-1520.

[0050] Preferably, a boost test on the classification task is also performed: At present, the application of convolutional neural networks in fields such as target detection and image segmentation is usually based on the architecture of classification networks. Therefore, verifying the impact of manual labeling on classification network training can indirectly prove the degree of influence of labeling on the deep learning structure based on classification networks. To this end, the present invention uses a UDer dataset containing manual labels and a UDRer dataset with labels removed by the algorithm of the present invention to perform a benign and malignant classification experiment of thyroid nodules on Vision Transformer to evaluate the impact of manual labeling on classification performance. The comparison of training and test results using the ViT classification model on the UDer and UDRer datasets is shown in Table 2: Table 2

[0051] As can be seen, Table 2 shows the performance of the classifier under different combinations of training data and test data. By comparing the performance of the classifiers trained with UDer and UDRer, it can be found that the classifier trained with UDer shows obvious instability in the test phase. Specifically, the test accuracy of ViT trained with UDer on the UDer dataset reached 92.85%, but this high accuracy does not mean that the model has effectively learned the characteristics of the thyroid nodule area. Further comparative analysis shows that the classification accuracy of the classifier trained with UDer on the UDer and UDRer datasets differs by as much as 27.38%. This significant difference indicates that the classifier trained with UDer is highly dependent on the manual markings in the ultrasound images during actual classification, and uses them as the key features for judging the benign and malignant nature of thyroid nodules. Therefore, when tested on the UDRer dataset, removing the manual markings caused the accuracy of the classifier to drop significantly. This means that the model has not really learned the imaging features of the nodules, but relies on manual markings for classification, which seriously affects the generalization ability of the model and the reliability of practical applications.

[0052] In view of the negative impact of manual labeling on deep learning models, the unlabeled dataset UDRer was constructed using the algorithm in this paper. The data of UDRer comes from the original ultrasound image dataset UDer containing manual labels. The two are completely consistent in data content and feature distribution, and only differ in whether they contain manual labels. The experimental results show that the classifier trained with UDRer shows stable performance regardless of whether the ultrasound image contains manual labels. The difference between the classification accuracy tested on UDer and UDRer is only 2.78%, which is much smaller than the model trained with UDer. The stability of the classifier performance shows that UDRer can effectively eliminate the negative impact of labeling on the classification network, thereby reflecting the actual learning effect of the classification network on thyroid ultrasound images.

[0053] Preferably, a test for the improvement of the segmentation task is also performed: Lesion segmentation is a key function in computer-aided detection systems, which can quickly locate the lesion area of ​​the patient, thereby assisting radiologists in diagnosis. In order to verify the negative impact of manual labeling on the segmentation task, the present invention selected the classic segmentation model Unet for experimentation. The Unet model is widely used in the field of medical image processing due to its efficient structure and good segmentation performance, and can effectively evaluate the impact of manual labeling on the quality of lesion segmentation. The quantitative comparison of the training and test results using the Unet segmentation model on the UDer and UDRer datasets is shown in Table 3 below: Table 3

[0054] Table 3 shows the performance of the segmentation model under different combinations of training data and test data. The test results of the segmentation model trained with UDer on UDer and UDRer show that the average pixel accuracy difference between the two is as high as 44.91%, and the average IOU decreases by 45.33%. These results clearly show that the segmentation model over-relies on manually labeled features and ignores the actual nodule features. Therefore, the model trained with UDer has difficulty in accurately completing the nodule segmentation task on images without manual labels. This result further reveals the misleading effect of manual labeling on the computer-aided detection system, causing the system to mistakenly regard the labeled features as nodule features, thereby affecting its segmentation performance on images without manual labeling.

[0055] For the segmentation model trained with UDRer, the difference in average pixel accuracy and average IoU is not large whether tested on UDer or UDRer. This shows that the segmentation model can effectively learn how to segment the lesion area without manual labeling. In addition, manual labeling as an obvious feature makes it easier for the model to obtain higher average pixel accuracy and average IoU on the dataset containing manual labeling. However, this does not mean that the segmentation model has truly learned the characteristics of the lesion, but because the model relies on the presence of markers.

[0056] like Figure 3 As shown in the figure, the training and testing results of the UDer and UDRer datasets were qualitatively compared using the unet segmentation model. It can be seen that the segmentation model trained with UDRer showed similar and excellent performance when testing UDer and UDRer, and the segmentation model trained with UDer showed effective segmentation capabilities for thyroid nodules. However, when the model trained with UDer was tested on UDRer, the segmentation effect dropped significantly, the probability of identifying thyroid nodules was greatly reduced, and the lesion area was almost impossible to identify. This phenomenon shows that the model trained with UDer relies too much on manual labeling and loses the ability to recognize nodule features in the test set without manual labeling, further verifying the negative impact of manual labeling on segmentation model training.

[0057] Preferably, an improvement test on the target detection task is also performed: Target detection has important applications in the auxiliary diagnosis of thyroid nodules, and is of significant significance for improving the work efficiency of radiologists and quickly locating the lesion area. However, there are the following challenges in thyroid ultrasound images: on the one hand, ultrasound images are usually of low resolution and poor quality, which puts higher requirements on the target detection model. The model needs to be able to extract implicit features from low-quality images; on the other hand, the size of thyroid nodules varies greatly, and the target detection model must have the ability to extract features from different scales and be able to focus on the overall features of the image. Based on these challenges, the present invention selects the YOLOv8 model to verify the negative impact of manual labeling on target detection. YOLOv8, with its ability to extract multi-scale features and its focus on overall features, has become an ideal choice for solving the above problems. By conducting experiments on the dataset after removing the manual labels, the present invention can evaluate the actual impact of manual labels on target detection performance. The quantitative comparison of the training and test results using the YOLOv8 target model on the UDer and UDRer datasets is shown in Table 4 below: Table 4

[0058] As shown in Table 4, the performance of the target detection models trained with different datasets on different test datasets is significantly different. For the target detection model trained with UDer, the test results on the UDer and UDRer datasets show that the difference in mAP50 (the value of the mean average precision at IoU=0.5) is 0.683, while the difference in mAP50-95 (the average value of the mean average precision at IoU from 0.5 to 0.95 with a step size of 0.05) is 0.516. The detection performance on the UDRer dataset is much lower than that on the UDer dataset. This experimental result clearly shows that the presence of manual labels has a significant impact on the training process of the target detection model. Specifically, the performance of the target detection model trained on UDer is significantly dependent on the manual labels rather than the actual nodule features. This results in the inability of the model to accurately locate the lesion area in practical applications, which seriously affects the effectiveness of the computer-aided detection system. Note: The test accuracy of the target detection model trained with UDRer on UDer here is slightly lower than that of the model trained with UDer. However, in the test on UDRer, its accuracy is significantly higher than that of the model trained by UDer, with mAP50 being 0.644 higher and mAP50-95 being 0.451 higher. This shows that the model trained using UDRer can perform well on test sets with or without manual labels. The results verify that the method of the present invention successfully eliminates the negative impact of manual labeling on model training, enabling the model to learn the true semantic information of nodules in thyroid ultrasound images, rather than relying solely on manual labeling.

[0059] Among them, mAP50 and mAP50-95 are key indicators for evaluating model performance in target detection tasks. They measure the accuracy of the model in predicting target location and category from different angles and with different degrees of rigor. In this paper, they are of great significance for evaluating the impact of manual labeling on target detection models and verifying the effectiveness of the de-labeling method.

[0060] like Figure 4 As shown in the figure, the training and testing results of the UDer and UDRer datasets are qualitatively compared using the YOLOv8 target detection model. It can be seen that the target detection model trained with UDRer shows good performance when testing UDer and UDRer, and the target detection model trained with UDer shows good performance when testing UDer, and can effectively locate thyroid nodules. However, when the model trained with UDer is tested on UDRer, the detection effect is significantly deteriorated, the recognition rate of nodules is significantly reduced, and sometimes nodules cannot even be located. This shows that the model trained with UDer is too dependent on manual labeling, which makes it unable to correctly identify nodule features on the test set without labels, further proving the adverse effects of manual labeling on the training of target detection models.

[0061] Embodiment 2 Based on the same concept, the present application also proposes a device for removing artificial marks in ultrasound images based on the Restormer model, comprising: An acquisition module collects thyroid ultrasound image data as a data set and performs preprocessing; Among them, the dataset includes samples with manual labels and unlabeled samples; The feature extraction module inputs the preprocessed data set into the Restormer model, extracts multi-scale features and reduces the resolution through the encoder of the Restormer model, and then restores the high-resolution features through the decoder of the Restormer model; Remove the manual labeling module, adjust the attention weight through the multi-dimensional convolutional head transposed attention module of the Restormer model, and filter and retain key features through the gated deep convolutional feedforward network module of the Restormer model to remove manual labels; The training module uses a progressive learning strategy to train the Restormer model in stages; The evaluation module conducts multi-task testing on the dataset after removing manual labels, including classification, segmentation, and object detection tasks, to comprehensively evaluate the model performance improvement.

[0062] Embodiment 3 This embodiment also provides an electronic device, referring to Figure 5, comprises a memory 404 and a processor 402, wherein the memory 404 stores a computer program, and the processor 402 is configured to run the computer program to execute the steps in any of the above method embodiments.

[0063] Specifically, the processor 402 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0064] Among them, the memory 404 may include a large capacity memory 404 for data or instructions. For example, but not limitation, the memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. In appropriate cases, the memory 404 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 404 may be inside or outside the data processing device. In a specific embodiment, the memory 404 is a non-volatile memory. In a specific embodiment, the memory 404 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable Programmable Read-Only Memory, EPROM for short), an electrically erasable PROM (Electrically Erasable Programmable Read-Only Memory, EEPROM for short), an electrically alterable ROM (Electrically Alterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. In appropriate circumstances, the RAM may be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM may be a fast page mode dynamic random access memory 404 (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0065] The memory 404 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 402 .

[0066] The processor 402 reads and executes the computer program instructions stored in the memory 404 to implement any method for removing artificial marks in ultrasound images in the above embodiments.

[0067] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408 , wherein the transmission device 406 is connected to the processor 402 , and the input / output device 408 is connected to the processor 402 .

[0068] The transmission device 406 can be used to receive or send data via a network. Specific examples of the above-mentioned network may include a wired or wireless network provided by a communication provider of the electronic device. In one example, the transmission device includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 406 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0069] Input / output devices 408 are used to input or output information.

[0070] Embodiment 4 This embodiment further provides a readable storage medium, in which a computer program is stored. The computer program includes a program code for controlling a process to execute the process. The process includes the method for removing artificial marks in an ultrasound image according to the first embodiment.

[0071] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0072] In general, various embodiments may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects of the invention may be implemented in hardware, while other aspects may be implemented in firmware or software executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flow charts, or using some other graphical representation, it should be understood that, as non-limiting examples, the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0073] Embodiments of the present invention may be implemented by computer software that is executable by a data processor of a mobile device, such as in a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets and / or macros may be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. A computer program product may include one or more computer executable components configured to perform an embodiment when the program is run. One or more computer executable components may be at least one software code or a portion thereof. In addition, at this point, it should be noted that, for example, Figure 1 Any block of the logic flow in the program may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on physical media such as memory chips or storage blocks implemented within the processor, magnetic media such as hard disk or floppy disks, and optical media such as, for example, DVDs and their data variants, CDs, etc. Physical media are non-transitory media.

[0074] Those skilled in the art should understand that the technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0075] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for removing artificial marks in an ultrasound image, characterized in that: The following steps are involved: S1, collect thyroid ultrasound image data as a data set and perform preprocessing; Wherein, the data set includes samples with artificial labels and samples without labels; S2. Input the preprocessed data set into the Restormer model, extract multi-scale features and reduce the resolution through the encoder of the Restormer model, and then restore the high-resolution features through the decoder of the Restormer model; S3, adjust the attention weight through the multi-dimensional convolutional head transposed attention module of the Restormer model, and filter and retain key features and remove artificial labels through the gated deep convolutional feedforward network module of the Restormer model; S4, using progressive learning strategy to train the Restormer model in stages; S5. Perform multi-task tests on the dataset after removing manual labels, including classification, segmentation, and object detection tasks, to comprehensively evaluate the model performance improvement.

2. The method for removing artificial marks in ultrasound images according to claim 1, characterized in that: In step S1, preprocessing includes cleaning and screening the collected thyroid ultrasound image data, removing blurry or low-quality samples, and then expanding the data set using data enhancement technology.

3. The method for removing artificial marks in ultrasound images according to claim 1, characterized in that: In step S2, the encoder extracts multi-scale features of the dataset through multi-layer convolution operations and self-attention mechanism and reduces the image resolution. The decoder restores high-resolution features through pixel reconstruction and skip connection mechanism to retain image detail information.

4. The method for removing artificial marks in ultrasound images according to claim 1, characterized in that: In step S3, the multi-dimensional convolutional head transposed attention module adjusts the attention weights between channels through dynamic convolution to adaptively capture the global and local features in the ultrasound image. The gated deep convolutional feedforward network module introduces a gating mechanism to filter and retain key feature information to effectively remove artificial labels.

5. The method for removing artificial marks in ultrasound images according to claim 1, characterized in that: In step S4, a progressive learning strategy is used to divide the training process into multiple stages, gradually adjusting the patch size and batch size of the input image, and using the AdamW optimizer to update the weights. The L1 loss function is used as the optimization target during training, and the peak signal-to-noise ratio and structural similarity index are used to evaluate the quality of the unlabeled image.

6. The method for removing artificial marks in ultrasound images according to claim 5, characterized in that: In step S4, during the training process, the initial learning rate of the AdamW optimizer was set to 0.00125 and the weight decay coefficient was set to 0.

001.

7. The method for removing artificial marks in an ultrasound image according to any one of claims 1 to 6, characterized in that: In step S5, the classification task uses a visual converter to classify benign and malignant thyroid nodules to evaluate the effect of unlabeled data on improving classification accuracy; The segmentation task uses the Unet model to accurately segment the nodule area and verify the improvement of the segmentation accuracy of the lesion area by the de-labeled image; The target detection task detects the nodule position based on the YOLOv8 architecture to evaluate the enhancement effect of de-labeling on target detection performance.

8. A device for removing artificial marks in ultrasound images based on the Restormer model, characterized in that: include: An acquisition module collects thyroid ultrasound image data as a data set and performs preprocessing; Among them, the dataset includes samples with manual labels and unlabeled samples; The feature extraction module inputs the preprocessed data set into the Restormer model, extracts multi-scale features and reduces the resolution through the encoder of the Restormer model, and then restores the high-resolution features through the decoder of the Restormer model; Remove the manual labeling module, adjust the attention weight through the multi-dimensional convolutional head transposed attention module of the Restormer model, and filter and retain key features through the gated deep convolutional feedforward network module of the Restormer model to remove manual labels; The training module uses a progressive learning strategy to train the Restormer model in stages; The evaluation module conducts multi-task testing on the dataset after removing manual labels, including classification, segmentation, and object detection tasks, to comprehensively evaluate the model performance improvement.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the method for removing artificial marks in an ultrasound image according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, wherein the computer program includes a program code for controlling a process to execute a process, wherein the process includes the method for removing artificial marks in an ultrasound image according to any one of claims 1 to 7.

Images