Image labeling method and device and electronic equipment

Abstract

The invention discloses an image annotation method and device and electronic equipment. The method comprises the following steps: determining the image quality of an original image; under the condition that the quality evaluation index of the image quality is larger than or equal to a preset threshold value and the original image is a two-dimensional image, the original image is analyzed and processed through a student model, an image annotation result is obtained, and the student model is determined according to a teacher model; the teacher model is obtained by training a convolutional neural network, the convolutional neural network comprises a target loss function, the convolutional neural network is used for generating a segmentation mask corresponding to the original image, and the target loss function is used for reducing the influence of the teacher model on noise data; the image annotation result comprises the original image and the segmentation mask. According to the method and the device, the technical problem that the medical image data labeling capability is poor due to the fact that the SAM-Med2D model is insufficient in generalization capability and still needs to be manually labeled and prompted in related technologies is solved.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and more specifically, to an image annotation method, device, and electronic device. Background Art

[0002] As big models become more mature in the field of artificial intelligence, they have been applied to many industries such as medicine, finance, and manufacturing. Image segmentation is of great significance for the auxiliary analysis and diagnosis of magnetic resonance imaging, computed tomography, ultrasound, etc. Therefore, a big model for the segmentation of medical 2D images, SAM-Med2D, is proposed. It not only opens up the largest medical image segmentation dataset currently, but also introduces Adapter technology for comprehensive fine-tuning.

[0003] Although the SAM-Med2D model has shown excellent results in the image segmentation performance of major organs, it still requires user interaction to input sparse prompts (points, bounding boxes) or dense prompts (masks) for segmentation, and cannot achieve fully automatic image annotation. At the same time, the massive data required for SAM-Med2D model training mainly comes from public data sets, while a small amount of private data for specific industries is scattered in various institutions. Medical institutions do not disclose stored data to the public to protect the personal privacy of patients. The lack of high-quality data for training and optimization of large models has led to insufficient personalization and generalization capabilities of the SAM-Med2D model.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide an image annotation method, device and electronic device to at least solve the technical problem that the medical image data annotation capability is poor due to the insufficient generalization capability of the SAM-Med2D model in the related technology and the need for manual annotation prompts.

[0006] According to one aspect of an embodiment of the present application, there is provided an image annotation method, comprising: determining the image quality of an original image; when a quality evaluation index of the image quality is greater than or equal to a preset threshold and the original image is a two-dimensional image, analyzing and processing the original image through a student model to obtain an image annotation result, wherein the student model is determined based on a teacher model, the student model is used to perform image segmentation on the original image, and generate a segmentation mask corresponding to the original image, the segmentation mask is used to represent label information after classifying pixel points in the original image; the teacher model is obtained by training a convolutional neural network, the convolutional neural network includes a target loss function, the convolutional neural network is used to generate a segmentation mask corresponding to the original image, the target loss function is used to reduce the influence of the teacher model when facing noise data; the image annotation result includes the original image and the segmentation mask.

[0007] Optionally, before determining the image quality of the original image, the method also includes: determining a first area and a second area of ​​the original image, wherein the first area is used to represent any valid signal area of ​​the original image, and the second area is used to represent a noise background area of ​​the original image; determining a first mean of the first area and a second mean of the second area, and determining a first standard deviation of the first area and a second standard deviation of the second area; and determining a contrast-to-noise ratio of the original image based on the first mean, the second mean, the first standard deviation, and the second standard deviation.

[0008] Optionally, determining the image quality of the original image includes: determining a target contrast-to-noise ratio of the original image, wherein the target contrast-to-noise ratio is used to represent an average contrast-to-noise ratio of the original image in multiple first regions; comparing the target contrast-to-noise ratio with a preset contrast-to-noise ratio threshold, and when the target contrast-to-noise ratio is greater than or equal to the preset contrast-to-noise ratio threshold, determining that the original image is a high-quality image; and when the target contrast-to-noise ratio is less than the preset contrast-to-noise ratio threshold, determining that the original image is a low-quality image.

[0009] Optionally, when the quality evaluation index of the image quality is less than a preset threshold, the method also includes: determining the noise data of the original image; when the original image contains impulse noise, denoising the original image by a first denoising method, wherein the first denoising method removes the impulse noise in the original image based on the grayscale average value, grayscale change rate and preset grayscale threshold of the original image; when the original image contains multiplicative speckle noise, denoising the original image by a second denoising method, wherein the second denoising method removes the multiplicative speckle noise in the original image through a logarithmic transformation; when the original image contains Gaussian noise, denoising the original image by a third denoising method, wherein the third denoising method removes the Gaussian noise in the original image through a soft threshold, and the soft threshold is used to filter the noise coefficient of the original image in the high-frequency sub-band coefficients.

[0010] Optionally, the student model is trained in the following manner: determine the initial student model of the edge server; determine the private data set of the edge server and the public data set of the central server; train the initial student model based on the public data set to obtain a prediction result score, wherein the prediction result score is used to represent the evaluation score of the image annotation effect after the initial student model analyzes and processes each data sample in the public data set; perform weighted averaging on the prediction result scores to obtain an average prediction result score; train the initial student model based on the average prediction result score and the private data set to obtain a student model.

[0011] Optionally, the method also includes: determining the dimensional information of the original image; when the dimensional information indicates that the original image is a three-dimensional image, adjusting the pixel value of the original image, and determining a slice image of the original image located on the plane where any two coordinate axes are located; and determining a two-dimensional image corresponding to the original image based on the slice image.

[0012] Optionally, the target loss function is determined in the following manner: determining the initial loss function and probability distribution type required for the teacher model to process data; determining the degrees of freedom parameter of the initial loss function, wherein the degrees of freedom parameter is used to adjust the sensitivity of abnormal data when processing data; determining the target loss function based on the dimensional information, the initial loss function, the probability distribution type and the degrees of freedom parameter.

[0013] According to another aspect of an embodiment of the present application, an image annotation device is also provided, including: a determination module, used to determine the image quality of an original image; an annotation module, used to analyze and process the original image through a student model when a quality evaluation index of the image quality is greater than or equal to a preset threshold and the original image is a two-dimensional image, to obtain an image annotation result, wherein the student model is determined based on a teacher model, the student model is used to perform image segmentation on the original image, and generate a segmentation mask corresponding to the original image, the segmentation mask is used to represent label information after classifying pixel points in the original image; the teacher model is obtained by convolutional neural network training, the convolutional neural network includes a target loss function, the convolutional neural network is used to generate a segmentation mask corresponding to the original image, the target loss function is used to reduce the impact of the teacher model when facing noise data; the image annotation result includes the original image and the segmentation mask.

[0014] According to another aspect of the embodiments of the present application, an electronic device is provided, including: a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the above-mentioned image annotation method.

[0015] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided, wherein the non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned image annotation method by running the computer program.

[0016] According to another aspect of the embodiments of the present application, a computer program product is provided, including computer instructions, which implement the above-mentioned image annotation method when executed by a processor.

[0017] In an embodiment of the present application, the image quality of the original image is determined; when the quality evaluation index of the image quality is greater than or equal to a preset threshold and the original image is a two-dimensional image, the original image is analyzed and processed by the student model to obtain an image annotation result, wherein the student model is determined based on the teacher model, the student model is used to perform image segmentation on the original image, and generate a segmentation mask corresponding to the original image, the segmentation mask is used to represent the label information after the pixels in the original image are classified; the teacher model is obtained by convolutional neural network training, the convolutional neural network includes a target loss function, the convolutional neural network is used to generate a segmentation mask corresponding to the original image, and the target loss function is used to reduce the impact of the teacher model when facing noisy data; the image annotation result includes the original image and the segmentation mask, which achieves the purpose of automated image segmentation and annotation, thereby achieving the technical effect of improving the efficiency of medical image processing and the accuracy of annotation, and thus solving the technical problem that the medical image data annotation ability is poor due to the insufficient generalization ability of the SAM-Med2D model in the related technology and the need for manual annotation prompts. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 is a hardware structure diagram of a computer terminal for implementing an image annotation method according to an embodiment of the present application;

[0020] Figure 2 is a flow chart of an image annotation method according to an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of an image annotation process according to an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of image window division according to an embodiment of the present application;

[0023] Figure 5 is a schematic diagram of a convolutional neural network structure according to an embodiment of the present application;

[0024] Figure 6 is a structural diagram of an image annotation device according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] The information collected in the embodiments of the present application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or reject automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered.

[0028] First, some nouns or terms that appear in the process of explaining the embodiments of the present application are subject to the following explanations:

[0029] CNR (Contrast-to-Noise Ratio): An objective indicator for evaluating the quality of medical images, that is, the contrast between the signal and the background in a given area.

[0030] NSCT (Nonsubsampled Contourlet Transform): A multi-scale transform method that combines the non-subsampled contourlet transform (NST) and the steerable subband filter (SCT). It has translation invariance, multi-directionality, and anisotropy.

[0031] PSNR (Peak Signal-to-Noise Ratio) is the ratio of the maximum possible signal power to the destructive noise power that affects its representation accuracy.

[0032] ROI (Region of Interest): The key area to be processed is outlined from the processed image in the form of a box, circle, ellipse, irregular polygon, etc.

[0033] RMSE (Root Mean Square Error): One of the indicators to measure image quality. The calculation principle is to square the difference between the true value and the predicted value, then sum and average, and finally take the square root.

[0034] SSIM (Structure Similarity Index Measure): measures the similarity between two images based on three key features: brightness, contrast, and structure. It uses the mean as an estimate of brightness, the standard deviation as an estimate of contrast, and the covariance as a measure of structural similarity.

[0035] IoU (Intersection over Union): The ratio of the area of ​​the intersection of two bounding boxes (or two segmentation masks) to the area of ​​the union of two bounding boxes (or two segmentation masks). An important metric in object detection and segmentation tasks.

[0036] In order to solve the problem of poor image annotation capability in the related art, the present application embodiment provides an image annotation method, which can be run on Figure 1 In the computer terminal shown, the computer terminal is described below.

[0037] The image annotation method embodiment provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing an image annotation method. Figure 1As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions connected via a wired and / or wireless network. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art can understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0038] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0039] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the image annotation method in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned image annotation method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0040] The transmission module 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0041] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0042] It should be noted that, in some optional embodiments, the above Figure 1 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0043] In the above operating environment, an embodiment of the present application provides an embodiment of an image annotation method. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0044] Figure 2 is a flow chart of an image annotation method according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0045] Step S202, determining the image quality of the original image.

[0046] In the above step S202, the contrast-to-noise ratio CNR can be used as an evaluation index of the original image to evaluate whether the original image is suitable for automatic segmentation and annotation. For example, if the CNR of the original image is greater than or equal to the preset contrast-to-noise ratio threshold, it is considered that the image quality of the original image is high and image segmentation can be performed directly; if the CNR of the original image is less than the preset contrast-to-noise ratio threshold, it is considered that the image quality of the original image is low and the original image needs to be denoised or other image processing is performed before image segmentation.

[0047] Step S204, when the quality evaluation index of the image quality is greater than or equal to a preset threshold and the original image is a two-dimensional image, the original image is analyzed and processed by the student model to obtain an image annotation result, wherein the student model is determined based on the teacher model, the student model is used to segment the original image, and generate a segmentation mask corresponding to the original image, the segmentation mask is used to represent the label information after the pixels in the original image are classified; the teacher model is obtained by convolutional neural network training, the convolutional neural network includes a target loss function, the convolutional neural network is used to generate a segmentation mask corresponding to the original image, the target loss function is used to reduce the influence of the teacher model when facing noise data; the image annotation result includes the original image and the segmentation mask.

[0048] In the above step S204, by utilizing the segmentation capability of the student model and combining the robustness of the teacher model, a high-quality image annotation result can be achieved, wherein the image annotation result includes the original image and the segmentation mask obtained by the student model segmentation, and the segmentation mask allows viewing the classification label in each pixel in the original image, which is used to represent the label information after classification of each pixel in the original image to identify different areas or objects. It should be noted that since the SAM-Med2D model can only be used to segment medical two-dimensional images, if the input is a three-dimensional image, it is necessary to perform dimensional conversion before image segmentation and annotation.

[0049] Through the above steps S202 to S204, the purpose of automatically performing image segmentation and annotation is achieved, thereby achieving the technical effect of improving the efficiency of medical image processing and the accuracy of annotation, thereby solving the technical problem of poor annotation ability of medical image data due to the insufficient generalization ability of the SAM-Med2D model in the related art and the need for manual annotation prompts. The following is a detailed description.

[0050] In an embodiment of the present application, an intelligent annotation system based on edge federated learning is proposed. The system mainly includes an edge federated learning framework, an image quality assessment module, an image multiple denoising module, and an image automatic segmentation module.

[0051] The specific module functions are as follows:

[0052] 1. Edge federated learning framework:

[0053] Image segmentation is achieved through edge federated learning with the collaborative participation of multiple parties, so that the model has both personalization and generalization capabilities. The SAM-Med2D open source data is stored in a central server that is accessible to all, and the private data of each medical institution is stored in its own edge server and not shared with the outside world. The improved SAM-Med2D is used as the teacher model. According to the computing power of each edge server, the model distillation technology is used to design its own student model. The network structure can be different. The prediction scores of all local models on the public data set are sent to the central server for aggregation to form a global consensus. Each model learns and approximates these consensuses, and then uses its own private data for repeated training for multiple rounds.

[0054] 2. Image quality assessment module:

[0055] The imaging quality of medical images is affected by many factors. Low-quality images with low clarity and contrast, as well as noise, will reduce the accuracy of diagnostic results and image annotation. Therefore, image quality can be evaluated through objective evaluation indicators such as PSNR, CNR, SSIM, and RMSE.

[0056] 3. Image multiple denoising module:

[0057] The high-quality images screened out by the image quality assessment module can be directly used for image segmentation, while the low-quality images need to undergo multiple denoising preprocessing operations. Since multimodal medical images contain impulse noise, Gaussian noise, multiplicative speckle noise, etc., the present invention combines multiple denoising methods for filtering. Among them, median filtering is a nonlinear smoothing technology that can protect image edge details while eliminating impulse noise. Logarithmic transformation is to take the logarithm of the pixel value of the image, thereby expanding the low grayscale range and compressing the high grayscale range, which plays a role in enhancing the details of the low-brightness area in the image and suppressing the overexposure of the high-brightness area in the image, so as to improve the contrast and brightness of the image. Multiplicative speckle noise can be approximately converted into Gaussian additive noise after logarithmic transformation. NSCT decomposition is performed on the image containing Gaussian noise to obtain low-frequency subband coefficients and high-frequency subband coefficients. The frequency coefficients of the noise are mostly concentrated in the high-frequency part. Only the high-frequency subband coefficients are soft-thresholded. The optimized high-frequency subband coefficients are fused with the low-frequency subband coefficients for NSCT inverse transformation reconstruction, thereby obtaining a denoised image.

[0058] 4. Automatic image segmentation module:

[0059] High-quality images or low-quality images after preprocessing are annotated through automatic segmentation. In order to achieve automation of the entire process and avoid manual participation, improvements can be made based on the SAM-Med2D model structure, such as replacing the original mask decoder so that it cannot input prompt information, and instead using a convolutional neural network to generate a predicted mask. The improved SAM-Med2D is used as the teacher model. Each medical institution simplifies the network structure and parameter quantity to design different student models based on the computing power of the edge server. In addition, the presence of noise labels in the data set caused by human mislabeling or interference in the collection process will affect the segmentation effect. The loss function is replaced with T-Loss, which is robust under high noise, to keep the model performing well.

[0060] Figure 3 is a schematic diagram of an image annotation process according to an embodiment of the present application, such as Figure 3 As shown, based on the multiple modules in the above intelligent annotation system, the overall image annotation process can be as follows: first input a medical image, then judge the image quality through the quality assessment module, if it is a high-quality image and a 2D (two-dimensional) image, directly enter the image segmentation module for processing; if it is not a high-quality image, then the image will be denoised by the image multiple denoising module to improve the image quality; for 3D (three-dimensional) images, it is necessary to perform dimensional conversion (such as normalization and slicing along three axes), and then send the converted 2D image to the image self-segmentation module for image segmentation and annotation. Among them, the dimensional conversion includes: determining the dimensional information of the original image; when the dimensional information indicates that the original image is a three-dimensional image, adjusting the pixel value of the original image, and determining the slice image of the original image located on the plane where any two coordinate axes are located; determining the two-dimensional image corresponding to the original image based on the slice image. For example, when the dimensional information of the original image indicates that the original image is a three-dimensional image, it is necessary to first normalize the voxel intensity of the original image to [0, 255], and then extract slices along the x, y, and z axes, and remove slices whose short sides are less than half of the long sides, and save them as two-dimensional images in png format.

[0061] In the above intelligent annotation system and automated image annotation process, each medical institution can use the model distillation technology to use the improved SAM-Med2D as a teacher model without sharing patient data, and design a student model based on the computing power of its own equipment, so as to achieve end-to-end image segmentation and fully automatic annotation without the need for manual input of prompt information. Useful high-quality images can be screened out through image quality assessment, and low-quality image noise can be filtered out to make full use of it. In order to avoid the impact of training data noise labels, a robust loss function can also be used to improve the accuracy and generalization ability of the model.

[0062] In the above step S202, it also includes: determining a first area and a second area of ​​the original image, wherein the first area is used to represent any valid signal area of ​​the original image, and the second area is used to represent the noise background area of ​​the original image; determining a first mean of the first area and a second mean of the second area, and determining a first standard deviation of the first area and a second standard deviation of the second area; determining a contrast-to-noise ratio of the original image based on the first mean, the second mean, the first standard deviation and the second standard deviation.

[0063] Further, determining the image quality of the original image includes: determining a target contrast-to-noise ratio of the original image, wherein the target contrast-to-noise ratio is used to represent an average contrast-to-noise ratio of the original image in multiple first regions; comparing the target contrast-to-noise ratio with a preset contrast-to-noise ratio threshold, and when the target contrast-to-noise ratio is greater than or equal to the preset contrast-to-noise ratio threshold, determining that the original image is a high-quality image; and when the target contrast-to-noise ratio is less than the preset contrast-to-noise ratio threshold, determining that the original image is a low-quality image.

[0064] In an embodiment of the present application, a contrast-to-noise ratio (CNR) indicator similar to human visual perception can be used to evaluate image quality, and corresponding processing measures can be taken according to the image quality. Specifically, first, an ROI is selected in the effective signal area of ​​the original image as the first area, denoted as A; the noise background area of ​​the original image is taken as the second area, denoted as B. Secondly, the means of A and B are calculated, that is, the first mean and the second mean mentioned above; and the standard deviations of A and B are calculated, that is, the first standard deviation and the second standard deviation mentioned above. Subsequently, the contrast-to-noise ratio (CNR) of a single ROI in the original image can be determined based on the first mean, the second mean, the first standard deviation and the second standard deviation. The specific expression is as follows:

[0065]

[0066] In the formula, CNR A represents the contrast-to-noise ratio CNR of the first area A in the original image; μ A represents the first mean; μ B represents the second mean; represents the first standard deviation; Indicates the second standard deviation.

[0067] Furthermore, in order to improve the accuracy of the data standard and make the calculation result more accurate, multiple ROIs can be selected in the effective signal area, and the average value of these ROIs (that is, the above-mentioned target contrast-to-noise ratio) can be calculated. The specific expression is as follows:

[0068]

[0069] In the formula, CNR represents the target contrast-to-noise ratio; N represents the number of ROIs selected in the effective signal area. It should be noted that the larger the CNR value, the greater the difference between the effective signal and the noise background, the stronger the contrast, and the easier it is to distinguish the noise.

[0070] Finally, by comparing the target CNR with the preset CNR threshold, the image quality of the original image can be evaluated. Specifically, if the target CNR is greater than or equal to the preset CNR threshold, the original image is determined to be a high-quality image, and the subsequent image segmentation and annotation process can be directly carried out; if the target CNR is greater than or equal to the preset CNR threshold, the original image is determined to be a high-quality image, and the subsequent image segmentation and annotation process can be directly carried out; if the target CNR is less than the preset CNR threshold, the original image is determined to be a low-quality image, and the image quality of the original image needs to be improved through denoising before the subsequent image segmentation and annotation process.

[0071] In the above step S204, when the quality evaluation index of the image quality is less than the preset threshold, it includes: determining the noise data of the original image; when the original image contains impulse noise, denoising the original image by a first denoising method, wherein the first denoising method removes the impulse noise in the original image according to the grayscale average value, the grayscale change rate and the preset grayscale threshold of the original image; when the original image contains multiplicative speckle noise, denoising the original image by a second denoising method, wherein the second denoising method removes the multiplicative speckle noise in the original image by logarithmic transformation; when the original image contains Gaussian noise, denoising the original image by a third denoising method, wherein the third denoising method removes the Gaussian noise in the original image by a soft threshold, and the soft threshold is used to filter the noise coefficient of the original image in the high-frequency sub-band coefficient.

[0072] In the embodiment of the present application, when different denoising methods are used to improve the image quality of the original image, it is first necessary to identify the type of noise data in the original image, such as impulse noise, multiplicative speckle noise and Gaussian noise. The following describes the above three denoising methods in detail according to different noise types.

[0073] 1. The first denoising method:

[0074] As an improvement of median filtering, the maximum-minimum method can still maintain a good denoising effect for more serious impulse noise, but the processing speed of recursive repeated comparison is slow. In order to balance the filtering effect and speed, on this basis, the threshold can be automatically determined by the rate of change of the image gray value to detect and filter out noise. The specific process is as follows:

[0075] First, take a 3x3 window W centered at point (i, j) on the original image, as Figure 4As shown, the grayscale average M of all pixels in the window is calculated. The specific formula is as follows:

[0076]

[0077] Where M(i,j) represents the average grayscale value of the pixels in the 3x3 window; f(k,l) represents the grayscale value of the pixel at the position (k,l) in the 3x3 window; and N represents the number of pixels in the window.

[0078] Secondly, calculate the grayscale change rate K(i,j) of the window to reflect the average level of deviation between each pixel in the window and the grayscale average M. The specific formula is as follows:

[0079]

[0080] Next, assuming Figure 4 The d9 in is a noise point, and its gray value is estimated using the maximum-minimum method. The specific formula is as follows:

[0081] L i =max(d i , d 9-i ), E i =min(d i , d 9-i ), i=1, 2, 3, 4;

[0082] P max =max(E1, E2, E3, E4), P min =min(L1, L2, L3, L4);

[0083]

[0084] Where, L i Indicates d i and d 9-i The maximum gray value in E i Indicates d i and d 9-i The minimum gray value in P max Indicates the maximum grayscale value among E1, E2, E3, and E4; P min represents the minimum grayscale value among L1, L2, L3, and L4; y represents the grayscale estimated value of the noise point d9.

[0085] Finally, the preset grayscale threshold is determined according to the grayscale change rate K(i,j). If |d9-M|≤K, d9 is not a noise point and the grayscale value of d9 is not changed; if |d9-M|>K, d9 is a noise point and the grayscale estimation value y is assigned to d9.

[0086] 2. The second denoising method:

[0087] The image contrast is enhanced by logarithmic transformation, and the multiplicative noise is converted into additive noise. The specific formula is as follows:

[0088] g(x,y)=λlog(1+f(x,y))

[0089] Where λ is the adjustment constant, which adjusts the contrast by controlling the gain, f(x, y) is the pixel value of the original image after the improved median filter, and g(x, y) is the pixel value of the original image after logarithmic transformation.

[0090] Since the multiplicative part of speckle noise has a much greater impact than the additive part, the speckle noise can be approximately expressed as:

[0091] f(x,y)=I(x,y)·η m (x, y)

[0092] Where I(x,y) is the noise-free image after filtering, and ηm(x,y) is the multiplicative speckle noise.

[0093] Finally, after logarithmic transformation, it is approximately converted into Gaussian additive noise and can be expressed as:

[0094] logη m (x,y)=logf(x,y)-logI(x,y)

[0095] In the formula, logη m (x,y) represents the value of Gaussian additive noise after logarithmic transformation of multiplicative speckle noise, logf(x,y) represents the logarithm of the original image pixel value, and logI(x,y) represents the logarithm of the noise-free image pixel value.

[0096] 3. The third denoising method:

[0097] The image is reconstructed after filtering out high-frequency noise mainly through NSCT decomposition and soft thresholding. The Bayes Shrink threshold T obtained under the condition of Bayesian risk minimization is Bayes as follows:

[0098]

[0099] In the formula, is the noise variance, σ x is the signal standard deviation, threshold T Bayes It only matters with these two values.

[0100] In order to make the decomposed sub-band coefficients use adaptive threshold filtering according to the characteristics of different regions, the image containing Gaussian noise is decomposed by NSCT. The noise intensity of the sub-band coefficients at each scale is different, the noise variance of the high-frequency coefficients is large, and the noise variance of the low-frequency coefficients is small.

[0101] Among them, the noise standard deviation of the high-frequency coefficient is estimated The median calculation is adopted, and the specific formula is as follows:

[0102]

[0103] Where, d i,j (k) is the high-frequency coefficient at the scale of the kth decomposition layer; Median is the median, which is used to estimate the noise standard deviation.

[0104] Maximum likelihood estimation of the standard deviation of the subband signal in the lth decomposition direction of the kth decomposition layer The calculation formula is as follows:

[0105]

[0106] Where M×N is the number of sub-band pixels, and the sub-band area energy distribution is comprehensively considered to obtain the sub-band soft threshold T l,k ;d i,j (l,k) is the subband coefficient of the lth decomposition direction of the kth decomposition layer.

[0107] Subband soft threshold T l,k The calculation formula is as follows:

[0108]

[0109] Where L is the number of decomposition directions of the kth decomposition layer, is the subband energy of the lth decomposition direction of the kth decomposition layer, is the total energy of the kth decomposition layer, the coefficient This is to reduce the threshold difference between different directions in the same layer.

[0110] After filtering the high-frequency sub-band coefficients by soft thresholding, the Gaussian noise of the image can be removed, and it is fused with the low-frequency sub-band coefficients for NSCT inverse transformation to obtain the reconstructed denoised image.

[0111] In the above step S204, the student model is trained in the following manner: determine the initial student model of the edge server; determine the private data set of the edge server and the public data set of the central server; train the initial student model based on the public data set to obtain a prediction result score, wherein the prediction result score is used to represent the evaluation score of the image annotation effect after the initial student model analyzes and processes each data sample in the public data set; perform weighted averaging on the prediction result scores to obtain an average prediction result score; train the initial student model based on the average prediction result score and the private data set to obtain a student model.

[0112] In the embodiment of the present application, the edge federated learning framework includes a teacher model in a central server and a student model in multiple edge servers. Assume that there are m edge servers, each of which has a small private dataset. There is also a large public dataset that all edge servers can access That is, the SAM-Med2D open source dataset. Each edge server can independently design its own student model based on its computing power. k To perform tasks, these student models can have different architectures, and each edge server does not need to share hyperparameters, but instead establishes collaboration to utilize locally accessible data D0 and D k To improve the performance of the student model. Based on this framework, the training process of the student model needs to iterate the following five steps until the student model fully converges, as follows:

[0113] Step 1: Each edge server uses the defined initial student model to calculate the prediction result score on the public dataset The prediction result score is used to represent the evaluation score of the image annotation effect after the initial student model analyzes and processes each data sample in the public dataset. In order to reduce the communication cost, a subset of 5000 is randomly selected in each round. The entire public dataset is not used.

[0114] Step 2: The central server aggregates the prediction results of all edge servers and calculates the weighted average score Weight c k In most cases

[0115] Step 3: Average the scores Distributed to each edge server.

[0116] Step 4: Each edge server trains an initial student model on the public dataset D0 to make it close to Each initial student model learns the global consensus and uses model distillation to fit the average score on the shared dataset. The knowledge of each participant can be understood by others without sharing their private data or model architecture.

[0117] Step 5: Each edge server uses its own private data to perform deep training on the initial student model to obtain the final student model f k .

[0118] For the teacher model, in order to reduce the manual input of prompt information and realize end-to-end automatic segmentation and annotation, the original mask decoder in the SAM-Med2D model structure can be replaced with a convolutional neural network to generate segmentation masks, such as Figure 5 As shown. First, the image is embedded (image embedding), that is, the original image is embedded (reshaped) into a feature map of size (256, 64, 64), and then it goes through 4 steps of the convolutional neural network. Each step consists of a convolution layer with a convolution kernel size of 3x3 and a stride of 1, and a deconvolution layer with a convolution kernel size of 2x2 and a stride of 2. The final segmentation head (Segmentation head) consists of a point-by-point convolution with a convolution kernel size of 1x1 to generate (output) a segmentation mask for each category. Among them, the upsampling step increases the size of the feature map to match the size of the original image. In some cases, feature maps at different levels can also be connected (Feature Concatenation) to merge feature information of different scales.

[0119] Furthermore, the target loss function in the teacher model is determined in the following manner: determining the initial loss function and probability distribution type required for the teacher model to process data; determining the degree of freedom parameter of the initial loss function, wherein the degree of freedom parameter is used to adjust the sensitivity of abnormal data when processing data; determining the target loss function based on the dimensional information, the initial loss function, the probability distribution type and the degree of freedom parameter.

[0120] In the embodiment of the present application, SAM-Med2D originally uses IoU to train the mean square error loss MSE between the predicted mask and the true mask. In order to improve the robustness to noise labels, it can be replaced with the T-Loss loss function, and then the segmentation result is evaluated by the Dice score. Specifically, the loss function of the image (i.e., the above-mentioned target loss function) obtained according to the T distribution (i.e., the above-mentioned probability distribution type) probability density function (i.e., the above-mentioned initial loss function) of the D-dimensional variable y can be simplified as:

[0121]

[0122] In the formula, -logp(y|μ,I D ; ν) is the negative logarithm of the probability density function of the T distribution, which represents the target loss function, where y is the predicted value, μ is the mean of the true value, and I D is the unit matrix with dimension D, ν is the degree of freedom parameter, which is used to adjust the sensitivity to outliers; Г represents the gamma function, which is used to calculate the normalization constant of the T distribution.

[0123] It should be noted that the image segmentation method proposed in the embodiment of the present application belongs to supervised learning. Although the reasoning process is intended to achieve automatic segmentation and annotation of medical images, the training process still needs to use labeled data, and the data may be affected by human errors, resulting in the introduction of noise labels. The T test is a commonly used medical statistical method for deriving the probability of difference occurrence. It assumes that the error term obeys the T distribution. Unlike the Gaussian distribution, it has a higher tolerance for noise and is more robust to outliers. The robust loss function T-Loss inspired by the T distribution is introduced, which can adaptively learn the optimal tolerance level of label noise in back propagation, and compared with modifying the network structure, this method of only changing a single component is simpler to reduce the impact of noise labels.

[0124] In the embodiment of the present application, the improved SAM-Med2D is used as the teacher model. Each medical institution simplifies the network structure and parameter quantity according to the computing power of its own edge server to design different student models, so that the student model trained on local data can meet both personalized needs and the prediction results on public data sets are close to the global consensus and meet the generalization ability. Among them, the teacher model uses a convolutional neural network to generate a predicted segmentation mask for each category, replacing the original mask decoder. The segmentation result can be obtained without manually entering prompt information to achieve automatic annotation of medical images, solving the problem that manual annotation consumes a lot of time and energy, and personal experience and subjectivity affect the quality and consistency of annotation. Since the supervised learning method is used in training the automatic segmentation model, the training data inevitably has noise labels, which leads to the reduction of the accuracy and generalization ability of the model. Therefore, by replacing the mean square error loss MSE and adopting the robust loss function T-Loss, only a single component is changed to improve the robustness to outliers and reduce the impact of noise labels on the model.

[0125] According to an embodiment of the present application, an image annotation device is provided. It should be noted that the image annotation device of the embodiment of the present application can be used to execute the image annotation method provided by the embodiment of the present application. The image annotation device provided by the embodiment of the present application is introduced below.

[0126] Figure 6 is a structural diagram of an image annotation device provided according to an embodiment of the present application. Figure 6 As shown, the device comprises:

[0127] A determination module 60, configured to determine the image quality of the original image;

[0128] The annotation module 62 is used to analyze and process the original image through the student model to obtain the image annotation result when the quality evaluation index of the image quality is greater than or equal to the preset threshold and the original image is a two-dimensional image, wherein the student model is determined based on the teacher model, the student model is used to segment the original image and generate a segmentation mask corresponding to the original image, the segmentation mask is used to represent the label information after the pixels in the original image are classified; the teacher model is obtained by convolutional neural network training, the convolutional neural network includes a target loss function, the convolutional neural network is used to generate a segmentation mask corresponding to the original image, the target loss function is used to reduce the influence of the teacher model when facing noise data; the image annotation result includes the original image and the segmentation mask.

[0129] Through the determination module 60 and the annotation module 62 in the above-mentioned image annotation device, the purpose of automated image segmentation and annotation is achieved, thereby realizing the technical effect of improving the efficiency of medical image processing and the accuracy of annotation, and further solving the technical problem of poor medical image data annotation capability due to the insufficient generalization ability of the SAM-Med2D model in the related technology and the need for manual annotation prompts.

[0130] In the image annotation device provided in an embodiment of the present application, the determination module is further used to determine a first area and a second area of ​​the original image, wherein the first area is used to represent any valid signal area of ​​the original image, and the second area is used to represent the noise background area of ​​the original image; determine a first mean of the first area and a second mean of the second area, and determine a first standard deviation of the first area and a second standard deviation of the second area; determine the contrast-to-noise ratio of the original image based on the first mean, the second mean, the first standard deviation, and the second standard deviation.

[0131] In the image annotation device provided in the embodiment of the present application, the determination module is further used to determine a target contrast-to-noise ratio of the original image, wherein the target contrast-to-noise ratio is used to represent an average contrast-to-noise ratio of the original image in multiple first regions; compare the target contrast-to-noise ratio with a preset contrast-to-noise ratio threshold, and when the target contrast-to-noise ratio is greater than or equal to the preset contrast-to-noise ratio threshold, determine that the original image is a high-quality image; when the target contrast-to-noise ratio is less than the preset contrast-to-noise ratio threshold, determine that the original image is a low-quality image.

[0132] In the image annotation device provided in the embodiment of the present application, the determination module is also used to determine the noise data of the original image when the quality evaluation index of the image quality is less than a preset threshold; when the original image contains impulse noise, the original image is denoised by a first denoising method, wherein the first denoising method removes the impulse noise in the original image based on the grayscale average value, the grayscale change rate and the preset grayscale threshold of the original image; when the original image contains multiplicative speckle noise, the original image is denoised by a second denoising method, wherein the second denoising method removes the multiplicative speckle noise in the original image through a logarithmic transformation; when the original image contains Gaussian noise, the original image is denoised by a third denoising method, wherein the third denoising method removes the Gaussian noise in the original image through a soft threshold, and the soft threshold is used to filter the noise coefficient of the original image in the high-frequency sub-band coefficient.

[0133] In the image annotation device provided in the embodiment of the present application, the determination module is also used to determine the initial student model of the edge server; determine the private data set of the edge server and the public data set of the central server; train the initial student model based on the public data set to obtain a prediction result score, wherein the prediction result score is used to represent the evaluation score of the image annotation effect after the initial student model analyzes and processes each data sample in the public data set; perform weighted averaging on the prediction result scores to obtain an average prediction result score; train the initial student model based on the average prediction result score and the private data set to obtain a student model.

[0134] In the image annotation device provided in the embodiment of the present application, the determination module is also used to determine the dimensional information of the original image; when the dimensional information indicates that the original image is a three-dimensional image, the pixel value of the original image is adjusted, and a slice image of the original image located on the plane where any two coordinate axes are located is determined; and the two-dimensional image corresponding to the original image is determined based on the slice image.

[0135] In the image annotation device provided in the embodiment of the present application, the determination module is also used to determine the initial loss function and probability distribution type required by the teacher model for data processing; determine the degree of freedom parameter of the initial loss function, wherein the degree of freedom parameter is used to adjust the sensitivity of abnormal data when performing data processing; determine the target loss function based on the dimensional information, the initial loss function, the probability distribution type and the degree of freedom parameter.

[0136] An embodiment of the present application further provides an electronic device, including: a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute and implement the above-mentioned image annotation method.

[0137] It should be noted that the above electronic equipment is used to execute Figure 2The image annotation method shown, therefore the relevant explanations in the above image annotation method are also applicable to the electronic device and will not be repeated here.

[0138] An embodiment of the present application further provides a non-volatile storage medium, which includes a stored computer program, wherein a device where the non-volatile storage medium is located executes the above-mentioned image annotation method by running the computer program.

[0139] It should be noted that the above non-volatile storage medium is used to execute Figure 2 The image annotation method shown, therefore the relevant explanations in the above image annotation method are also applicable to the non-volatile storage medium, and will not be repeated here.

[0140] The embodiment of the present application also provides a computer program product, including computer instructions, which implement the above-mentioned image annotation method when executed by a processor.

[0141] It should be noted that the above-mentioned computer program product is used to execute Figure 2 The image annotation method shown, therefore the relevant explanations in the above image annotation method are also applicable to the computer program product, and will not be repeated here.

[0142] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0143] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0144] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0145] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0146] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0147] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0148] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image annotation method, characterized in that: include: Determining image quality of original images; When the quality evaluation index of the image quality is greater than or equal to a preset threshold and the original image is a two-dimensional image, the original image is analyzed and processed by a student model to obtain an image annotation result, wherein the student model is determined based on a teacher model, the student model is used to perform image segmentation on the original image, and generate a segmentation mask corresponding to the original image, the segmentation mask is used to represent the label information after the pixels in the original image are classified; the teacher model is obtained by convolutional neural network training, the convolutional neural network includes a target loss function, the convolutional neural network is used to generate a segmentation mask corresponding to the original image, the target loss function is used to reduce the influence of the teacher model when facing noise data; the image annotation result includes the original image and the segmentation mask.

2. The method according to claim 1, characterized in that Before determining the image quality of the original image, the method further includes: Determine a first area and a second area of ​​the original image, wherein the first area is used to represent any valid signal area of ​​the original image, and the second area is used to represent a noise background area of ​​the original image; determining a first mean of the first region and a second mean of the second region, and determining a first standard deviation of the first region and a second standard deviation of the second region; A contrast-to-noise ratio of the original image is determined according to the first mean, the second mean, the first standard deviation, and the second standard deviation.

3. The method according to claim 2, characterized in that Determining the image quality of the original image includes: Determining a target contrast-to-noise ratio of the original image, wherein the target contrast-to-noise ratio is used to represent an average contrast-to-noise ratio of the original image in a plurality of first regions; comparing the target contrast-to-noise ratio with a preset contrast-to-noise ratio threshold, and determining that the original image is a high-quality image when the target contrast-to-noise ratio is greater than or equal to the preset contrast-to-noise ratio threshold; When the target contrast-to-noise ratio is less than the preset contrast-to-noise ratio threshold, the original image is determined to be a low-quality image.

4. The method according to claim 1, characterized in that: When the quality evaluation index of the image quality is less than a preset threshold, the method further includes: Determining noise data of the original image; In the case where the original image contains impulse noise, denoising the original image by a first denoising method, wherein the first denoising method removes the impulse noise in the original image according to the grayscale average value, grayscale change rate and preset grayscale threshold of the original image; In the case where the original image contains multiplicative speckle noise, denoising the original image by a second denoising method, wherein the second denoising method removes the multiplicative speckle noise in the original image by logarithmic transformation; In the case where the original image contains Gaussian noise, the original image is denoised by a third denoising method, wherein the third denoising method removes the Gaussian noise in the original image by a soft threshold, and the soft threshold is used to filter the noise coefficient of the original image in the high-frequency sub-band coefficient.

5. The method according to claim 1, characterized in that: The student model is trained in the following way: Determine the initial student model of the edge server; Determining a private data set of the edge server and a public data set of the central server; The initial student model is trained according to the public data set to obtain a prediction result score, wherein the prediction result score is used to represent an evaluation score of the image annotation effect after the initial student model analyzes and processes each data sample in the public data set; Performing weighted average processing on the prediction result scores to obtain an average prediction result score; The initial student model is trained according to the average prediction result score and the private data set to obtain the student model.

6. The method according to claim 1, characterized in that The method further comprises: Determining dimensional information of the original image; When the dimension information indicates that the original image is a three-dimensional image, adjusting the pixel value of the original image, and determining a slice image of the original image located on a plane where any two coordinate axes are located; A two-dimensional image corresponding to the original image is determined according to the slice image.

7. The method according to claim 6, characterized in that The objective loss function is determined in the following way: Determining an initial loss function and a probability distribution type required for the teacher model to process data; Determining a degree of freedom parameter of the initial loss function, wherein the degree of freedom parameter is used to adjust the sensitivity of abnormal data when performing data processing; The target loss function is determined based on the dimensional information, the initial loss function, the probability distribution type and the degree of freedom parameter.

8. An image annotation device, characterized in that: include: A determination module, used for determining the image quality of the original image; A labeling module, used to analyze and process the original image through a student model to obtain an image labeling result when the quality evaluation index of the image quality is greater than or equal to a preset threshold and the original image is a two-dimensional image, wherein the student model is determined based on the teacher model, the student model is used to perform image segmentation on the original image and generate a segmentation mask corresponding to the original image, and the segmentation mask is used to represent the label information after the pixels in the original image are classified; the teacher model is obtained by convolutional neural network training, the convolutional neural network includes a target loss function, the convolutional neural network is used to generate a segmentation mask corresponding to the original image, and the target loss function is used to reduce the influence of the teacher model when facing noise data; the image labeling result includes the original image and the segmentation mask.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store program instructions; The processor is connected to the memory and is used to execute the image annotation method according to any one of claims 1 to 7.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the image annotation method according to any one of claims 1 to 7 by running the computer program.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the image annotation method according to any one of claims 1 to 7 is implemented.

Images