Skin image supplementing method and device, equipment and storage medium

Abstract

The invention relates to a skin image supplementing method, device and equipment and a storage medium, and is applied to the field of image processing, and the method comprises the steps: obtaining lesion sample images of c lesion regions, each lesion region comprising m lesion sample images; de-noising processing is carried out on the m focus sample images of each focus area, and n filtering low-noise images are generated; performing image fusion on the n filtered low-noise images of each lesion area to generate a clear complementary image; dividing the filtered low-noise images of the c focus areas and the corresponding clear complementary images into training set data and verification set data according to a preset proportion; constructing a neural network model, performing training and verification, and setting the trained and verified neural network model as a skin image completion model; and when a to-be-supplemented skin image is received, inputting the to-be-supplemented skin image into the skin image supplementing model for skin image supplementing. The method has the technical effects that the skin image is supplemented, and fungus detection and recognition are facilitated.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a skin image completion method, device, equipment and storage medium. Background Art

[0002] There are about 300 species of fungi known to humans that are pathogenic to themselves. Severe fungal infections may cause systemic disseminated infections. Therefore, it is critical to diagnose and treat fungal infections in the early stages of infection in order to reduce mortality. In the process of clinical diagnosis and treatment, fluorescence detection has developed rapidly and the staining operation is relatively simple. Compared with traditional microscopic examination methods, it has obvious advantages, making fluorescence detection more and more widely used in the diagnosis of fungal infections.

[0003] After fluorescent staining, fungi (including hyphae and spores) appear clear bright blue under a microscope. In most cases, the septum structure of the hyphae and the budding morphology of the spores can be clearly seen, so that the fungi can be effectively detected. However, due to the thickness of the skin sample, in the existing detection method, some skin images can only capture a part of the target, resulting in incomplete display of hyphae and spores, making it difficult to detect and identify fungi based on the detected hyphae and spores. Therefore, there is an urgent need for a skin image completion method that can complete the skin image so that the hyphae and spores are fully displayed, so as to help subsequent clinicians or artificial intelligence to further distinguish the image recognition results. Summary of the invention

[0004] In order to help solve the problem in existing detection methods that some skin images can only capture a part of the target, resulting in incomplete display of hyphae and spores, and difficulty in detecting and identifying fungi based on the detected hyphae and spores, the present application provides a skin image completion method, device, equipment and storage medium.

[0005] In a first aspect, the present application provides a skin image completion method, which adopts the following technical solution: the method comprises:

[0006] Acquire c lesion sample images of lesion regions, each of the lesion regions including m lesion sample images;

[0007] Performing denoising processing on the m lesion sample images of each lesion area and generating n filtered low-noise images, wherein n is less than m;

[0008] Performing image fusion on the n filtered low-noise images of each lesion area to generate a clear and completed image;

[0009] Dividing the c filtered low-noise images of the lesion area and the corresponding clear padded images into training set image data and verification set image data according to a preset ratio;

[0010] Constructing a neural network model, using the training set image data for training, using the verification set image data for verification, and setting the trained and verified neural network model as a skin image completion model;

[0011] When a skin image to be completed is received, the skin image to be completed is input into the skin image completion model to complete the skin image.

[0012] In a specific implementation scheme, performing denoising on the m lesion sample images of each lesion area and generating n filtered low-noise images comprises:

[0013] Calculating the frequency intensity of the lesion sample image of each lesion area at a preset frequency, and screening n frequency screening images of each lesion area that meet preset conditions according to the frequency intensity;

[0014] The n frequency-screened images of each lesion area are filtered to generate n filtered low-noise images.

[0015] In a specific implementation scheme, the step of calculating the frequency intensity of the lesion sample image of each lesion area at a preset frequency, and screening n frequency screening images of each lesion area that meet preset conditions according to the frequency intensity includes:

[0016] Grayscale processing is performed on the m lesion sample images of each lesion area to generate m lesion grayscale images;

[0017] Converting the m lesion grayscale images of each lesion area from the spatial domain to the frequency domain;

[0018] Calculating the frequency intensity of the m lesion grayscale images of each lesion area at a preset frequency, and generating a frequency intensity set;

[0019] Selecting an image with the highest frequency intensity value in the frequency intensity set;

[0020] The image with the highest frequency intensity value and several images before and after the image with the highest frequency intensity value are set as the frequency screening images, wherein the total number of frequency screening images for each lesion area is n, and n is less than m.

[0021] In a specific implementation scheme, filtering the n frequency-screened images of each lesion area and generating n filtered low-noise images comprises:

[0022] Creating a neighborhood window for each pixel point in the n frequency-screened images of each lesion area;

[0023] Calculating the weight of each pixel in the neighborhood window, and generating a Gaussian kernel according to the weight of each pixel in the neighborhood window;

[0024] Using the Gaussian kernel, the RGB value of each pixel in the n frequency-screened images of each lesion area is updated;

[0025] The updated image is set as a filtered low-noise image, and each of the lesion areas contains n filtered low-noise images.

[0026] In a specific implementation scheme, fusing the n filtered low-noise images of each lesion area to generate a clear and completed image comprises:

[0027] Segmenting the n filtered low-noise images according to a preset first image size to generate a plurality of first low-noise sub-image blocks, and fusing the plurality of the first low-noise sub-image blocks according to a preset first image fusion method to generate a first fused image;

[0028] Segmenting the n filtered low-noise images according to a preset second image size to generate a plurality of second low-noise sub-image blocks, and fusing the plurality of second low-noise sub-image blocks according to a preset first image fusion method to generate a second fused image;

[0029] Segmenting the n filtered low-noise images according to a preset third image size to generate a plurality of third low-noise sub-image blocks, and fusing the plurality of third low-noise sub-image blocks according to a preset first image fusion method to generate a third fused image;

[0030] fusing the first fused image, the second fused image and the third fused image according to a preset weight, and generating a final clear complemented image;

[0031] The same image fusion process is performed on the n filtered low-noise images of each lesion area to finally generate c clear and completed images.

[0032] In a specific implementation scheme, the preset first image fusion method includes:

[0033] After the n filtered low-noise images are divided into a plurality of low-noise sub-image blocks according to a preset image size, a frequency intensity value of each low-noise sub-image block in each of the filtered low-noise images is calculated;

[0034] The n low-noise sub-image blocks at the same position after segmentation are fused according to the RGB values ​​of the pixels of the low-noise sub-image blocks and the corresponding frequency intensity values, and a fused sub-image block is generated. The calculation method of the fused sub-image block includes:

[0035]

[0036] Among them, F (i) represents the frequency intensity value of the i-th low-noise sub-image block at the same position after segmentation, img (i) (x, y) represents the RGB value of the pixel coordinates of the i-th low-noise sub-image block at the same position after segmentation, and n is the number of low-noise sub-image blocks at the same position after segmentation;

[0037] All the fused sub-image blocks are spliced ​​together to generate a fused image.

[0038] In a specific implementation scheme, constructing a neural network model includes:

[0039] Constructing a neural network model structure and a loss function, wherein the structure of the neural network model includes a convolutional layer, a multi-head attention mechanism layer, a pooling layer, an upsampling layer, and a fully connected layer;

[0040] The loss function includes:

[0041]

[0042] Where p represents the total number of validation set image data, Z 生i represents the i-th image generated by the neural network model, Z 原i represents a clear padded image corresponding to the image generated by the neural network model, and δ represents a constant.

[0043] In a second aspect, the present application provides a skin image completion device, which adopts the following technical solution: the device comprises:

[0044] A sample image acquisition module, used for acquiring lesion sample images of c lesion regions, each of which includes m lesion sample images;

[0045] A sample image denoising module, used for denoising the m lesion sample images of each lesion area and generating n filtered low-noise images, wherein n is less than m;

[0046] A low-noise image fusion module, used for fusing the n filtered low-noise images of each lesion area to generate a clear and completed image;

[0047] An image data division module, used for dividing the c filtered low-noise images of the lesion area and the corresponding clear padded images into training set image data and verification set image data according to a preset ratio;

[0048] A completion model generation module, used to construct a neural network model, use the training set image data for training, use the verification set image data for verification, and set the trained and verified neural network model as a skin image completion model;

[0049] The skin image completion module is used to input the skin image to be completed into the skin image completion model to complete the skin image when receiving the skin image to be completed.

[0050] In a third aspect, the present application provides a computer device, which adopts the following technical solution: it includes a memory and a processor, and the memory stores a computer program that can be loaded by the processor and execute any of the above-mentioned skin image completion methods.

[0051] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution: storing a computer program that can be loaded by a processor and execute any of the above-mentioned skin image completion methods.

[0052] In summary, this application has the following beneficial technical effects:

[0053] A large number of original lesion sample images are obtained, and the lesion sample images are processed to generate completed skin images. The known completed skin images are trained and learned through a neural network model, and the model is verified to generate a final skin image completion model. After fluorescent staining, the hyphae and spores that are not fully displayed in the skin image can be completed so that the hyphae and spores can be fully displayed, which helps to improve the accuracy of fungal detection and identification, and then facilitates subsequent clinicians to interpret the fungal detection and identification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flow chart of a skin image completion method in an embodiment of the present application;

[0055] Figure 2 This is a schematic diagram of the structure of a neural network model in an embodiment of the present application;

[0056] Figure 3 is a schematic diagram of a skin image completion device in an embodiment of the present application;

[0057] Figure 4 It is a schematic diagram used to embody a computer device in an embodiment of the present application.

[0058] Reference numerals: 301, sample image acquisition module; 302, sample image denoising module; 303, low-noise image fusion module; 304, image data division module; 305, completion model generation module; 306, skin image completion module. DETAILED DESCRIPTION

[0059] The following combination Figure 1-Figure 4 This application is described in further detail.

[0060] An embodiment of the present application discloses a skin image completion method, which can be used to complete skin images so that hyphae and spores in the skin images can be fully displayed, thereby facilitating subsequent clinical doctors or artificial intelligence to further interpret the results.

[0061] There are about 300 species of fungi known to humans that are pathogenic to themselves. Clinically, pathogenic fungi are divided into two categories: superficial fungi and deep fungi, based on the differences in the parts of the human body that fungi invade. Superficial fungi (also known as tinea) only invade the skin, hair, and nails, while deep fungi can invade the mucous membranes, deep tissues, and internal organs of the human body. In severe cases, they can even cause systemic disseminated infections. Therefore, it is extremely important to make a diagnosis and treat fungal infections in the early stages of infection to reduce mortality. A large class of infectious diseases caused by the invasion of pathogenic fungi on the skin, resulting in infections of the skin, appendages, subcutaneous tissues, and mucous membranes is called dermatophyte infection. It is one of the most common types of skin diseases, and most of them present as blister-scaly symptoms in clinical practice. Dermatophytes are the main pathogens that cause fungal skin diseases, such as Trichophyton, Trichophyton, Trichophyton rubrum, Trichophyton gypseum, Trichophyton verrucosum, Microsporum, etc. all belong to this category. This disease has the characteristics of high incidence, high infectiousness, and high recurrence rate after treatment, which has a great impact on the daily life and psychology of patients. In recent years, the incidence of fungal infection has increased, and the number of secondary fungal infections has also increased. Therefore, effective fungal detection is particularly important in the process of clinical diagnosis and treatment.

[0062] At present, the mainstream fungal detection methods usually use wet mount microscopy, Gram staining, fungal culture and fluorescence detection. The wet mount microscopy method is simple to operate, but it has high requirements for the inspectors. Only professional inspectors can ensure a high detection rate, and it is difficult to promote on a large scale. The Gram staining method is relatively complicated and time-consuming. It is easily affected by various factors such as the quality of the dye solution and the level of the inspectors. Therefore, it is rarely used in clinical practice. The fungal culture method takes a long time and can easily cause the person being tested to miss the best time for treatment. The fluorescence detection method has developed rapidly in recent years, and the staining operation is relatively simple. Compared with the traditional microscopy method, it has obvious advantages. In recent years, it has been more and more widely used in the diagnosis of fungal infections.

[0063] After fluorescent staining, fungi (including hyphae and spores) appear clear bright blue under the microscope. In most cases, the septum structure of hyphae and the budding morphology of spores can be clearly seen. Especially in KOH smears, fungal spores and fat droplets are sometimes difficult to distinguish, but the fluorescent staining method does not mark fat droplets, which greatly improves the specificity and makes up for the defect of the traditional KOH smear method with a low detection rate when the number of fungi is small.

[0064] However, due to the thickness of the skin sample, in the existing detection method, some skin images can only capture a part of the target, such as long hyphae, only one-fifth or even one-tenth of the hyphae can be captured, resulting in incomplete display of hyphae and spores, making it difficult to detect and identify fungi based on the detected hyphae and spores. In order to help fully display hyphae and spores and improve the detection rate of fungal detection, the present application provides a skin image completion method.

[0065] Reference Figure 1 , the method comprises the following steps:

[0066] S10, acquiring lesion sample images of c lesion regions, each lesion region including m lesion sample images.

[0067] Specifically, the original lesion sample data for model training and model verification is obtained. The original lesion sample data can be directly obtained from the existing database, or it can be obtained by taking images of known patient lesion areas, where the values ​​of c and m are positive integers. In the embodiment of the present application, the sample data is obtained by taking images of known patient lesion areas; 1000 patients are used as samples in this application, 100 lesion areas are selected for each patient, and 20 pictures are taken from bottom to top in each lesion area within the preset depth of field with a step distance of 2 microns, that is, the number of lesion areas c in this application is 1000*100=100,000, and the number of lesion sample images m of each lesion area is 20. In the embodiment of the present application, the number of c and m is only used as an example, and the specific numerical value set by the user can be set according to actual needs, and there is no restriction here, as long as c and m are positive integers. It should be noted that the processing of skin image filling is generally performed after fluorescent staining. Therefore, the original lesion sample image used in this application has been fluorescently stained by default.

[0068] S20, performing denoising processing on the m lesion sample images of each lesion area and generating n filtered low-noise images, wherein n is less than m.

[0069] Specifically, the m lesion sample images of each lesion area are denoised, which can also be understood as, for the m lesion sample images of a lesion area, the m lesion sample images are all denoised, and finally n filtered low-noise images are screened out. The n filtered low-noise images are clear images after filtering and denoising. The lesion sample images of other lesion areas are also denoised and filtered in the same way to obtain a relatively clear skin image after processing; wherein the number of n is less than m, and n is also a positive integer. In this application, m is set to 20 and n is set to 11. The specific values ​​are set by the user according to actual needs and are not limited here. It should be noted that the original lesion sample images and the filtered low-noise images after denoising are both incomplete skin images, and the hyphae or spores at the lesions in the skin images are not fully displayed.

[0070] S30, fusing the n filtered low-noise images of each lesion area to generate a clear complement image.

[0071] Specifically, for an image of a lesion area, n filtered low-noise images after filtering and denoising are fused to generate a complete, padded, clear image. The clear padded image is a padded image, which can also be understood as the hyphae and spores at the lesion in the image can be fully displayed, which is helpful for subsequent clinicians or artificial intelligence to further interpret the results. The same method is used for image fusion processing for the remaining lesion areas, and the number of clear padded images finally generated is consistent with the number of lesion areas. In this application, 100,000 images of lesion areas are selected, and 100,000 clear padded images are finally generated.

[0072] S40, dividing the filtered low-noise images of the c lesion areas and the corresponding clear complement images into training set image data and verification set image data according to a preset ratio.

[0073] Specifically, in the present application, 100,000 images of lesion areas are selected, and finally 100,000 clear and completed images are generated. The filtered low-noise images corresponding to the 100,000 lesion areas and the clear and completed images generated by the fusion of the filtered low-noise images are divided into training set image data and verification set image data according to a preset ratio. In the embodiment of the present application, 80% of the data is set as training set data, and 20% of the data is set as verification set data. It can also be understood that 80,000 filtered low-noise images and the corresponding clear and completed images of lesion areas are set as training set data, and the remaining 20,000 filtered low-noise images and the corresponding clear and completed images of lesion areas are set as verification set data.

[0074] S50, constructing a neural network model, using the training set image data for training, using the verification set image data for verification, and setting the trained and verified neural network model as a skin image completion model.

[0075] Specifically, a neural network model is constructed, and the training set image data is input into the neural network model for model training. After the model training, the remaining validation set image data is input into the neural network model for model verification. After the training and model verification are completed, the model is set as the final skin image completion model.

[0076] It should be noted that the model training is to input the generated filtered low-noise image and the corresponding clear complement image into the model for training, and a total of 11 filtered low-noise images are obtained in the processing process. In the actual training process, one of the 11 filtered low-noise images is selected as the input image of the model, and the remaining 10 filtered low-noise images are input into the model as supplementary external data to extract more details and improve the accuracy of model training; among them, the one input image can be the image with the highest clarity among the 11 filtered low-noise images.

[0077] S60: When a skin image to be completed is received, the skin image to be completed is input into a skin image completion model to complete the skin image.

[0078] Specifically, in practical applications, when a skin image to be completed is received, the skin image to be completed is input into a skin image completion model for skin image completion, and finally a skin-completed image that can be easily interpreted by a clinician is generated.

[0079] In the present application scheme, a large number of original lesion sample images are obtained, the lesion sample images are processed to generate completed skin images, the known completed skin images are trained and learned through a neural network model, and the model is verified to generate a final skin image completion model, so that after fluorescent staining, the hyphae and spores that are not fully displayed in the skin image can be completed so that the hyphae and spores can be fully displayed, which helps to improve the accuracy of fungal detection and identification, and thus facilitates subsequent clinicians to interpret the fungal detection and identification results.

[0080] In one embodiment, the step of performing denoising on the m lesion sample images of each lesion region and generating n filtered low-noise images may be specifically performed as follows:

[0081] First, the frequency intensity of the lesion sample image of each lesion area at the preset frequency is calculated, and n frequency screening images that meet the preset conditions of each lesion area are screened according to the frequency intensity. Specifically, the m lesion sample images of each lesion area are grayed, and m lesion gray images are generated; in this application, the number of m is set to 20, which can also be understood as graying 20 lesion sample images of a lesion area to generate 20 lesion gray images, and the lesion sample images of the remaining lesion areas are processed in the same way to generate lesion gray images. The data channel can be reduced by graying, thereby reducing the amount of data calculation and improving the speed of data processing. Afterwards, the m lesion gray images of each lesion area are converted from the spatial domain to the frequency domain representation, that is, 20 lesion gray images are converted to the frequency domain representation; based on the less interference characteristics of the skin fluorescence picture, converting the picture from the spatial domain to the frequency domain is conducive to analyzing the frequency components in the image, and the image with high definition contains more high-frequency components, which are strongly related to the texture and edge in the image.

[0082] Afterwards, the frequency intensities of the m lesion grayscale images of each lesion area at the preset frequency are calculated, and a frequency intensity set is generated; the calculation method of the frequency intensity value can be expressed as:

[0083]

[0084] Among them, f(x,y) represents the grayscale value of a certain coordinate (x,y) after the image is converted into a lesion grayscale image, F(u,v) represents the frequency intensity at the frequency (u,v) after conversion to the frequency domain, a and b represent the width and height of the image respectively, u and v represent variables in the frequency domain, j represents the imaginary unit, and e represents the base of the natural logarithm.

[0085] The calculated frequency intensity values ​​are combined into a frequency intensity set, which can be expressed as:

[0086] [F(u,v)1, F(u,v)2, F(u,v)3,…,F(u,v) 20 ],

[0087] That is, after converting 20 lesion grayscale images of a lesion area into the frequency domain, the frequency intensities of the 20 images at the preset frequency (u, v) are calculated to form the above-mentioned frequency intensity set.

[0088] After that, the image with the highest frequency intensity value in the frequency intensity set is selected, that is, the image with the highest frequency intensity value in the range of F(u,v)1 to F(u,v) 20, select the image with the highest frequency intensity value; set the image with the highest frequency intensity value and several images before and after the image with the highest frequency intensity value as frequency screening images, where the total number of frequency screening images for each lesion area is n, and n is less than m. For example, assuming that the frequency intensity value of the i-th value is the highest, the number of the front and back selected is set to 10, 5 before and after, then select the i-th image and five images before and after the i-th image to form the clearest image set, which can be expressed as:

[0089] [F(u,v) i-5 ,F(u,v) i-4 ,…,F(u,v) i ,…,F(u,v) i+5 ].

[0090] It should be noted that the number of images before and after the image with the highest frequency intensity value may be consistent or inconsistent, and the number of images set is set by the user according to actual conditions; in actual operation, it is set to select a number of images before and after, and when the number of images before and after the image is insufficient, no supplementary processing is performed. For example, the i-th image and five images before and after the i-th image are selected to form the clearest image set, and when the number of images before or after the i-th image is less than 5, no supplementation is performed. In addition, the frequency-filtered image is an image obtained by frequency filtering, and is still processed in the spatial domain in subsequent processing, and does not represent an image in the frequency domain.

[0091] After filtering the image according to the frequency intensity value, the n frequency-filtered images of each lesion area are filtered and n filtered low-noise images are generated. Specifically, a neighborhood window is created for each pixel point in the n frequency-filtered images of each lesion area; creating a neighborhood window can be understood as constructing a new coordinate system with the pixel point as the center. In this application, the new coordinate system is composed of 7*7 points, and the constructed new coordinate system can be shown in Table 1.

[0092] Table 1, new coordinate system composed of 7*7 points

[0093] (-3,-3) (-3,-2) (-3,-1) (-3,0) (-3,1) (-3,2) (-3,3) (-2,-3) (-2,-2) (-2,-1) (-2,0) (-2,1) (-2,2) (-2,3) (-1,-3) (-1,-2) (-1,-1) (-1,0) (-1,1) (-1,2) (-1,3) (0,-3) (0,-2) (0,-1) (0,0) (0,1) (0,2) (0,3) (1,-3) (1,-2) (1,-1) (1,0) (1,1) (1,2) (1,3) (2,-3) (2,-2) (2,-1) (2,0) (2,1) (2,2) (2,3) (3,-3) (3,-2) (3,-1) (3,0) (3,1) (3,2) (3,3)

[0094] It should be noted that the neighborhood window, that is, the constructed new coordinate system can be composed of 7*7 points, 5*5, or 3*3 points, and the specific size is not limited here.

[0095] Based on the constructed neighborhood window, that is, the new coordinate system, the weight of each pixel in the neighborhood window is calculated, and the Gaussian kernel is generated according to the weight of each pixel in the neighborhood window. The weight calculation method can be expressed as:

[0096]

[0097] Wherein, σ represents a variable constant. After calculation, for the skin fluorescence image, the best performance is achieved when the value of σ is 0.8. Therefore, in the embodiment of the present application, the value of σ is set to 0.8; (x, y) represents the coordinates of any pixel point in the frequency screening image; μ x and μ y Indicates the field number, in this application μ x =μ y =0, when μ x =μ y = 0, the best performance is achieved for skin fluorescence images. In addition, if the edge pixels are difficult to form a 7*7 area, the value of the missing area is replaced by 0.

[0098] The Gaussian kernel can be calculated based on the 7*7 area and the calculated weight. In this application, a 7*7 area is set. Therefore, the Gaussian kernel finally calculated is also a 7*7 matrix. The Gaussian kernel calculated in this application is:

[0099]

[0100] After obtaining the Gaussian kernel, the RGB value of each pixel in the n frequency-filtered images of each lesion area is updated using the Gaussian kernel, that is, the RGB value in the 11 frequency-filtered images is updated using the Gaussian kernel, and the frequency-filtered images of all lesion areas are updated repeatedly. Finally, the updated image is set as a filtered low-noise image, and each lesion area contains n filtered low-noise images. It can also be understood that each lesion area after the update includes 11 filtered low-noise images.

[0101] In the present application scheme, multiple images with high frequency intensity values ​​are obtained through frequency screening, and the first denoising and screening process is completed to obtain multiple images with higher clarity. Afterwards, the RGB value of each pixel in the image screened according to the frequency value is updated using a Gaussian kernel to filter out noise interference caused by external factors such as the shooting environment and light, thereby completing the second denoising process, improving image quality, and helping to improve the accuracy of the model training and verification process.

[0102] In one embodiment, the step of fusing n filtered low-noise images of each lesion area to generate a clear complement image may be specifically performed as follows:

[0103] The n filtered low-noise images are respectively divided according to a preset first image size to generate a plurality of first low-noise sub-image blocks, and the plurality of first low-noise sub-image blocks are image-fused according to a preset first image fusion method to generate a first fused image; the n filtered low-noise images are respectively divided according to a preset second image size to generate a plurality of second low-noise sub-image blocks, and the plurality of second low-noise sub-image blocks are image-fused according to a preset first image fusion method to generate a second fused image; the n filtered low-noise images are respectively divided according to a preset third image size to generate a plurality of third low-noise sub-image blocks, and the plurality of third low-noise sub-image blocks are image-fused according to a preset first image fusion method to generate a third fused image. In the present application, it can also be understood that, for a lesion area, it includes 11 filtered low-noise images after denoising, and the 11 filtered low-noise images are segmented three times according to different sizes. The size of the segmented image can be set by the user and is not limited here; in the present application, the image sizes of the three segmentations are set to 4*4, 8*8 and 16*16, and the 4*4 size segmentation is performed first. After the segmentation, the low-noise sub-image blocks after the 11 filtered low-noise images are segmented are fused according to the preset first image fusion method to generate a first fused image, and then, the 8*8 size segmentation and fusion operations and the 16*16 size segmentation and fusion operations are performed in sequence to generate the second and third fused images, and the image segmentation and image fusion operations of the 8*8 size and the 16*16 size are consistent with the segmentation and fusion of the 4*4 size.

[0104] Afterwards, the first fused image, the second fused image and the third fused image are fused according to preset weights to generate a final clear complement image. The weights can be set by the user according to actual conditions. In this application, the sizes of the three segmented images are set to 4*4, 8*8 and 16*16, and the weights are set to 0.2, 03 and 0.5 respectively. The clear complement image generated by the final fusion can be expressed as:

[0105] B 清晰 (x, y) = 0.5*B 16*16 (x,y)+0.3*B 8*8 (x,y)+0.2*B 4*4 (x, y),

[0106] Among them, B 4*4 (x, y) represents the first fused image, B 8*8 (x, y) represents the second fused image, B 16*16 (x, y) represents the third fused image, B 清晰 (x, y) represents the final generated clear complement image.

[0107] A lesion area contains 11 filtered low-noise images, and the 11 filtered low-noise images are finally fused to generate 1 clear and completed image. Therefore, the same image fusion processing is performed on the n filtered low-noise images of each lesion area, and finally c clear and completed images can be generated. In this application, 100,000 lesion areas are selected, which means that 100,000 clear and completed images are finally generated.

[0108] The steps of the preset first image fusion method may be specifically performed as follows:

[0109] After n filtered low-noise images are divided into several low-noise sub-image blocks according to a preset image size, the frequency intensity value of each low-noise sub-image block in each filtered low-noise image is calculated; for example, 11 filtered low-noise images are divided into 4*4 sizes, and finally each filtered low-noise image can be divided into 16 sub-image blocks, and the frequency intensity value of each sub-image block is calculated. The calculation method of the frequency intensity value can be expressed as:

[0110]

[0111] Among them, F(u, v) small Represents the frequency intensity value of each sub-image block; represents a coefficient constant, which increases the weight of blocks with high frequency domain. f(x, y) represents the grayscale value of a coordinate in the filtered sub-image block. a and b represent the width and height of the sub-image block respectively. u and v represent variables in the frequency domain. j represents the imaginary unit. e represents the base of the natural logarithm.

[0112] Afterwards, the n low-noise sub-image blocks at the same position after segmentation are fused according to the RGB values ​​of the pixels of the low-noise sub-image blocks and the corresponding frequency intensity values, and a fused sub-image block is generated; in the present application, it can also be understood that the 11 filtered low-noise images are all segmented into 4*4, each image is segmented into 16 sub-image blocks, and the sub-image blocks at the same position of each filtered low-noise image are fused. For example, taking the sub-image block in the upper left corner as an example, after the 4*4 image segmentation, each image has 16 sub-image blocks, and the set of sub-image blocks in the upper left corner can be expressed as:

[0113]

[0114] The frequency intensity values ​​calculated for the upper left sub-image block form a set F, which can be expressed as:

[0115]

[0116] For the convenience of calculation, for set F, the corresponding upper left corner sub-image block set img can be fused into set F to facilitate image fusion. The calculation method of the fused sub-image block includes:

[0117]

[0118] Among them, F (i) represents the frequency intensity value of the i-th low-noise sub-image block at the same position after segmentation, img (i) (x, y) represents the RGB value of the pixel coordinates of the i-th low-noise sub-image block at the same position after segmentation, and n is the number of low-noise sub-image blocks at the same position after segmentation.

[0119] Taking 4*4 as an example, 11 sub-image blocks at 16 corresponding positions are fused according to the RGB value fusion method to generate 16 fused sub-image blocks. After that, all the fused sub-image blocks are spliced ​​to generate a fused image, that is, the 16 fused sub-image blocks are spliced ​​to generate the final fused image.

[0120] According to the image fusion method, 4*4, 8*8 and 16*16 segmentation and fusion processing are performed in sequence to obtain the first, second and third fused images.

[0121] In the present application scheme, the image blocks obtained by different specifications of division finally display different details. After division by blocks, the image blocks at the same position are fused, and then the fused images are fused again, which improves the clarity of the fused image and makes the fused and completed image more accurate, which helps to improve the accuracy of the model during the model training process.

[0122] In one embodiment, the method of constructing a neural network model can be specifically implemented as follows:

[0123] Reference Figure 2 , construct the neural network model structure and loss function. The structure of the neural network model includes convolutional layers, multi-head attention mechanism layers, pooling layers, upsampling layers and fully connected layers; specifically, the constructed neural network model includes 3 convolutional layers, several multi-head attention mechanism layers, one pooling layer, two upsampling layers and one fully connected layer.

[0124] When using this network model for processing, one of the 11 filtered low-noise images after denoising is used as the input image. Taking an image with a size of 1200*1200*3 as an example, after the first convolution, the data size is 600*600*6, and after the second convolution, the data size is 300*300*12. Then the data is convolved for the third time, and the data size becomes 150*150*24. Then the data is put into a set composed of eight multi-head attention mechanism layers, and the external The remaining 10 filtered low-noise images are used as supplementary external data input, and the data dimension becomes 150*150*128. The data is then pooled. After the operation, the data becomes 300*300*32. The data is then upsampled for the first time, and the first upsampled data is superimposed with the data of the third convolutional layer. The data dimension is 600*600*6. The second upsampling is then started, and the data of the second convolutional layer is added. The data dimension becomes 1200*1200*3, and then the image is output after passing through the fully connected layer.

[0125] In the embodiment of the present application, the constructed neural network integrates the convolution layer, wherein the size of the convolution kernel is 3*3, the step size is 2, and the nonlinear transformation; 8 multi-head attention mechanism layer sets, pooling layer, upsampling layer, the role of the convolution layer is to perform local perception and feature extraction on the input data and use nonlinear transformation in it, so that the neural network can fit and express complex nonlinear relationships; and the multi-head attention mechanism layer set is the core part inside, considering that the demand is to complete the image, and the multi-head attention mechanism has a very large perception field of view, and because of the multi-head reason, it can extract multi-dimensional feature information, so that the entire model can better understand the data from multiple dimensions; at the same time, the multi-head attention mechanism layer is calculated in parallel, which can save computing resources. In addition, when using the multi-head attention mechanism layer for calculation, during the calculation, one of the 11 filtered low-noise images obtained is used as the input image, and at the same time, the other ten pictures of the same field of view are supplemented as external data input to ensure that the data volume is reorganized during training and the global feature data is not missed. After that, the pooling layer modifies the data dimension to facilitate upsampling. Upsampling gradually restores the data dimension based on the previous data. Finally, all features are mapped to the sample label space through the fully connected layer and the final result is output.

[0126] During verification, 20% of the data, that is, the data of 20,000 lesion areas, were used for verification. The loss function constructed during the verification process can be expressed as:

[0127]

[0128] Among them, p represents the total number of validation set image data, Z 生i represents the i-th image generated by the neural network model, Z 原i represents the clear padded image corresponding to the i-th image generated by the neural network model, δ represents a constant, specifically a variable constant, and in the embodiment of the present application, the value is set to 5 in the operation of skin fluorescence. The constructed backbone feature extraction network is trained using the above formula to obtain a trained backbone feature extraction network.

[0129] The training data is the data processed according to the above steps. The training cycle epochs is set to 2000, the batch size is set to 128, the initial learning rate is set to 0.01, and the learning rate decay strategy is to multiply the original learning rate by 0.1 when the epoch is 100 and 1000, that is, at the 100th and 1000th epochs, the learning rates are 0.001 and 0.0001 respectively. During the training process, we calculate the value of Loss. If the Loss value is less than 10, it stops and it is considered that the model can generate a complete picture. If it is greater than 10, it continues to execute until the epoch reaches two thousand.

[0130] In the present application, the setting of a multi-head attention mechanism layer helps to extract image features and multi-dimensional feature information, so that the entire model can better understand the data from multiple dimensions, improve the accuracy of the model in completing the image, and help to fully display the hyphae and spores contained in the skin image of the lesion, which is beneficial to the subsequent detection and identification of fungi by clinicians or artificial intelligence and further interpretation of the disease.

[0131] Figure 1 FIG. 1 is a flow chart of a skin image completion method in one embodiment. It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but the steps are not necessarily executed in the order indicated by the arrows; unless otherwise specified in this document, there is no strict order restriction for the execution of the steps, and the steps may be executed in other orders; and Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0132] Based on the above method, an embodiment of the present application also discloses a skin image completion device.

[0133] Reference Figure 3 , the device includes the following modules:

[0134] The sample image acquisition module 301 is used to acquire lesion sample images of c lesion regions, each lesion region including m lesion sample images;

[0135] The sample image denoising module 302 is used to perform denoising on the m lesion sample images of each lesion area and generate n filtered low-noise images, where n is less than m;

[0136] A low-noise image fusion module 303 is used to fuse n filtered low-noise images of each lesion area to generate a clear and complete image;

[0137] An image data division module 304 is used to divide the filtered low-noise images of the c lesion areas and the corresponding clear padded images into training set image data and verification set image data according to a preset ratio;

[0138] The completion model generation module 305 is used to construct a neural network model, use the training set image data for training, use the verification set image data for verification, and set the trained and verified neural network model as a skin image completion model;

[0139] The skin image completion module 306 is used to input the skin image to be completed into the skin image completion model to complete the skin image when receiving the skin image to be completed.

[0140] In one embodiment, the sample image denoising module 302 is specifically used to calculate the frequency intensity of the lesion sample image of each lesion area at a preset frequency, and to screen n frequency-screened images of each lesion area that meet preset conditions according to the frequency intensity; the n frequency-screened images of each lesion area are filtered and n filtered low-noise images are generated.

[0141] In one embodiment, the sample image denoising module 302 is specifically used to grayscale the m lesion sample images of each lesion area and generate m lesion grayscale images; convert the m lesion grayscale images of each lesion area from the spatial domain to the frequency domain representation; calculate the frequency intensity of the m lesion grayscale images of each lesion area at a preset frequency and generate a frequency intensity set; filter the image with the highest frequency intensity value in the frequency intensity set; set the image with the highest frequency intensity value and several images before and after the image with the highest frequency intensity value as frequency screening images, wherein the total number of frequency screening images for each lesion area is n, and n is less than m.

[0142] In one embodiment, the sample image denoising module 302 is specifically used to create a neighborhood window for each pixel in the n frequency-filtered images of each lesion area; calculate the weight of each pixel in the neighborhood window, and generate a Gaussian kernel according to the weight of each pixel in the neighborhood window; use the Gaussian kernel to update the RGB value of each pixel in the n frequency-filtered images of each lesion area; set the updated image as a filtered low-noise image, and each lesion area contains n filtered low-noise images.

[0143] In one embodiment, the low-noise image fusion module 303 is specifically used to divide the n filtered low-noise images according to a preset first image size and generate a plurality of first low-noise sub-image blocks, and fuse the plurality of first low-noise sub-image blocks according to a preset first image fusion method to generate a first fused image; divide the n filtered low-noise images according to a preset second image size and generate a plurality of second low-noise sub-image blocks, and fuse the plurality of second low-noise sub-image blocks according to a preset first image fusion method to generate a second fused image; divide the n filtered low-noise images according to a preset third image size and generate a plurality of third low-noise sub-image blocks, and fuse the plurality of third low-noise sub-image blocks according to a preset first image fusion method to generate a third fused image; fuse the first fused image, the second fused image and the third fused image according to a preset weight, and generate a final clear complement image; perform the same image fusion process on the n filtered low-noise images of each lesion area, and finally generate c clear complement images.

[0144] In one embodiment, the low-noise image fusion module 303 is specifically used to calculate the frequency intensity value of each low-noise sub-image block in each filtered low-noise image after segmenting the n filtered low-noise images according to a preset image size to generate a number of low-noise sub-image blocks; and to fuse the n low-noise sub-image blocks at the same position after segmentation according to the RGB values ​​of the pixels of the low-noise sub-image blocks and the corresponding frequency intensity values, and generate a fused sub-image block. The calculation method of the fused sub-image block includes:

[0145]

[0146] Among them, F (i) represents the frequency intensity value of the i-th low-noise sub-image block at the same position after segmentation, img (i) (x, y) represents the RGB value of the pixel coordinates of the i-th low-noise sub-image block at the same position after segmentation, and n is the number of low-noise sub-image blocks at the same position after segmentation; all fused sub-image blocks are spliced ​​to generate a fused image.

[0147] In one embodiment, the filling model generation module 305 is specifically used to construct a neural network model structure and a loss function. The structure of the neural network model includes a convolution layer, a multi-head attention mechanism layer, a pooling layer, an upsampling layer and a fully connected layer; the loss function includes:

[0148]

[0149] Among them, p represents the total number of validation set image data, Z 生i represents the i-th image generated by the neural network model, Z 原i represents the sharp padded image corresponding to the image generated by the neural network model, and δ represents a constant.

[0150] The skin image completion device provided in the embodiment of the present application can be applied to the skin image completion method provided in the above embodiment. For relevant details, refer to the above method embodiment. The implementation principle and technical effect are similar and will not be repeated here.

[0151] It should be noted that: when the skin image completion device provided in the embodiment of the present application performs skin image completion, only the division of the above-mentioned functional modules / functional units is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules / functional units as needed, that is, the internal structure of the skin image completion device is divided into different functional modules / functional units to complete all or part of the functions described above. In addition, the implementation method of the skin image completion method provided in the above-mentioned method embodiment and the implementation method of the skin image completion device provided in this embodiment belong to the same concept. The specific implementation process of the skin image completion device provided in this embodiment is detailed in the above-mentioned method embodiment, and will not be repeated here.

[0152] The embodiment of the present application also discloses a computer device.

[0153] Specifically, if Figure 4As shown, the computer device can be a computer device such as a desktop computer, a laptop computer, a PDA, and a cloud server. The computer device may include, but is not limited to, a processor and a memory. The processor and the memory may be connected via a bus or otherwise. The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, graphics processors (GPU), embedded neural network processors (NPU) or other dedicated deep learning coprocessors, discrete gates or transistor logic devices, discrete hardware components and other chips, or a combination of the above-mentioned various chips.

[0154] As a non-transient computer-readable storage medium, the memory can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the methods in the above-mentioned embodiments of the present application. The processor executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory, that is, the method in the above-mentioned method implementation is realized. The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0155] The embodiment of the present application also discloses a computer-readable storage medium.

[0156] Specifically, a computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method in the above-mentioned method implementation is implemented. Those skilled in the art will understand that the implementation of all or part of the process in the above-mentioned implementation method of the present application can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and the program, when executed, may include the process of the implementation of the above-mentioned methods. Among them, the storage medium may be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated as: HDD) or a solid-state drive (SSD), etc.; the storage medium may also include a combination of the above-mentioned types of memory.

[0157] This specific embodiment is merely an explanation of the present invention and is not a limitation of the present invention. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed. However, as long as they are within the scope of the claims of the present invention, they are protected by the patent law.

Claims

1. A skin image completion method, characterized in that: The method comprises: Acquire c lesion sample images of lesion regions, each of the lesion regions including m lesion sample images; Performing denoising processing on the m lesion sample images of each lesion area and generating n filtered low-noise images, wherein n is less than m; Performing image fusion on the n filtered low-noise images of each lesion area to generate a clear and completed image; Dividing the c filtered low-noise images of the lesion area and the corresponding clear padded images into training set image data and verification set image data according to a preset ratio; Constructing a neural network model, using the training set image data for training, using the verification set image data for verification, and setting the trained and verified neural network model as a skin image completion model; When a skin image to be completed is received, the skin image to be completed is input into the skin image completion model to complete the skin image.

2. The method according to claim 1, characterized in that: The performing denoising on the m lesion sample images of each lesion area and generating n filtered low-noise images comprises: Calculating the frequency intensity of the lesion sample image of each lesion area at a preset frequency, and screening n frequency screening images of each lesion area that meet preset conditions according to the frequency intensity; The n frequency-screened images of each lesion area are filtered to generate n filtered low-noise images.

3. The method according to claim 2, characterized in that: The step of calculating the frequency intensity of the lesion sample image of each lesion area at a preset frequency, and screening n frequency screening images of each lesion area that meet preset conditions according to the frequency intensity comprises: Grayscale processing is performed on the m lesion sample images of each lesion area to generate m lesion grayscale images; Converting the m lesion grayscale images of each lesion area from the spatial domain to the frequency domain; Calculating the frequency intensity of the m lesion grayscale images of each lesion area at a preset frequency, and generating a frequency intensity set; Selecting an image with the highest frequency intensity value in the frequency intensity set; The image with the highest frequency intensity value and several images before and after the image with the highest frequency intensity value are set as the frequency screening images, wherein the total number of frequency screening images for each lesion area is n, and n is less than m.

4. The method according to claim 2, characterized in that: The filtering process of the n frequency-screened images of each lesion area to generate n filtered low-noise images comprises: Creating a neighborhood window for each pixel point in the n frequency-screened images of each lesion area; Calculating the weight of each pixel in the neighborhood window, and generating a Gaussian kernel according to the weight of each pixel in the neighborhood window; Using the Gaussian kernel, the RGB value of each pixel in the n frequency-screened images of each lesion area is updated; The updated image is set as a filtered low-noise image, and each of the lesion areas contains n filtered low-noise images.

5. The method according to claim 1, characterized in that: The step of fusing the n filtered low-noise images of each lesion area to generate a clear and completed image comprises: Segmenting the n filtered low-noise images according to a preset first image size to generate a plurality of first low-noise sub-image blocks, and fusing the plurality of the first low-noise sub-image blocks according to a preset first image fusion method to generate a first fused image; Segmenting the n filtered low-noise images according to a preset second image size to generate a plurality of second low-noise sub-image blocks, and fusing the plurality of second low-noise sub-image blocks according to a preset first image fusion method to generate a second fused image; Segmenting the n filtered low-noise images according to a preset third image size to generate a plurality of third low-noise sub-image blocks, and fusing the plurality of third low-noise sub-image blocks according to a preset first image fusion method to generate a third fused image; fusing the first fused image, the second fused image and the third fused image according to a preset weight, and generating a final clear complemented image; The same image fusion process is performed on the n filtered low-noise images of each lesion area to finally generate c clear and completed images.

6. The method according to claim 5, characterized in that: The preset first image fusion method includes: After the n filtered low-noise images are divided into a plurality of low-noise sub-image blocks according to a preset image size, a frequency intensity value of each low-noise sub-image block in each of the filtered low-noise images is calculated; The n low-noise sub-image blocks at the same position after segmentation are fused according to the RGB values ​​of the pixels of the low-noise sub-image blocks and the corresponding frequency intensity values, and a fused sub-image block is generated. The calculation method of the fused sub-image block includes: Among them, F (i) represents the frequency intensity value of the i-th low-noise sub-image block at the same position after segmentation, img (i) (x, y) represents the RGB value of the pixel coordinates of the i-th low-noise sub-image block at the same position after segmentation, and n is the number of low-noise sub-image blocks at the same position after segmentation; All the fused sub-image blocks are spliced ​​together to generate a fused image.

7. The method according to claim 1, characterized in that: The construction of the neural network model comprises: Constructing a neural network model structure and a loss function, wherein the structure of the neural network model includes a convolutional layer, a multi-head attention mechanism layer, a pooling layer, an upsampling layer, and a fully connected layer; The loss function includes: Where p represents the total number of validation set image data, Z 生i represents the i-th image generated by the neural network model, Z 原i represents a clear padded image corresponding to the image generated by the neural network model, and δ represents a constant.

8. A skin image completion device, characterized in that: The device comprises: A sample image acquisition module (301) is used to acquire lesion sample images of c lesion regions, each of which includes m lesion sample images; A sample image denoising module (302), used for performing denoising processing on the m lesion sample images of each lesion area and generating n filtered low-noise images, wherein n is less than m; A low-noise image fusion module (303) is used to fuse the n filtered low-noise images of each lesion area to generate a clear and complete image; An image data division module (304) is used to divide the c filtered low-noise images of the lesion area and the corresponding clear padded images into training set image data and verification set image data according to a preset ratio; A completion model generation module (305) is used to construct a neural network model, use the training set image data for training, use the verification set image data for verification, and set the trained and verified neural network model as a skin image completion model; The skin image completion module (306) is used to input the skin image to be completed into the skin image completion model to complete the skin image when receiving the skin image to be completed.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program which can be loaded by the processor and executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 7.

Images