Skin image completion methods, devices, equipment and storage media

Abstract

This application relates to a method, apparatus, device, and storage medium for skin image completion, applied in the field of image processing. The method includes acquiring lesion sample images of c lesion regions, each lesion region comprising m lesion sample images; denoising the m lesion sample images of each lesion region to generate n filtered low-noise images; fusing the n filtered low-noise images of each lesion region to generate a clear completed image; dividing the filtered low-noise images and corresponding clear completed images of the c lesion regions into training set data and validation set data according to a preset ratio; constructing a neural network model and training and validating it, setting the trained and validated neural network model as a skin image completion model; and inputting the skin image to be completed into the skin image completion model upon receiving a skin image to be completed. The technical effect of this application is that skin image completion aids in fungal detection and identification.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, device and storage medium for skin image completion. Background Technology

[0002] There are approximately 300 known species of fungi that are pathogenic to humans. Severe fungal infections can lead to disseminated infection throughout the body. Therefore, early diagnosis and timely treatment of fungal infections are crucial for reducing mortality. In clinical diagnosis and treatment, fluorescence detection methods have developed rapidly, and the staining procedure is relatively simple, offering significant advantages over traditional microscopic methods. This has led to the increasingly widespread application of fluorescence detection in the diagnosis of fungal infections.

[0003] After fluorescent staining, fungi (including hyphae and spores) appear clearly bright blue under a microscope. In most cases, the septate structure of the hyphae and the budding morphology of the spores can be clearly seen, thus enabling effective detection of fungi. However, due to the thickness of skin samples, existing detection methods sometimes only capture a portion of the target in some skin images, resulting in incomplete display of hyphae and spores, making it difficult to detect and identify fungi based on the detected hyphae and spores. Therefore, there is an urgent need for a skin image completion method that can complete the skin image to fully display hyphae and spores, thus aiding in further interpretation of image recognition results by clinicians or artificial intelligence. Summary of the Invention

[0004] To address the problem in existing detection methods that only a portion of the target is captured in some skin images, resulting in incomplete display of hyphae and spores and making it difficult to detect and identify fungi based on the detected hyphae and spores, this application provides a skin image completion method, apparatus, device, and storage medium.

[0005] Firstly, this application provides a method for skin image completion, employing the following technical solution: the method includes:

[0006] Acquire lesion sample images of c lesion regions, where each lesion region includes m lesion sample images;

[0007] For each lesion region, m sample images of the lesion are denoised to generate n filtered low-noise images, where n is less than m;

[0008] The n filtered low-noise images of each lesion region are fused to generate a clear and complete image;

[0009] The filtered low-noise images of c lesion regions and their corresponding clear and complete images are divided into training set image data and validation set image data according to a preset ratio;

[0010] A neural network model is constructed, trained using the image data in the training set, validated using the image data in the validation set, and the trained and validated neural network model is set as a skin image completion model.

[0011] Upon receiving a skin image to be filled in, the skin image to be filled in is input into the skin image filling model to fill in the skin image.

[0012] In one specific implementation scheme, the step of denoising m lesion sample images for each lesion region and generating n filtered low-noise images includes:

[0013] Calculate the frequency intensity of the lesion sample image of each lesion region at a preset frequency, and filter n frequency-filtered images of each lesion region that meet the preset conditions based on the frequency intensity;

[0014] The n frequency-selected images for each lesion region are filtered to generate n filtered low-noise images.

[0015] In one specific implementation, calculating the frequency intensity of the lesion sample image of each lesion region at a preset frequency, and filtering n frequency-selected images of each lesion region that meet the preset conditions based on the frequency intensity, includes:

[0016] Each of the m lesion sample images in each lesion region is converted to grayscale, and m grayscale images of the lesion are generated.

[0017] Convert the m grayscale images of each lesion region from the spatial domain to the frequency domain representation;

[0018] Calculate the frequency intensity of m grayscale images of each lesion region at a preset frequency, and generate a set of frequency intensities;

[0019] Filter the image with the highest frequency intensity value from the set of frequency intensity values;

[0020] The image with the highest frequency intensity value, along with several images before and after the image with the highest frequency intensity value, are set as the frequency screening images. The total number of frequency screening images for each lesion region is n, where n is less than m.

[0021] In one specific implementation, the step of filtering the n frequency-selected images for each lesion region and generating n filtered low-noise images includes:

[0022] For each pixel in the n frequency-filtered images of each lesion region, a neighborhood window is created;

[0023] Calculate the weight of each pixel in the neighborhood window, and generate a Gaussian kernel based on the weight of each pixel in the neighborhood window;

[0024] The RGB values ​​of each pixel in the n frequency-filtered images for each lesion region are updated using the Gaussian kernel;

[0025] The updated image is set as a filtered low-noise image, and each lesion region contains n filtered low-noise images.

[0026] In one specific implementation scheme, the step of fusing the n filtered low-noise images of each lesion region to generate a clear and complete image includes:

[0027] The n filtered low-noise images are divided into several first low-noise sub-image blocks according to a preset first image size, and the several first low-noise sub-image blocks are fused according to a preset first image fusion method to generate a first fused image.

[0028] The n filtered low-noise images are divided into several second low-noise sub-image blocks according to a preset second image size, and the several second low-noise sub-image blocks are fused according to a preset first image fusion method to generate a second fused image;

[0029] The n filtered low-noise images are divided into several third low-noise sub-image blocks according to a preset third image size, and the several third low-noise sub-image blocks are fused according to a preset first image fusion method to generate a third fused image.

[0030] The first fused image, the second fused image, and the third fused image are fused according to preset weights to generate the final clear and complete image;

[0031] The same image fusion process is performed on n filtered low-noise images of each lesion region to finally generate c clear and complete images.

[0032] In one specific implementation scheme, the preset first image fusion method includes:

[0033] After dividing the n filtered low-noise images into several low-noise sub-image blocks according to a preset image size, the frequency intensity value of each low-noise sub-image block in each filtered low-noise image is calculated.

[0034] The n low-noise sub-image blocks at the same location after segmentation are fused according to the RGB values ​​of the pixels of the low-noise sub-image blocks and the corresponding frequency intensity values ​​to generate a fused sub-image block. The calculation method of the fused sub-image block includes:

[0035]

[0036] Among them, F (i) This represents the frequency intensity value of the i-th low-noise sub-image patch at the same location after segmentation, img (i) (x, y) represents the RGB value of the pixel coordinates of the i-th low-noise sub-image block at the same position after segmentation, and n is the number of low-noise sub-image blocks at the same position after segmentation.

[0037] All the fused sub-image blocks are stitched together to generate a fused image.

[0038] In one specific implementation, the construction of the neural network model includes:

[0039] Construct a neural network model structure and loss function. The structure of the neural network model includes convolutional layers, multi-head attention mechanism layers, pooling layers, upsampling layers, and fully connected layers.

[0040] The loss function includes:

[0041]

[0042] Where p represents the total number of image data in the verification set, Z 生i Z represents the i-th image generated by the neural network model. 原i The image represents the sharp, filled image corresponding to the image generated by the neural network model, and δ represents a constant.

[0043] Secondly, this application provides a skin image completion device, which adopts the following technical solution: the device includes:

[0044] The sample image acquisition module is used to acquire lesion sample images of c lesion regions, each lesion region including m lesion sample images;

[0045] The sample image denoising module is used to denoise m sample images of each lesion region and generate n filtered low-noise images, where n is less than m;

[0046] A low-noise image fusion module is used to fuse n filtered low-noise images of each lesion region to generate a clear and complete image.

[0047] The image data partitioning module is used to divide the filtered low-noise images and corresponding clear and complete images of the c lesion regions into training set image data and validation set image data according to a preset ratio.

[0048] The skin image completion model generation module is used to construct a neural network model. It is trained using the training set image data and validated using the validation set image data. The trained and validated neural network model is then set as the skin image completion model.

[0049] The skin image completion module is used to input the skin image to be completed into the skin image completion model when a skin image to be completed is received.

[0050] Thirdly, this application provides a computer device that adopts the following technical solution: it includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed as any of the skin image completion methods described above.

[0051] Fourthly, this application provides a computer-readable storage medium that stores a computer program capable of being loaded by a processor and executing any of the above-mentioned skin image completion methods.

[0052] In summary, this application has the following beneficial technical effects:

[0053] By acquiring a large number of original lesion sample images, the lesion sample images are processed to generate completed skin images. A neural network model is then trained and learned on the known completed skin images, and the model is validated to generate the final skin image completion model. This allows the incomplete display of hyphae and spores in the skin image to be completed after fluorescent staining, so that hyphae and spores can be fully displayed. This helps to improve the accuracy of fungal detection and identification, and makes it easier for clinicians to interpret the results of fungal detection and identification. Attached Figure Description

[0054] Figure 1 This is a flowchart of the skin image completion method in the embodiments of this application;

[0055] Figure 2 This is a schematic diagram of the neural network model structure in an embodiment of this application;

[0056] Figure 3 This is a schematic diagram of the skin image completion device in the embodiments of this application;

[0057] Figure 4 This is a schematic diagram used to illustrate a computer device in the embodiments of this application.

[0058] Figure labels: 301, Sample image acquisition module; 302, Sample image denoising module; 303, Low-noise image fusion module; 304, Image data partitioning module; 305, Completion model generation module; 306, Skin image completion module. Detailed Implementation

[0059] The following combination Figures 1-4 This application will be described in further detail.

[0060] This application discloses a skin image completion method. This method can complete the skin image so that hyphae and spores in the skin image can be fully displayed, which can facilitate further interpretation of the results by clinicians or artificial intelligence.

[0061] There are approximately 300 known species of fungi that are pathogenic to humans. Clinically, based on the location of infection, pathogenic fungi are divided into superficial and deep fungi. Superficial fungi (dermatophytes) only affect the skin, hair, and nails, while deep fungi can invade mucous membranes, deep tissues, and internal organs, and in severe cases, can even cause disseminated infection. Therefore, early diagnosis and timely treatment of fungal infections are crucial for reducing mortality. Dermatophyte infections are a large category of infectious diseases affecting the skin, appendages, subcutaneous tissues, and mucous membranes caused by pathogenic fungi. They are among the most common types of skin diseases, often presenting clinically as vesicular-squamous symptoms. Dermatophytes are the main pathogens causing fungal skin diseases, including species such as Trichophyton flavum, Trichophyton mentagrophytes, Trichophyton rubrum, Trichophyton mentagrophytes, Trichophyton verrucous, and Microsporum. This disease is characterized by a high incidence rate, strong infectivity, and a high rate of reinfection or recurrence after treatment, significantly impacting patients' daily lives and mental well-being. In recent years, the incidence of fungal infections has increased, and secondary fungal infections are also on the rise. Therefore, effective fungal detection is particularly important in clinical diagnosis and treatment.

[0062] Currently, mainstream fungal detection methods typically employ wet mount microscopy, Gram staining, fungal culture, and fluorescence detection. Wet mount microscopy is simple to perform, but it requires highly skilled personnel; only specialized technicians can ensure a high detection rate, making widespread adoption difficult. Gram staining is complex and time-consuming, and is easily affected by factors such as stain quality and the skill level of the technician, thus it is rarely used clinically. Fungal culture is also time-consuming, potentially causing patients to miss the optimal treatment window. Fluorescence detection has seen rapid development in recent years; the staining process is relatively simple, offering significant advantages over traditional microscopy methods, and its application in the diagnosis of fungal infections has become increasingly widespread.

[0063] After fluorescent staining, fungi (including hyphae and spores) appear as a clear bright blue color under a microscope. In most cases, the septate structure of the hyphae and the budding morphology of the spores can be clearly seen. Especially in KOH smears, fungal spores and lipid droplets are sometimes difficult to distinguish, but fluorescent staining does not label lipid droplets, which greatly improves specificity and overcomes the deficiency of low detection rate of traditional KOH smears when the number of fungi is small.

[0064] However, due to the thickness of skin samples, existing detection methods can only capture a portion of the target in some skin images. For example, when capturing long hyphae, only one-fifth or even one-tenth of the hyphae can be seen, resulting in incomplete display of hyphae and spores. This makes it difficult to detect and identify fungi based on the detected hyphae and spores. To help fully display hyphae and spores and improve the detection rate of fungi, this application provides a skin image completion method.

[0065] Reference Figure 1 The method includes the following steps:

[0066] S10, acquire lesion sample images of c lesion regions, each lesion region includes m lesion sample images.

[0067] Specifically, raw lesion sample data is acquired for model training and validation. This raw lesion sample data can be obtained directly from existing databases or by capturing images of known patient lesion areas. Here, c and m are positive integers. In this embodiment, sample data is acquired by capturing images of known patient lesion areas. This application uses 1000 patients as samples, selecting 100 lesion areas from each patient. For each lesion area, 20 images are captured from bottom to top within a preset depth of field with a step size of 2 micrometers. That is, the number of lesion areas c in this application is 1000 * 100 = 100,000, and the number of lesion sample images m for each lesion area is 20. In this embodiment, the values ​​of c and m are only illustrative; users can set the specific values ​​according to actual needs. There are no restrictions here, as long as c and m are positive integers. It should be noted that skin image completion processing is generally performed after fluorescence staining. Therefore, the raw lesion sample images used in this application are assumed to have already undergone fluorescence staining.

[0068] S20, Denoise the m lesion sample images of each lesion region and generate n filtered low-noise images, where n is less than m.

[0069] Specifically, denoising is performed on m lesion sample images for each lesion region. This can also be understood as denoising all m lesion sample images for a single lesion region, ultimately selecting n filtered low-noise images. These n filtered low-noise images are the clear images after denoising. The same denoising filtering process is applied to lesion sample images for other lesion regions to obtain relatively clearer skin images. Here, n is less than m and is also a positive integer. In this application, m is set to 20 and n is set to 11. The specific values ​​can be set by the user according to actual needs and are not limited here. It should be noted that both the original lesion sample images and the denoised filtered low-noise images are incomplete skin images; the hyphae or spores at the lesions in the skin images are not fully displayed.

[0070] S30, performs image fusion on n filtered low-noise images of each lesion area to generate a clear and complete image.

[0071] Specifically, for an image of a lesion area, n filtered, low-noise images are fused together after denoising to generate a complete, clear image. This clear, clear image is a fully completed image, meaning that the hyphae and spores at the lesion site are fully visible, which helps clinicians or artificial intelligence to further interpret the results. The same image fusion process is used for the remaining lesion areas, and the number of clear, clear images generated is consistent with the number of lesion areas. In this application, images of 100,000 lesion areas were selected, resulting in 100,000 clear, clear images.

[0072] S40, the filtered low-noise images of c lesion regions and the corresponding clear and complete images are divided into training set image data and validation set image data according to a preset ratio.

[0073] Specifically, this application selects images of 100,000 lesion regions, ultimately generating 100,000 clear incomplete images. The filtered low-noise images corresponding to the 100,000 lesion regions, and the clear incomplete images generated by fusing the filtered low-noise images, are divided into training set image data and validation set image data according to a preset ratio. In this embodiment, 80% of the data is set as training set data and 20% of the data is set as validation set data. Alternatively, it can be understood that the filtered low-noise images of 80,000 lesion regions and their corresponding clear incomplete images are set as training set data, and the filtered low-noise images of the remaining 20,000 lesion regions and their corresponding clear incomplete images are set as validation set data.

[0074] S50, construct a neural network model, train it using training set image data, validate it using validation set image data, and set the trained and validated neural network model as a skin image completion model.

[0075] Specifically, a neural network model is constructed, and the training set image data is input into the neural network model for model training. After the model is trained, the remaining validation set image data is input into the neural network model for model validation. The model after training and model validation is set as the final skin image completion model.

[0076] It should be noted that model training involves inputting the generated filtered low-noise images and their corresponding sharp, completed images into the model. During the processing, a total of 11 filtered low-noise images were obtained. In the actual training process, one of these 11 images was selected as the model's input image, while the remaining 10 images were used as supplementary external data to extract more details and improve the accuracy of the model training. The input image can be the one with the highest sharpness among the 11 filtered low-noise images.

[0077] S60: When a skin image to be filled is received, the skin image to be filled is input into the skin image filling model for skin image filling.

[0078] Specifically, in practical applications, when a skin image to be filled is received, it is input into a skin image filling model to fill the skin image, and finally a skin image filled in can be easily interpreted by clinicians.

[0079] In this application, a large number of original lesion sample images are acquired, and the lesion sample images are processed to generate completed skin images. A neural network model is used to train and learn from the known completed skin images, and the model is validated to generate the final skin image completion model. This allows the incomplete display of hyphae and spores in the skin image to be completed after fluorescent staining, so that the hyphae and spores can be fully displayed. This helps to improve the accuracy of fungal detection and identification, and makes it easier for clinicians to interpret the fungal detection and identification results.

[0080] In one embodiment, the step of denoising m lesion sample images for each lesion region and generating n filtered low-noise images can be specifically performed as follows:

[0081] First, the frequency intensity of the lesion sample image for each lesion region at a preset frequency is calculated, and n frequency-selected images for each lesion region that meet the preset conditions are selected based on the frequency intensity. Specifically, m lesion sample images for each lesion region are grayscaled to generate m grayscale lesion images; in this application, m is set to 20, which can also be understood as grayscaled processing of 20 lesion sample images for one lesion region to generate 20 grayscale lesion images, and the same processing is performed on lesion sample images for the remaining lesion regions to generate grayscale lesion images. Grayscale processing can reduce the number of data channels, thereby reducing the amount of data computation and improving the data processing speed. Next, the m grayscale lesion images for each lesion region are converted from the spatial domain to the frequency domain representation, that is, the 20 grayscale lesion images are converted to the frequency domain representation. Based on the characteristic that skin fluorescence images have less interference, converting the image from the spatial domain to the frequency domain is beneficial for analyzing the frequency components in the image. Images with high clarity contain more high-frequency components, and these high frequencies are strongly correlated with the texture and edges in the image.

[0082] Next, the frequency intensity of the m grayscale images of each lesion region at a preset frequency is calculated, and a set of frequency intensity values ​​is generated; the calculation method for the frequency intensity values ​​can be expressed as:

[0083]

[0084] Where f(x,y) represents the gray value at a certain coordinate (x,y) after the image is converted to a grayscale image of the lesion, F(u,v) represents the frequency intensity at frequency (u,v) after conversion to the frequency domain, a and b represent the width and height of the image, u and v represent the variables in the frequency domain, j represents the imaginary unit, and e represents the base of the natural logarithm.

[0085] The calculated frequency intensity values ​​are grouped into a frequency intensity set, which can be represented as:

[0086] [F(u,v)1, F(u,v)2, F(u,v)3,…,F(u,v) 20 ],

[0087] That is, after converting 20 grayscale images of a lesion region to the frequency domain, the frequency intensity of the 20 images at a preset frequency (u,v) is calculated, forming the aforementioned frequency intensity set.

[0088] Next, the images with the highest frequency intensity values ​​in the frequency intensity set are selected, that is, those between F(u,v)1 and F(u,v). 20Between these, the image with the highest frequency intensity value is selected; the image with the highest frequency intensity value, along with several images before and after it, are set as frequency-filtered images. The total number of frequency-filtered images for each lesion region is n, where n is less than m. For example, assuming the i-th lesion has the highest frequency intensity value, and the number of images before and after it is set to 10 (5 before and 5 after), then the i-th image and the five images before and after it are selected to form the clearest image set. This image set can be represented as:

[0089] [F(u,v) i-5 , F(u,v) i-4 ,…,F(u,v) i ,…,F(u,v) i+5 ].

[0090] It should be noted that the number of images before and after the image with the highest frequency intensity value can be the same or different, and the set number of images is determined by the user. In practice, if a certain number of images are selected before and after the image, and the number of images before and after the image is insufficient, no supplementary processing is performed. For example, if the i-th image and five images before and after the i-th image are selected to form the clearest image set, and the number of images before or after the i-th image is less than five, no supplementary processing is performed. Furthermore, the frequency-filtered image is the image obtained based on frequency filtering and is still processed in the spatial domain in subsequent processing; it does not represent an image in the frequency domain.

[0091] After filtering the images based on frequency intensity values, the n frequency-filtered images for each lesion region are filtered to generate n filtered low-noise images. Specifically, a neighborhood window is created for each pixel in the n frequency-filtered images for each lesion region. Creating a neighborhood window can be understood as constructing a new coordinate system centered on the pixel. In this application, the new coordinate system is composed of 7*7 points, and the constructed new coordinate system can be shown in Table 1.

[0092] Table 1 shows the new coordinate system formed by 7*7 points.

[0093] (-3,-3) (-3,-2) (-3,-1) (-3,0) (-3,1) (-3,2) (-3,3) (-2,-3) (-2,-2) (-2,-1) (-2,0) (-2,1) (-2,2) (-2,3) (-1,-3) (-1,-2) (-1,-1) (-1,0) (-1,1) (-1,2) (-1,3) (0,-3) (0,-2) (0,-1) (0,0) (0,1) (0,2) (0,3) (1,-3) (1,-2) (1,-1) (1,0) (1,1) (1,2) (1,3) (2,-3) (2,-2) (2,-1) (2,0) (2,1) (2,2) (2,3) (3,-3) (3,-2) (3,-1) (3,0) (3,1) (3,2) (3,3)

[0094] It should be noted that the neighborhood window, i.e. the new coordinate system constructed, can be composed of 7*7 points, 5*5 points, or 3*3 points; the specific size is not limited here.

[0095] Based on the constructed neighborhood window, i.e., the new coordinate system, the weight of each pixel in the neighborhood window is calculated, and a Gaussian kernel is generated according to the weight of each pixel in the neighborhood window. The weight calculation method can be expressed as:

[0096]

[0097] Where σ represents a variable constant, and calculations show that a value of 0.8 yields the best performance for skin fluorescence images; therefore, in this embodiment, the value of σ is set to 0.8; (x,y) represents the coordinates of any pixel in the frequency-selected image; μ x and μ y The field number is represented by μ in this application. x =μ y =0, when μ x =μ y A value of 0 provides the best representation of skin fluorescence images. Furthermore, if pixels at the edges are difficult to form a 7x7 area, the value for the missing area is replaced with 0.

[0098] The Gaussian kernel can be calculated based on a 7x7 region and the calculated weights. In this application, a 7x7 region is used; therefore, the final calculated Gaussian kernel is also a 7x7 matrix. The Gaussian kernel calculated in this application is as follows:

[0099]

[0100] After obtaining the Gaussian kernel, the RGB values ​​of each pixel in the n frequency-selected images for each lesion region are updated using the Gaussian kernel. That is, the RGB values ​​in the 11 frequency-selected images are updated using the Gaussian kernel, and this update process is repeated for all lesion regions. Finally, the updated images are set as filtered low-noise images. Each lesion region contains n filtered low-noise images, which can also be understood as each updated lesion region comprising 11 filtered low-noise images.

[0101] In this application, multiple images with high frequency intensity values ​​are obtained through frequency screening to complete the first denoising screening process, resulting in multiple images with high clarity. Then, the RGB values ​​of each pixel in the image selected based on frequency values ​​are updated using a Gaussian kernel to filter out noise interference caused by external factors such as shooting environment and lighting, thereby completing the second denoising process, improving image quality, and helping to improve the accuracy of model training and validation processes.

[0102] In one embodiment, the step of fusing n filtered low-noise images of each lesion region to generate a clear and complete image can be specifically performed as follows:

[0103] The n filtered low-noise images are divided into several first low-noise sub-image blocks according to a preset first image size. The several first low-noise sub-image blocks are then fused according to a preset first image fusion method to generate a first fused image. The n filtered low-noise images are divided into several second low-noise sub-image blocks according to a preset second image size. The several second low-noise sub-image blocks are then fused according to a preset first image fusion method to generate a second fused image. The n filtered low-noise images are divided into several third low-noise sub-image blocks according to a preset third image size. The several third low-noise sub-image blocks are then fused according to a preset first image fusion method to generate a third fused image. In this application, it can also be understood that for a lesion area, there are 11 filtered low-noise images after denoising. These 11 filtered low-noise images are divided into three segments of different sizes. The size of the segmented images can be set by the user and is not limited here. In this application, the image sizes of the three segments are set to 4*4, 8*8 and 16*16. First, a 4*4 segmentation is performed. After the segmentation, the low-noise sub-image blocks of the 11 filtered low-noise images are fused to generate a first fused image according to a preset first image fusion method. Then, the 8*8 segmentation and fusion operation and the 16*16 segmentation and fusion operation are performed in sequence to generate the second and third fused images. The image segmentation and fusion operation of the 8*8 and 16*16 images is the same as that of the 4*4 segmentation and fusion operation.

[0104] Then, the first, second, and third fused images are fused according to preset weights to generate the final sharp and complete image. The weights can be set by the user according to actual needs. In this application, the image sizes of the three segments are set to 4*4, 8*8, and 16*16, and the weights are set to 0.2, 0.3, and 0.5 respectively. The final sharp and complete image generated by fusion can be represented as follows:

[0105] B 清晰 (x, y) = 0.5 * B 16*16 (x, y) + 0.3*B 8*8 (x, y) + 0.2*B 4*4 (x, y),

[0106] Among them, B 4*4 (x, y) represents the first fused image, B 8*8 (x, y) represents the second fused image, B 16*16 (x, y) represents the third fused image, B 清晰 (x, y) represents the final sharp, completed image.

[0107] A lesion region contains 11 filtered low-noise images. The 11 filtered low-noise images are finally fused to generate one clear and complete image. Therefore, by performing the same image fusion process on n filtered low-noise images of each lesion region, c clear and complete images can be generated. In this application, 100,000 lesion regions were selected, which means that 100,000 clear and complete images were generated in the end.

[0108] The steps of the preset first image fusion method can be specifically executed as follows:

[0109] After dividing n filtered low-noise images into several low-noise sub-image blocks according to a preset image size, the frequency intensity value of each low-noise sub-image block in each filtered low-noise image is calculated. For example, if 11 filtered low-noise images are divided into 4*4 blocks, each filtered low-noise image can be divided into 16 sub-image blocks. The frequency intensity value of each sub-image block is calculated, and the calculation method of the frequency intensity value can be expressed as:

[0110]

[0111] Where, F(u, v) small This represents the frequency intensity value of each sub-image patch; denoted as a constant coefficient, which increases the weight of blocks with higher frequency domain. f(x, y) represents the gray value of a certain coordinate in the filtered sub-image block. a and b represent the width and height of the sub-image block, respectively. u and v represent variables in the frequency domain. j represents the imaginary unit. e represents the base of the natural logarithm.

[0112] Next, the n low-noise sub-image blocks at the same location after segmentation are fused based on the RGB values ​​of the pixels and the corresponding frequency intensity values ​​to generate a fused sub-image block. In this application, it can also be understood that all 11 filtered low-noise images are segmented into 4*4, and each image is divided into 16 sub-image blocks. The sub-image blocks at the same location in each filtered low-noise image are fused. For example, taking the sub-image block in the upper left corner as an example, after 4*4 image segmentation, each image has 16 sub-image blocks. The set of sub-image blocks in the upper left corner can be represented as:

[0113]

[0114] The frequency intensity values ​​calculated from the top-left sub-image patch form a set F, which can be represented as:

[0115]

[0116] To facilitate computation, for set F, the corresponding top-left sub-image patch set `img` can be merged into set F to perform image fusion. The calculation methods for merging sub-image patches include:

[0117]

[0118] Among them, F (i) This represents the frequency intensity value of the i-th low-noise sub-image patch at the same location after segmentation, img (i) (x,y) represents the RGB value of the pixel coordinates of the i-th low-noise sub-image block at the same position after segmentation, and n is the number of low-noise sub-image blocks at the same position after segmentation.

[0119] Taking 4x4 as an example, the 11 sub-image blocks at 16 corresponding positions are fused according to the RGB value fusion method to generate 16 fused sub-image blocks. Then, all the fused sub-image blocks are stitched together to generate a fused image, that is, the 16 fused sub-image blocks are stitched together to generate the final fused image.

[0120] Following this image fusion method, 4*4, 8*8, and 16*16 segmentation and fusion processes are performed sequentially to obtain the first, second, and third fused images.

[0121] In this application, the details of the image blocks obtained by different partitioning methods are different. After partitioning by blocks, the image blocks at the same position are fused, and then the images fused by blocks are fused a second time, which improves the clarity of the fused image and makes the image after fusion more accurate, which helps to improve the accuracy of the model during the model training process.

[0122] In one embodiment, the construction of a neural network model can be specifically performed as follows:

[0123] Reference Figure 2 The neural network model structure and loss function are constructed. The structure of the neural network model includes convolutional layers, multi-head attention mechanism layers, pooling layers, upsampling layers, and fully connected layers. Specifically, the constructed neural network model includes 3 convolutional layers, several multi-head attention mechanism layers, one pooling layer, two upsampling layers, and one fully connected layer.

[0124] When using this network model for processing, one of the 11 filtered low-noise images after denoising is used as the input image. Taking an image with a size of 1200*1200*3 as an example, after the first convolution, the data size is 600*600*6. After the second convolution, the data size is 300*300*12. Then, the data undergoes a third convolution, and the data size becomes 150*150*24. The data is then fed into a set consisting of eight multi-head attention mechanism layers, and external attention mechanisms are added. The remaining 10 filtered, low-noise images are used as supplementary external data inputs, resulting in a data dimension of 150*150*128. The data is then pooled, resulting in a data dimension of 300*300*32. The data is then upsampled for the first time, and the upsampled data is superimposed with the data from the third convolutional layer, resulting in a data dimension of 600*600*6. A second upsampling is then performed, and data from the second convolutional layer is added, resulting in a data dimension of 1200*1200*3. Finally, the image is output after passing through a fully connected layer.

[0125] In this embodiment, the constructed neural network integrates convolutional layers with a kernel size of 3*3 and a stride of 2, employing nonlinear transformations; a set of eight multi-head attention mechanism layers; pooling layers; and upsampling layers. The convolutional layers perform local perception and feature extraction on the input data, using nonlinear transformations to enable the neural network to fit and express complex nonlinear relationships. The multi-head attention mechanism layers are the core component. Considering the requirement of image completion, the multi-head attention mechanism has a very large receptive field, and due to the multiple heads, it can extract multi-dimensional feature information, allowing the entire model to better understand the data from multiple dimensions. Simultaneously, the multi-head attention mechanism layers are computed in parallel, saving computational resources. Furthermore, when using the multi-head attention mechanism layers for computation, one of the eleven filtered low-noise images is used as the input image, while the other ten images in the same field of view are used as external data inputs to supplement the input, ensuring data reorganization during training and that no global feature data is missed. The pooling layer then modifies the data dimensions to facilitate upsampling. Upsampling gradually restores the data dimensions based on the previous data. Finally, the fully connected layer maps all the features to the sample label space and outputs the final result.

[0126] During validation, 20% of the data, i.e., 20,000 lesion areas, were used. The loss function constructed during the validation process can be expressed as:

[0127]

[0128] Where p represents the total number of image data in the validation set, Z 生i Z represents the i-th image generated by the neural network model. 原i This represents the clear, filled-in image corresponding to the i-th image generated by the neural network model. δ represents a constant, specifically a variable constant, which is set to 5 in the skin fluorescence operation in this embodiment. The constructed backbone feature extraction network is trained using the above formula to obtain the trained backbone feature extraction network.

[0129] The training data is the data processed according to the above steps. The training epochs are set to 2000, the batch size is set to 128, and the initial learning rate is set to 0.01. The learning rate decay strategy is to multiply the original learning rate by 0.1 at epochs 100 and 1000, that is, at the 100th and 1000th epochs, the learning rates are 0.001 and 0.0001, respectively. During the training process, we calculate the loss value. If the loss value is less than 10, we stop, assuming that the model can generate complete images. If it is greater than 10, we continue to execute until the epoch reaches 2000.

[0130] In this application, the multi-head attention mechanism layer helps to extract image features and multi-dimensional feature information, enabling the model to better understand the data from multiple dimensions. This improves the accuracy of the model in completing the image and helps to fully display the hyphae and spores contained in the skin image of the lesion. This is beneficial for subsequent detection and identification of fungi by clinicians or artificial intelligence and further interpretation of the disease.

[0131] Figure 1 This is a flowchart illustrating a skin image completion method in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows; unless explicitly stated otherwise, there is no strict order requirement for the execution of these steps, and they can be executed in other orders; and Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0132] Based on the above method, this application also discloses a skin image completion device.

[0133] Reference Figure 3 The device includes the following modules:

[0134] The sample image acquisition module 301 is used to acquire lesion sample images of c lesion regions, each lesion region including m lesion sample images;

[0135] The sample image denoising module 302 is used to denoise m lesion sample images for each lesion region and generate n filtered low-noise images, where n is less than m;

[0136] The low-noise image fusion module 303 is used to fuse n filtered low-noise images of each lesion area to generate a clear and complete image.

[0137] The image data partitioning module 304 is used to divide the filtered low-noise images of c lesion regions and the corresponding clear and complete images into training set image data and validation set image data according to a preset ratio.

[0138] The skin image completion model generation module 305 is used to build a neural network model. It is trained using training set image data and validated using validation set image data. The trained and validated neural network model is then set as the skin image completion model.

[0139] The skin image completion module 306 is used to input the skin image to be completed into the skin image completion model when a skin image to be completed is received.

[0140] In one embodiment, the sample image denoising module 302 is specifically used to calculate the frequency intensity of the lesion sample image of each lesion region at a preset frequency, and to filter n frequency-filtered images of each lesion region that meet the preset conditions according to the frequency intensity; to perform filtering processing on the n frequency-filtered images of each lesion region and generate n filtered low-noise images.

[0141] In one embodiment, the sample image denoising module 302 is specifically used to perform grayscale processing on m lesion sample images of each lesion region and generate m lesion grayscale images; convert the m lesion grayscale images of each lesion region from the spatial domain to the frequency domain representation; calculate the frequency intensity of the m lesion grayscale images of each lesion region at a preset frequency and generate a frequency intensity set; filter the image with the highest frequency intensity value in the frequency intensity set; set the image with the highest frequency intensity value and several images before and after the image with the highest frequency intensity value as frequency filtering images, wherein the total number of frequency filtering images for each lesion region is n, where n is less than m.

[0142] In one embodiment, the sample image denoising module 302 is specifically used to create a neighborhood window for each pixel in the n frequency screening images of each lesion region; calculate the weight of each pixel in the neighborhood window; generate a Gaussian kernel based on the weight of each pixel in the neighborhood window; update the RGB value of each pixel in the n frequency screening images of each lesion region using the Gaussian kernel; and set the updated image as a filtered low-noise image, with each lesion region containing n filtered low-noise images.

[0143] In one embodiment, the low-noise image fusion module 303 is specifically configured to: divide n filtered low-noise images according to a preset first image size to generate several first low-noise sub-image blocks; fuse the several first low-noise sub-image blocks according to a preset first image fusion method to generate a first fused image; divide n filtered low-noise images according to a preset second image size to generate several second low-noise sub-image blocks; fuse the several second low-noise sub-image blocks according to a preset first image fusion method to generate a second fused image; divide n filtered low-noise images according to a preset third image size to generate several third low-noise sub-image blocks; fuse the several third low-noise sub-image blocks according to a preset first image fusion method to generate a third fused image; fuse the first fused image, the second fused image, and the third fused image according to a preset weight to generate a final clear and complete image; and perform the same image fusion processing on n filtered low-noise images for each lesion region to finally generate c clear and complete images.

[0144] In one embodiment, the low-noise image fusion module 303 is specifically used to calculate the frequency intensity value of each low-noise sub-image block in each filtered low-noise image after dividing n filtered low-noise images into several low-noise sub-image blocks according to a preset image size; and to fuse the n low-noise sub-image blocks at the same position after segmentation based on the RGB values ​​of the pixels of the low-noise sub-image blocks and the corresponding frequency intensity values ​​to generate a fused sub-image block. The calculation method of the fused sub-image block includes:

[0145]

[0146] Among them, F (i) This represents the frequency intensity value of the i-th low-noise sub-image patch at the same location after segmentation, img (i) (x, y) represents the RGB value of the pixel coordinates of the i-th low-noise sub-image block at the same position after segmentation, and n is the number of low-noise sub-image blocks at the same position after segmentation; all fused sub-image blocks are stitched together to generate a fused image.

[0147] In one embodiment, the model generation module 305 is specifically used to construct the neural network model structure and loss function. The neural network model structure includes convolutional layers, multi-head attention mechanism layers, pooling layers, upsampling layers, and fully connected layers; the loss function includes:

[0148]

[0149] Where p represents the total number of image data in the validation set, Z 生i Z represents the i-th image generated by the neural network model. 原i This represents the sharp, filled image corresponding to the image generated by the neural network model, where δ represents a constant.

[0150] The skin image completion device provided in this application embodiment can be applied to the skin image completion method provided in the above embodiment. For relevant details, please refer to the above method embodiment. The implementation principle and technical effect are similar, and will not be repeated here.

[0151] It should be noted that the skin image completion device provided in this embodiment is only illustrated by the above-described division of functional modules / units when performing skin image completion. In practical applications, the above functions can be assigned to different functional modules / units as needed, that is, the internal structure of the skin image completion device can be divided into different functional modules / units to complete all or part of the functions described above. Furthermore, the implementation method of the skin image completion method provided in the above-described method embodiment and the implementation method of the skin image completion device provided in this embodiment belong to the same concept. The specific implementation process of the skin image completion device provided in this embodiment is detailed in the above-described method embodiment and will not be repeated here.

[0152] This application also discloses a computer device.

[0153] Specifically, such as Figure 4As shown, the computer device can be a desktop computer, laptop computer, handheld computer, or cloud server, etc. The computer device may include, but is not limited to, a processor and memory. The processor and memory can be connected via a bus or other means. The processor can be a Central Processing Unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, graphics processing units (GPUs), embedded neural network processing units (NPUs) or other dedicated deep learning coprocessors, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0154] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the above embodiments of this application. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include a program storage area and a data storage area, wherein the program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0155] This application also discloses a computer-readable storage medium.

[0156] Specifically, the computer-readable storage medium is used to store a computer program, which, when executed by a processor, implements the methods described in the above-described method embodiments. Those skilled in the art will understand that implementing all or part of the processes in the methods described in the above-described embodiments of this application can be accomplished by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.

[0157] This specific embodiment is merely an explanation of the present invention and is not intended to limit the invention. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they are within the scope of the claims of the present invention.

Claims

1. A method for skin image completion, characterized in that: The method includes: Acquire lesion sample images of c lesion regions, where each lesion region includes m lesion sample images; For each lesion region, m sample images of the lesion are denoised to generate n filtered low-noise images, where n is less than m; The n filtered low-noise images of each lesion region are fused to generate a clear and complete image; The filtered low-noise images of c lesion regions and their corresponding clear and complete images are divided into training set image data and validation set image data according to a preset ratio; A neural network model is constructed, trained using the image data in the training set, validated using the image data in the validation set, and the trained and validated neural network model is set as a skin image completion model. Upon receiving a skin image to be filled in, the skin image to be filled in is input into the skin image filling model to fill in the skin image; The step of denoising m lesion sample images for each lesion region and generating n filtered low-noise images includes: Calculate the frequency intensity of the lesion sample image of each lesion region at a preset frequency, and filter n frequency-filtered images of each lesion region that meet the preset conditions based on the frequency intensity; The n frequency-selected images for each lesion region are filtered to generate n filtered low-noise images; The step of calculating the frequency intensity of the lesion sample image of each lesion region at a preset frequency, and filtering n frequency-selected images of each lesion region that meet the preset conditions based on the frequency intensity, includes: Each of the m lesion sample images in each lesion region is converted to grayscale, and m grayscale images of the lesion are generated. Convert the m grayscale images of each lesion region from the spatial domain to the frequency domain representation; Calculate the frequency intensity of m grayscale images of each lesion region at a preset frequency, and generate a set of frequency intensities; Filter the image with the highest frequency intensity value from the set of frequency intensity values; The image with the highest frequency intensity value, along with several images before and after the image with the highest frequency intensity value, are set as the frequency screening images. The total number of frequency screening images for each lesion region is n, where n is less than m.

2. The method according to claim 1, characterized in that: The step of filtering the n frequency-selected images for each lesion region and generating n filtered low-noise images includes: For each pixel in the n frequency-filtered images of each lesion region, a neighborhood window is created; Calculate the weight of each pixel in the neighborhood window, and generate a Gaussian kernel based on the weight of each pixel in the neighborhood window; The RGB values ​​of each pixel in the n frequency-filtered images for each lesion region are updated using the Gaussian kernel; The updated image is set as a filtered low-noise image, and each lesion region contains n filtered low-noise images.

3. The method according to claim 1, characterized in that: The step of fusing the n filtered low-noise images of each lesion region to generate a clear and complete image includes: The n filtered low-noise images are divided into several first low-noise sub-image blocks according to a preset first image size, and the several first low-noise sub-image blocks are fused according to a preset first image fusion method to generate a first fused image. The n filtered low-noise images are divided into several second low-noise sub-image blocks according to a preset second image size, and the several second low-noise sub-image blocks are fused according to a preset first image fusion method to generate a second fused image; The n filtered low-noise images are divided into several third low-noise sub-image blocks according to a preset third image size, and the several third low-noise sub-image blocks are fused according to a preset first image fusion method to generate a third fused image. The first fused image, the second fused image, and the third fused image are fused according to preset weights to generate the final clear and complete image; The same image fusion process is performed on n filtered low-noise images of each lesion region to finally generate c clear and complete images.

4. The method according to claim 3, characterized in that: The preset first image fusion method includes: After dividing the n filtered low-noise images into several low-noise sub-image blocks according to a preset image size, the frequency intensity value of each low-noise sub-image block in each filtered low-noise image is calculated. The n low-noise sub-image blocks at the same location after segmentation are fused according to the RGB values ​​of the pixels of the low-noise sub-image blocks and the corresponding frequency intensity values ​​to generate a fused sub-image block. The calculation method of the fused sub-image block includes: ， Among them, F (i) This represents the frequency intensity value of the i-th low-noise sub-image patch at the same location after segmentation, img (i) (x, y) represents the RGB value of the pixel coordinates of the i-th low-noise sub-image block at the same position after segmentation, and n is the number of low-noise sub-image blocks at the same position after segmentation. All the fused sub-image blocks are stitched together to generate a fused image.

5. The method according to claim 1, characterized in that: The construction of the neural network model includes: Construct a neural network model structure and loss function. The structure of the neural network model includes convolutional layers, multi-head attention mechanism layers, pooling layers, upsampling layers, and fully connected layers. The loss function includes: ， Where p represents the total number of image data in the verification set, Z 生i Z represents the i-th image generated by the neural network model. 原i This represents the sharp, filled-in image corresponding to the image generated by the neural network model. Represents a constant.

6. A skin image completion device, characterized in that: The device includes: The sample image acquisition module (301) is used to acquire lesion sample images of c lesion regions, each lesion region including m lesion sample images; The sample image denoising module (302) is used to denoise m lesion sample images for each lesion region and generate n filtered low-noise images, where n is less than m; The low-noise image fusion module (303) is used to fuse the n filtered low-noise images of each lesion region to generate a clear and complete image. The image data partitioning module (304) is used to divide the filtered low-noise images and the corresponding clear and complete images of the c lesion regions into training set image data and validation set image data according to a preset ratio. The skin image completion model generation module (305) is used to construct a neural network model, which is trained using the training set image data, verified using the verification set image data, and the trained and verified neural network model is set as a skin image completion model. The skin image completion module (306) is used to input the skin image to be completed into the skin image completion model to complete the skin image when a skin image to be completed is received. The sample image denoising module (302) is used to calculate the frequency intensity of the lesion sample image of each lesion region at a preset frequency, and to filter n frequency-filtered images of each lesion region that meet the preset conditions according to the frequency intensity; to filter the n frequency-filtered images of each lesion region and generate n filtered low-noise images; The sample image denoising module (302) is used to perform grayscale processing on m lesion sample images of each lesion region and generate m lesion grayscale images; convert the m lesion grayscale images of each lesion region from the spatial domain to the frequency domain representation; calculate the frequency intensity of the m lesion grayscale images of each lesion region at a preset frequency and generate a frequency intensity set; filter the image with the highest frequency intensity value in the frequency intensity set; set the image with the highest frequency intensity value and several images before and after the image with the highest frequency intensity value as frequency filtering images, wherein the total number of frequency filtering images for each lesion region is n, and n is less than m.

7. A computer device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer program is stored that can be loaded by a processor and executed according to any one of claims 1 to 5.

Images