An image defogging method, device, electronic equipment and computer storage medium

Abstract

The application discloses an image defogging method and device, electronic equipment and computer storage medium, and is used for improving the image processing quality of an embedded device. The method comprises the following steps: acquiring a target meibomian gland image, wherein the target meibomian gland image is a near-infrared light image; inputting the target meibomian gland image into a pre-trained defogging model to obtain a defogging meibomian gland image output by the defogging model, wherein the defogging model is a lightweight neural network model adopting an encoder-decoder structure, and the defogging model fuses feature maps of an up-sampling layer and a down-sampling layer through an integrated spatial feature transformation module.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image defogging method and device, an electronic device, and a computer storage medium. BACKGROUND

[0002] In modern medical image analysis, the diagnosis of meibomian gland dysfunction is of great significance for the early detection of eye diseases such as dry eye. The existing technology usually uses near-infrared light imaging to obtain meibomian gland images in order to observe the structure and distribution of the glands. However, due to the influence of light scattering, fogging and acquisition conditions, the obtained meibomian gland images often have low contrast, unclear details, and blurred edges. The insufficient quality of such images directly affects the accuracy of subsequent automatic analysis algorithms or manual diagnosis, especially in embedded devices, due to limited computing resources and storage capacity, it is more difficult to perform high-quality image enhancement and restoration through complex algorithms.

[0003] In the prior art, although some methods attempt to improve image quality through image defogging or enhancement algorithms, most of the schemes have defects such as high model complexity, insufficient feature information extraction, or limited recovery effect on local edges and details, which cannot be efficiently run in an embedded device environment. This results in that the image clarity and structural fidelity are still not ideal when the embedded device processes near-infrared images of meibomian glands, which is difficult to meet the needs of clinical diagnosis or automatic analysis. SUMMARY

[0004] The present application provides an image defogging method, device, electronic device and computer storage medium, which can improve the image processing quality of embedded devices.

[0005] The first aspect of the present application provides an image defogging method, comprising:

[0006] obtaining a target meibomian gland image, the target meibomian gland image being a near-infrared light image;

[0007] inputting the target meibomian gland image into a pre-trained defogging model to obtain a defogging meibomian gland image output by the defogging model, the defogging model being a lightweight neural network model using an encoder-decoder structure, and the defogging model fusing feature maps of up-sampling layers and down-sampling layers through an integrated spatial feature transformation module.

[0008] Optionally, before inputting the target meibomian gland image into the pre-trained defogging model, the method further comprises:

[0009] extracting a sample pair from a data set, the sample pair including a foggy sample image and a corresponding clear sample image;

[0010] inputting the foggy sample image into an initial model;

[0011] perform feature extraction on the foggy sample image through multiple down-sampling layers of the initial model to obtain multiple layers of down-sampled feature maps;

[0012] perform feature fusion on the multiple layers of down-sampled feature maps through a bottleneck layer and multiple up-sampling layers of the initial model to obtain a final feature map;

[0013] generate a de-fogged sample image according to the final feature map;

[0014] calculate a total loss value according to the de-fogged sample image and the clear sample image;

[0015] determine whether the initial model converges according to the total loss value, and if the initial model converges, determine the initial model as a de-fogging model; if the initial model does not converge, update the initial model reversely and perform next round of training.

[0016] Optionally, the calculating of the total loss value according to the de-fogged sample image and the clear sample image comprises:

[0017] perform image enhancement on the de-fogged sample image to obtain an enhanced image comprising high-frequency edge information;

[0018] calculate a first loss value according to the de-fogged sample image and the clear sample image;

[0019] calculate a second loss value according to the de-fogged sample image and the enhanced image;

[0020] calculate the total loss value according to the first loss value and the second loss value.

[0021] Optionally, the calculating of the first loss value according to the de-fogged sample image and the clear sample image comprises:

[0022] obtain hidden features obtained from a pre-trained model;

[0023] calculate the first loss value according to the de-fogged sample image, the clear sample image and the hidden features.

[0024] Optionally, the performing of the image enhancement on the de-fogged sample image to obtain the enhanced image comprising high-frequency edge information comprises:

[0025] perform histogram equalization processing on the de-fogged sample image to obtain an initial enhanced image;

[0026] perform processing on the initial enhanced image through a Prewitt operator to obtain the enhanced image comprising high-frequency edge information.

[0027] Optionally, the feature fusion on the multi-layer down-sampling feature maps through the bottleneck layer and the plurality of up-sampling layers of the initial model is configured to obtain a final feature map, and the feature fusion comprises:

[0028] The last layer of down-sampling feature maps is dynamically enhanced through the bottleneck layer of the initial model to obtain an enhanced feature map.

[0029] The enhanced feature map is extracted through the plurality of up-sampling layers of the initial model, and the multi-layer down-sampling feature maps are fused by using the spatial feature transformation module to obtain a final feature map.

[0030] Optionally, the number of down-sampling layers of the initial model is 2, and the number of up-sampling layers is 2; the bottleneck layer comprises 6 layers of feature attention modules and 2 layers of deformation convolution.

[0031] The second aspect of the present application provides an image defogging device, comprising:

[0032] An acquisition unit is configured to acquire a target meibomian gland image, wherein the target meibomian gland image is a near-infrared image.

[0033] An input unit is configured to input the target meibomian gland image into a pre-trained defogging model to obtain a defogging meibomian gland image output by the defogging model, wherein the defogging model is a lightweight neural network model adopting an encoder-decoder structure, and the defogging model fuses feature maps of up-sampling layers and down-sampling layers by integrating a spatial feature transformation module.

[0034] Optionally, the device further comprises:

[0035] An extraction unit is configured to extract a sample pair from a data set, wherein the sample pair comprises a hazy sample image and a corresponding clear sample image.

[0036] A sample input unit is configured to input the hazy sample image into an initial model.

[0037] An extraction unit is configured to extract features of the hazy sample image through a plurality of down-sampling layers of the initial model to obtain a plurality of down-sampling feature maps.

[0038] A fusion unit is configured to fuse features of the plurality of down-sampling feature maps through a bottleneck layer and a plurality of up-sampling layers of the initial model to obtain a final feature map.

[0039] A generation unit is configured to generate a defogging sample image according to the final feature map.

[0040] A calculation unit is configured to calculate a total loss value according to the defogging sample image and the clear sample image.

[0041] A judging unit is configured to judge whether the initial model converges according to the total loss value, and if yes, determine the initial model as a defogging model; if not, update the initial model reversely and perform next round of training.

[0042] Optionally, the computing unit comprises:

[0043] An enhancement module is configured to perform image enhancement on the defogging sample image to obtain an enhanced image comprising high-frequency edge information.

[0044] A first computing module is configured to calculate a first loss value according to the defogging sample image and the clear sample image.

[0045] A second computing module is configured to calculate a second loss value according to the defogging sample image and the enhanced image.

[0046] A third computing module is configured to calculate a total loss value according to the first loss value and the second loss value.

[0047] Optionally, the first computing module is specifically configured to:

[0048] obtain hidden features from a pre-trained model;

[0049] calculate a first loss value according to the defogging sample image, the clear sample image and the hidden features.

[0050] Optionally, the enhancement module is specifically configured to:

[0051] perform histogram equalization processing on the defogging sample image to obtain an initial enhanced image;

[0052] perform processing on the initial enhanced image by a Prewitt operator to obtain an enhanced image comprising high-frequency edge information.

[0053] Optionally, the fusion unit is specifically configured to:

[0054] perform dynamic feature enhancement on a last layer down-sampling feature map by a bottleneck layer of the initial model to obtain an enhanced feature map;

[0055] perform feature extraction on the enhanced feature map by a plurality of up-sampling layers of the initial model, and fuse the plurality of down-sampling feature maps by using the spatial feature transformation module to obtain a final feature map.

[0056] Optionally, the number of down-sampling layers of the initial model is 2, and the number of up-sampling layers is 2; the bottleneck layer comprises 6 layers of feature attention modules and 2 layers of deformation convolution.

[0057] The third aspect of the present application provides an electronic device, comprising:

[0058] a processor, a memory, an input / output unit and a bus;

[0059] the processor is connected with the memory, the input / output unit and the bus;

[0060] the memory stores a program, and the processor invokes the program to execute the method in the first aspect and any possible implementation manner of the first aspect.

[0061] The fourth aspect of the present application provides a computer readable storage medium, which stores a program, and the program causes a computer to execute the method in the first aspect and any possible implementation manner of the first aspect when the computer executes the program.

[0062] From the above technical solutions, the present application has the following advantages:

[0063] In the method provided by the present application, the near-infrared light image is first acquired and processed by using a lightweight defogging model with an encoder-decoder structure and a spatial feature transformation fusion module, which can realize higher defogging quality and detail preservation under the limited computing resources and storage conditions of embedded devices, reduce artifacts and noise residues, improve the structure recovery capability, and thus improve the image processing quality of the embedded device, facilitating subsequent automatic analysis or manual diagnosis. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 The flowchart of an embodiment of the image defogging method in the present application is shown;

[0065] Figure 2 The flowchart of an embodiment of training the initial model in the present application is shown;

[0066] Figure 3 The flowchart of an embodiment of calculating the total loss value in the present application is shown;

[0067] Figure 4 The flowchart of an embodiment of obtaining the final feature Figure 1 in the present application is shown;

[0068] Figure 5 The structural diagram of an embodiment of the image defogging device in the present application is shown;

[0069] Figure 6 The structural diagram of an embodiment of the electronic device in the present application is shown. DETAILED DESCRIPTION

[0070] The embodiments of the present application provide an image defogging method, device, electronic device and computer storage medium, which are used to improve the image processing quality of the embedded device.

[0071] The method of the present application can be applied to a server, a terminal or other device with logical processing capability, and the present application is not limited thereto. For convenience of description, the following will be described by taking the terminal as an example.

[0072] The embodiments of the present application will be described below with reference to the accompanying drawings.

[0073] Please refer to Figure 1 , Figure 1 An embodiment of the image defogging method provided by the present application includes the following steps:

[0074] 101. Obtain a target meibomian gland image, which is a near-infrared light image;

[0075] The terminal obtains the image of the user's meibomian gland area through the near-infrared image acquisition device (such as a CMOS camera) connected thereto. The image is acquired under the near-infrared light waveband, and due to the scattering characteristics of biological tissues and device noise, the target meibomian gland image usually presents a low-contrast, hazy and foggy visual effect.

[0076] 102. Input the target meibomian gland image into a pre-trained defogging model to obtain a defogging meibomian gland image output by the defogging model, the defogging model being a lightweight neural network model adopting an encoder-decoder structure, and the defogging model fusing feature maps of up-sampling layers and down-sampling layers through an integrated spatial feature transformation module.

[0077] The terminal calls a pre-trained defogging model stored internally or downloaded from a server. The defogging model receives the target meibomian gland image as input and performs forward propagation calculation through its internal network structure. The core of the model lies in that it adopts an encoder-decoder structure to realize end-to-end image-to-image conversion, and through an integrated spatial feature transformation module, it adaptively fuses the multi-layer down-sampling feature maps extracted by the down-sampling layers in the encoder path and the feature maps generated by the up-sampling layers in the decoder path. Finally, the defogging model outputs a defogging meibomian gland image, which has a significantly removed fog effect compared to the input image, and the contrast and clarity of the gland structure are greatly improved. It should be noted that the number of up-sampling layers is consistent with the number of down-sampling layers, such as 3, 4 or other numbers, and the same layers are corresponding.

[0078] In this embodiment, the terminal first acquires a near-infrared light image and processes it using a lightweight defogging model with an encoder-decoder structure and a spatial feature transformation fusion module, which can achieve higher defogging quality and detail preservation under the limited computing resources and storage conditions of embedded devices, reduce artifacts and noise residues, and improve the structure recovery capability, thereby improving the image processing quality of embedded devices and facilitating subsequent automatic analysis or manual diagnosis.

[0079] Referring to Figure 2 In some embodiments of the present application, before step 102 in the above embodiments inputs the target meibomian gland image into the pre-trained dehazing model, the method can further include the following steps:

[0080] 201、Extract a sample pair from a data set, the sample pair including a hazy sample image and a corresponding clear sample image;

[0081] The terminal extracts a sample pair from a pre-constructed data set, and each sample pair is composed of a hazy sample image and a corresponding clear sample image. The hazy sample image is a blurred image caused by interference, light scattering or insufficient imaging conditions during the collection process, which is used to simulate the low-quality situation that may occur during actual collection; the clear sample image is a high-quality image collected under ideal conditions, which is used as a reference to provide a supervision signal to gradually optimize the model output during the training process.

[0082] 202、Input the hazy sample image into an initial model;

[0083] The terminal inputs the extracted hazy sample image into an initial model, and the initial model adopts a lightweight encoder-decoder structure. The encoder part is used to compress the spatial size of the input image and extract multi-level semantic features, and the decoder part is used to gradually restore the image resolution and try to reconstruct the clear image in terms of structure and details. Through this input process, the terminal can activate the feature extraction and restoration capability of the model.

[0084] 203、Extract features from the hazy sample image through multiple down-sampling layers of the initial model to obtain multiple layers of down-sampled feature maps;

[0085] The terminal uses multiple down-sampling layers in the encoder part of the initial model to extract features from the hazy sample image one by one. Each layer of down-sampling reduces the spatial resolution of the image while retaining and extracting more abstract feature information. For example, the first layer of down-sampled feature map can highlight edges and local details, and the second layer and deeper layers of down-sampled feature map can represent higher-level structural patterns. Through layer-by-layer extraction, the terminal obtains multiple layers of down-sampled feature maps containing different scale features, which lays a foundation for subsequent feature fusion.

[0086] 204、Fuse features of the multiple layers of down-sampled feature maps through a bottleneck layer and multiple up-sampling layers of the initial model to obtain a final feature map;

[0087] The terminal inputs the obtained multi-layer down-sampled feature maps to a bottleneck layer, which is responsible for integrating and filtering multi-scale features in a compressed semantic space, and retaining key structural information. Subsequently, the terminal gradually recovers the spatial resolution through multiple up-sampling layers of the decoder, and each up-sampling layer combines a spatial feature transformation module to fuse the corresponding feature maps from the down-sampling layer with the up-sampling result, so as to simultaneously take into account the low-level texture details and high-level structural features. After this fusion and recovery process, the terminal finally obtains the final feature map containing rich details and clear structures.

[0088] 205. generating a defogging sample image according to the final feature map;

[0089] The terminal generates a defogging sample image according to the final feature map, which should remove the fogging interference in the input image as much as possible while maintaining the structural integrity and edge clarity of the meibomian gland tissue. In the generation process, the terminal maps the high-resolution feature map recovered by the decoder to the output image, so that the defogging sample image is close to the clear sample image in terms of overall brightness, contrast and detail level.

[0090] 206. calculating a total loss value according to the defogging sample image and the clear sample image;

[0091] The terminal compares the generated defogging sample image with the corresponding clear sample image pixel by pixel, and calculates the total loss value using a plurality of preset loss functions. The loss function not only includes the mean square error at the pixel level, which is used to measure the numerical difference between the two images, but also includes the perception loss, which is used to evaluate the consistency of high-level semantic features, and the structural similarity loss, which is used to compare the similarity of images in terms of texture and edge. Through multi-dimensional comprehensive loss calculation, the terminal can comprehensively measure the gap between the defogging sample image and the clear sample image.

[0092] 207. determining whether the initial model converges according to the total loss value, and if the initial model converges, determining the initial model as a defogging model; if the initial model does not converge, updating the initial model in reverse and performing the next round of training.

[0093] The terminal determines whether the initial model converges according to the total loss value. If the total loss value remains stable and low in multiple training rounds, it means that the initial model has good defogging ability, and the terminal determines this model as the final defogging model. If the model has not converged, the terminal updates the parameters of the model according to the back propagation algorithm, adjusts the convolution kernel weight, bias value and feature fusion weight, so as to reduce the total loss value. After updating the parameters, the terminal inputs new samples again for the next round of training (i.e. executes steps 201-207 again), until the model reaches a high similarity between the generated defogging sample image and the clear sample image, ensuring that the defogging effect is stable and reliable.

[0094] In this embodiment, the terminal first constructs sample pairs of foggy and clear images from the data set, then inputs the foggy sample image into the initial model of the encoder-decoder structure, and sequentially passes through multi-layer down-sampling feature extraction, bottleneck layer integration, and up-sampling and spatial feature transformation fusion, to generate a de-fogging sample image, and calculate the total loss value by comparing with the clear image, and iteratively optimize the model parameters combined with the back propagation mechanism, and finally obtain the converged de-fogging model. This training method can ensure the model to be lightweight while having strong detail recovery and structure restoration ability under the condition of limited resources of embedded devices, effectively improving the image processing quality of embedded devices when processing meibomian gland near-infrared images.

[0095] Please refer to Figure 3 In some embodiments of the present application, step 206 in the above embodiment can include the following steps:

[0096] 301, performing image enhancement on the de-fogging sample image to obtain an enhanced image including high-frequency edge information;

[0097] The terminal performs image enhancement processing on the de-fogging sample image, and the enhancement method can include filtering, gradient operation or edge detection method, etc., for highlighting the texture details and edge structure information in the image. After enhancement processing, the terminal obtains an enhanced image containing high-frequency edge information, which can more obviously reflect the edge and detail features of the meibomian gland tissue, providing more targeted reference data for subsequent loss value calculation.

[0098] 302, calculating a first loss value according to the de-fogging sample image and the clear sample image;

[0099] The terminal compares the de-fogging sample image with the corresponding clear sample image, calculates the difference between them through a preset loss function, and obtains a first loss value. The first loss value is mainly reflected in the overall brightness, pixel distribution and global structure level, and measures the closeness of the de-fogging sample image to the clear sample image in overall image quality.

[0100] 303, calculating a second loss value according to the de-fogging sample image and the enhanced image;

[0101] The terminal compares the de-fogging sample image with the enhanced image, calculates the difference between them by using a loss function for high-frequency components, and obtains a second loss value. The second loss value mainly measures the performance of the de-fogging sample image in edge sharpness, detail fidelity and local texture recovery, to ensure that the de-fogging model can remove fogging interference while preserving detail information.

[0102] 304, calculating a total loss value according to the first loss value and the second loss value.

[0103] The terminal performs weighted fusion or comprehensive calculation according to the first loss value and the second loss value to obtain a final total loss value. The total loss value not only reflects the recovery of the defogging sample image in the overall quality, but also takes into account the preservation effect of the edge and the detail, thereby providing a more comprehensive basis for the terminal to subsequently judge whether the model converges.

[0104] In this embodiment, when calculating the total loss value, the terminal first performs image enhancement on the defogging sample image to obtain an enhanced image containing high-frequency edge information, and then combines the comparison results of the defogging sample image and the clear sample image and the comparison results of the defogging sample image and the enhanced image to respectively obtain the first loss value related to the overall quality and the second loss value related to the detail. Finally, the two types of losses are fused into the total loss value. This design can enable the defogging model to simultaneously focus on the global image quality and the local detail fidelity during the training process, thereby improving the overall image processing quality of the embedded device in the meibomian gland image processing.

[0105] In step 302, the first loss value is calculated according to the defogging sample image and the clear sample image, which can specifically include:

[0106] 3021, obtaining hidden features extracted from a pre-trained model (i.e. other pre-trained neural networks, generally using open source models or authorized closed source models);

[0107] The terminal obtains hidden features extracted from a pre-trained model (i.e. other pre-trained neural networks, generally using open source models or authorized closed source models). The hidden features usually come from the output of deep convolutional layers and can represent high-level information of images in the semantic level, such as texture distribution, structure pattern and edge relationship. These hidden features can better reflect the differences of images in the high-level semantic space than the original pixel values, providing more representative feature references for subsequent loss calculation.

[0108] 3022, calculating the first loss value according to the defogging sample image, the clear sample image and the hidden features.

[0109] The terminal calculates the difference between the two images in the high-dimensional semantic feature space by using the defogging sample image, the clear sample image and the extracted hidden features, thereby obtaining the first loss value. This calculation not only considers the pixel-level difference, but also integrates the semantic consistency information contained in the hidden features, so that the first loss value can more accurately measure the closeness of the defogging sample image and the clear sample image in the overall structure and semantic expression.

[0110] When calculating the first loss value, the terminal introduces the hidden features of the pre-trained model to extend the loss calculation from a single pixel difference to a semantic feature space, which can more comprehensively constrain the recovery effect of the defogging model on the structure and the detail during the training process, thereby improving the overall image processing quality of the embedded device in the meibomian gland image processing.

[0111] In the step 301, the fog sample image is subjected to image enhancement to obtain an enhanced image including high-frequency edge information, which can specifically include:

[0112] 3011. Perform histogram equalization processing on the fog sample image to obtain an initial enhanced image.

[0113] The terminal performs histogram equalization processing on the fog sample image. Histogram equalization re-distributes the gray scale distribution of the image to make the overall brightness level more balanced and enhance the contrast of the image. After this processing, the terminal obtains an initial enhanced image, and the brightness distribution of this image is more reasonable, and the visibility of the detail area is improved, creating conditions for subsequent edge information extraction.

[0114] 3012. Process the initial enhanced image by using a Prewitt operator to obtain an enhanced image including high-frequency edge information.

[0115] The terminal processes the initial enhanced image by using a Prewitt operator. The Prewitt operator highlights the gradient changes of the image through convolution operations in the horizontal and vertical directions, thereby effectively extracting the edge information in the image. After Prewitt operator processing, the terminal obtains an enhanced image containing high-frequency edge information, which can clearly present the outline and detailed structure of the meibomian gland, providing data support for measuring detailed features in subsequent loss value calculation.

[0116] In this embodiment, the terminal first performs histogram equalization processing on the fog sample image to enhance the overall contrast, and then uses the Prewitt operator to extract high-frequency edge information in the image, finally obtaining an enhanced image that can highlight detailed and structural features. This design can better constrain the recovery effect of the edge and details of the dehazing model in the loss value calculation process, thereby improving the image processing quality of the embedded device in meibomian gland image processing.

[0117] In one possible embodiment, the terminal can calculate the total loss value by using the following formula.

[0118]

[0119]

[0120]

[0121] wherein, is the total loss value, is the first loss value, is the second loss value.

[0122] The first loss value includes two parts, the first part represents using a data fidelity term of the paradigm, used to represent the image reconstruction loss, for calculating the clear image corresponding to the clear sample image) and the predicted image corresponding to the dehazed sample image), wherein is the foggy image (corresponding to the foggy sample image), represents the parameters of the model; the second part is used to represent the contrast regular term of the recovered image, as an adjustment parameter, used to control the weight ratio of the data fidelity term and the contrast regular term. In the formula represents the features of the first layer obtained from the pre-trained model, is the and norm distance between is the corresponding weight.

[0123] The second loss value is calculated by the loss function , is the point-by-point multiplication of the difference between and , and then normalized, because the generated low-quality picture (i.e. ) and the ideal picture (i.e. ) have a certain gap, so is used to constrain the generation of the dehazed image, and such constraint has a particularly obvious effect on foggy medical images.

[0124] Please refer to Figure 4 In some embodiments of the present application, the step 204 in the above embodiment can include the following steps:

[0125] 401. Perform dynamic feature enhancement on the last layer of down-sampled feature map through the bottleneck layer of the initial model to obtain an enhanced feature map;

[0126] The terminal performs dynamic feature enhancement on the last layer of down-sampled feature map through the bottleneck layer of the initial model, and the bottleneck layer performs re-weighting distribution and enhancement on deep features through adaptive convolution and nonlinear activation function, strengthens the key information in the image such as the edge of the meibomian gland and the texture structure, and suppresses noise and redundant features at the same time. After this processing, the terminal obtains an enhanced feature map, so that the deep features have stronger expression ability while maintaining the integrity of the semantics, providing high-quality input for upsampling and feature fusion.

[0127] ​402、The enhanced feature map is feature extracted through multiple up-sampling layers of the initial model, and a spatial feature transformation module is used to fuse the multi-layer down-sampled feature maps to obtain a final feature map.

[0128] The terminal performs layer-by-layer feature extraction on the enhanced feature map through multiple up-sampling layers of the initial model, and restores the image spatial resolution. In the up-sampling process, the terminal uses a spatial feature transformation module to fuse the down-sampled feature maps of different levels with the up-sampling results, realizes the integration of multi-scale features, and makes the final feature map not only retain low-level detail information, but also have high-level semantic expression ability. The final feature map provides complete and clear structure and detail information for generating a defogging sample image.

[0129] In this embodiment, the terminal dynamically enhances deep features through the bottleneck layer, and then combines the up-sampling layer and the spatial feature transformation module to perform multi-scale feature fusion, which can effectively retain the detail information and structural features of the image, suppress noise, improve the expression ability of deep features, and thus improve the image quality and detail recovery ability of the embedded device in meibomian gland image processing.

[0130] In an embodiment of the present application, the initial model can be set as follows: the number of down-sampling layers of the initial model is 2, and the number of up-sampling layers is 2; the bottleneck layer includes 6 feature attention modules and 2 deformation convolutions. At this time, the final feature map can be obtained by the following formula.

[0131] Formula 4

[0132] Wherein, in the formula and are the feature maps from the first i down-sampling layer and the up-sampling layer, respectively. is the final output, i.e., the final feature map. , i = 1,2 represent the feature fusion of the first i layer, which combines the inputs from the first i down-sampling layer and the first i up-sampling layer. These values are determined by the parameter scaling coefficients and the bias coefficients . In the training process, these parameters are trained in reverse to restore the contrast enhancement and image superposition operations.

[0133] In this embodiment, the initial model sets the up-sampling layer and the down-sampling layer to be 2 layers, so that the network maintains the necessary multi-scale representation capability while significantly reducing the calculation amount and the parameter size; the bottleneck layer introduces 6 feature attention modules to enhance the response to key texture and structure features in the semantic compression space and improve the selective retention of the meibomian gland edge and details; the 2 deformation convolutions configured in the bottleneck enhance the adaptability to local geometric deformation and slight position offset, which is beneficial to more accurate alignment and recovery of fine structures. Thus, the beneficial effects can be deduced: the model significantly reduces the model complexity and the calculation overhead while maintaining or enhancing the detail recovery and edge restoration capability, which is suitable for real-time deployment on resource-limited embedded terminals, thereby improving the meibomian gland near-infrared image processing quality of the embedded device and improving the processing efficiency.

[0134] Referring to Figure 5 , Figure 5 An embodiment of the image defogging device provided in the present application includes:

[0135] The acquisition unit 501 is configured to acquire a target meibomian gland image, the target meibomian gland image being a near-infrared image;

[0136] The input unit 502 is configured to input the target meibomian gland image into a pre-trained defogging model to obtain a defogging meibomian gland image output by the defogging model, the defogging model being a lightweight neural network model adopting an encoder-decoder structure, and the defogging model fusing feature maps of the up-sampling layer and the down-sampling layer through an integrated spatial feature transformation module.

[0137] Optionally, the device further includes:

[0138] The extraction unit is configured to extract a sample pair from the data set, the sample pair including a hazy sample image and a corresponding clear sample image;

[0139] The sample input unit is configured to input the hazy sample image into the initial model;

[0140] The extraction unit is configured to extract features of the hazy sample image through the multiple down-sampling layers of the initial model to obtain multiple layers of down-sampled feature maps;

[0141] The fusion unit is configured to fuse the multiple layers of down-sampled feature maps through the bottleneck layer and the multiple up-sampling layers of the initial model to obtain a final feature map;

[0142] The generation unit is configured to generate a defogging sample image according to the final feature map;

[0143] The calculation unit is configured to calculate a total loss value according to the defogging sample image and the clear sample image;

[0144] The judging unit is configured to judge whether the initial model converges according to the total loss value, and if yes, determine the initial model as the defogging model; if not, update the initial model reversely and perform the next round of training.

[0145] Optionally, the calculating unit comprises:

[0146] The enhancement module is configured to perform image enhancement on the defogging sample image to obtain an enhanced image comprising high-frequency edge information.

[0147] The first calculating module is configured to calculate a first loss value according to the defogging sample image and the clear sample image.

[0148] The second calculating module is configured to calculate a second loss value according to the defogging sample image and the enhanced image.

[0149] The third calculating module is configured to calculate a total loss value according to the first loss value and the second loss value.

[0150] Optionally, the first calculating module is specifically configured to:

[0151] obtain hidden features from a pre-trained model;

[0152] calculate the first loss value according to the defogging sample image, the clear sample image and the hidden features.

[0153] Optionally, the enhancement module is specifically configured to:

[0154] perform histogram equalization processing on the defogging sample image to obtain an initial enhanced image;

[0155] perform processing on the initial enhanced image by a Prewitt operator to obtain the enhanced image comprising the high-frequency edge information.

[0156] Optionally, the fusion unit is specifically configured to:

[0157] perform dynamic feature enhancement on the last layer down-sampling feature map by a bottleneck layer of the initial model to obtain an enhanced feature map;

[0158] perform feature extraction on the enhanced feature map by a plurality of up-sampling layers of the initial model, and fuse a plurality of down-sampling feature maps by a spatial feature transformation module to obtain a final feature map.

[0159] Optionally, the number of down-sampling layers of the initial model is 2, and the number of up-sampling layers is 2; the bottleneck layer comprises 6 feature attention modules and 2 deformation convolutions.

[0160] In the embodiment, the functions of the units and modules correspond to the functions of the units and modules in the foregoing Figures 1 to 4 embodiments, and thus will not be described herein.

[0161] Please refer to Figure 6 ,Figure 6 An embodiment of an electronic device provided in the present application includes:

[0162] The processor 601, the memory 602, the input output unit 603 and the bus 604;

[0163] The processor 601 is connected with the memory 602, the input output unit 603 and the bus 604;

[0164] The memory 602 stores a program, and the processor 601 invokes the program to execute Figures 1 to 4 the steps in the embodiment shown.

[0165] In the embodiment, the functions of the processor 601 correspond to the steps in the foregoing Figures 1 to 4 embodiment, which will not be repeated here.

[0166] The present application also provides a computer readable storage medium, which stores a program, and the program makes the computer execute the method in any one of the foregoing Figures 1 to 4 possible implementation manners when the program is executed on the computer.

[0167] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the system, device and unit described above can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0168] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented by other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0169] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0170] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0171] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, the integrated unit can be stored in a computer readable storage medium. Based on such an understanding, the technical solutions of the present application, essentially or in other words, the part of the prior art that contributes to the present application, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, read-only memory), a random access memory (RAM, random access memory), a magnetic disk or an optical disk, and various media that can store program codes.

Claims

1. An image dehazing method, characterized in that, include: Acquire a target meibomian gland image, wherein the target meibomian gland image is a near-infrared light image; The target meibomian gland image is input into a pre-trained dehazing model to obtain a dehazed meibomian gland image output by the dehazing model. The dehazing model is a lightweight neural network model with an encoder-decoder structure. The dehazing model fuses the feature maps of the upsampling layer and the downsampling layer by integrating a spatial feature transformation module. Before inputting the target meibomian gland image into the pre-trained dehazing model, the method further includes: Sample pairs are extracted from the dataset, each sample pair consisting of a foggy sample image and a corresponding clear sample image; Input the foggy sample image into the initial model; Feature extraction is performed on the foggy sample image through multiple downsampling layers of the initial model to obtain a multi-layer downsampling feature map; The bottleneck layer and multiple upsampling layers of the initial model are used to fuse the multi-layer downsampling feature map to obtain the final feature map; Generate a dehazed sample image based on the final feature map; Calculate the total loss value based on the dehazed sample image and the clear sample image; The initial model is determined to have converged based on the total loss value. If it has converged, the initial model is determined to be a dehazing model. If it has not converged, the initial model is updated in reverse and the next round of training is performed.

2. The method according to claim 1, characterized in that, The step of calculating the total loss value based on the dehazed sample image and the clear sample image includes: Image enhancement is performed on the dehazed sample image to obtain an enhanced image including high-frequency edge information; Calculate the first loss value based on the dehazed sample image and the clear sample image; Calculate a second loss value based on the dehazed sample image and the enhanced image; The total loss value is calculated based on the first loss value and the second loss value.

3. The method according to claim 2, characterized in that, The step of calculating the first loss value based on the dehazed sample image and the clear sample image includes: Obtain hidden features from a pre-trained model; The first loss value is calculated based on the dehazed sample image, the clear sample image, and the hidden features.

4. The method according to claim 2, characterized in that, The step of enhancing the dehazed sample image to obtain an enhanced image including high-frequency edge information includes: Histogram equalization is performed on the dehazed sample images to obtain an initial enhanced image; The initial enhanced image is processed by the Prewitt operator to obtain an enhanced image that includes high-frequency edge information.

5. The method according to claim 1, characterized in that, The step of fusing features from the bottleneck layer and multiple upsampling layers of the initial model to obtain the final feature map includes: The bottleneck layer of the initial model is used to dynamically enhance the features of the last downsampled feature map, resulting in an enhanced feature map. The enhanced feature map is extracted by multiple upsampling layers of the initial model, and the multi-layer downsampling feature map is fused using the spatial feature transformation module to obtain the final feature map.

6. The method according to claim 1, characterized in that, The initial model has 2 downsampling layers and 2 upsampling layers; the bottleneck layer includes 6 feature attention modules and 2 deformable convolutional layers.

7. An image dehazing device, characterized in that, include: An acquisition unit is used to acquire a target meibomian gland image, wherein the target meibomian gland image is a near-infrared light image; The input unit is used to input the target meibomian gland image into a pre-trained dehazing model to obtain the dehazed meibomian gland image output by the dehazing model. The dehazing model is a lightweight neural network model with an encoder-decoder structure. The dehazing model fuses the feature maps of the upsampling layer and the downsampling layer by integrating a spatial feature transformation module. The device further includes: An extraction unit is used to extract sample pairs from a dataset, the sample pair comprising a foggy sample image and a corresponding clear sample image; A sample input unit is used to input the foggy sample image into the initial model; An extraction unit is used to extract features from the foggy sample image through multiple downsampling layers of the initial model to obtain a multi-layer downsampling feature map; The fusion unit is used to perform feature fusion on the multi-layer downsampled feature map through the bottleneck layer and multiple upsampling layers of the initial model to obtain the final feature map; A generation unit is used to generate a dehazed sample image based on the final feature map; The calculation unit is used to calculate the total loss value based on the dehazed sample image and the clear sample image; The judgment unit is used to determine whether the initial model has converged based on the total loss value. If it has converged, the initial model is determined to be a dehazing model; if it has not converged, the initial model is updated in reverse and the next round of training is performed.

8. An electronic device, characterized in that, include: Processor, memory, input / output units, and bus; The processor is connected to the memory, the input / output unit, and the bus; The memory stores a program, and the processor calls the program to execute the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains a program that, when executed on a computer, causes the computer to perform the method as described in any one of claims 1 to 6.

Images