Image processing devices and air conditioners

Abstract

An image processing apparatus and air conditioner include: an acquisition unit configured to acquire low-light and infrared images; and a processing unit comprising: a low-light image enhancement algorithm model, which includes: a gradient reconstruction module comprising: a frequency domain processing branch configured to transform the gradient map of the low-light image from the spatial domain to the frequency domain to extract the phase spectrum and amplitude spectrum; and a spatial domain processing branch configured to fuse the gradient map of the low-light image with the infrared image in the channel dimension; the gradient reconstruction module adds the residual of the spatial domain processing branch to the amplitude spectrum to obtain an enhanced amplitude spectrum; an amplitude-enhanced gradient map of the low-light image is obtained through an inverse Fourier transform between the enhanced amplitude spectrum and the original phase spectrum; and an enhanced visible light image is generated through an encoder-decoder architecture based on the gradient features of the amplitude-enhanced gradient map of the low-light image and the image features of the low-light image; this application has high accuracy and is suitable for use in smart home appliances.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more particularly to an image processing apparatus and an air conditioner. Background Technology

[0002] Smart home appliances can utilize visible light images in various ways, primarily in areas such as intelligent monitoring and security, smart home control, health monitoring, entertainment and interaction, and appliance management. By acquiring visible light images and processing them, performing pattern recognition, data analysis, and control feedback, smart home appliances enhance their functionality and user experience. However, under low-light conditions, the quality of visible light images deteriorates, leading to reduced recognition accuracy, and impacting real-time performance and response speed.

[0003] Low-light image enhancement can employ traditional methods, such as Retinex-based enhancement methods, gamma correction, and histogram equalization. The drawbacks of these traditional methods are their sensitivity to parameters and reliance on manual adjustment, leading to difficulties in achieving ideal enhancement results and potentially noticeable color discrepancies. With the development of artificial intelligence, deep learning-based low-light image enhancement algorithms have emerged, such as RetinexNet and RetinexFormer. However, some of these methods directly model the low-light domain as a normal-light domain, a process that results in unstable enhancement results and amplifies noise. Furthermore, some methods introduce additional prior knowledge, but this prior knowledge itself is violated, leading to the loss of detail in the enhanced results.

[0004] Chinese patent application (CN 115063329A) discloses a method and system for fusion and enhancement of visible light and infrared images in low-light environments, specifically including:

[0005] "S1, fuse the visible light image and the infrared image under low light environment to obtain a fused image that combines the texture information of the visible light image and the target thermal information in the infrared image;

[0006] S2 enhances the fused image to remove the influence of light intensity;

[0007] S3 sharpens the enhanced fused image to compensate for image contours, enhance edges and areas of abrupt grayscale changes, improve image clarity and contrast, and finally obtain the fused output image.

[0008] In step S1, fusion is performed by inputting a visible light image and an infrared image from a low-light environment into a pre-trained fusion network to obtain a fused image. The visible light image from the low-light environment is input into a feature branch, which includes a common convolutional layer and a gradient residual dense block. The gradient residual dense block includes a dense flow branch and a residual flow branch, used to extract deep features.

[0009] In the aforementioned comparison files, the visible light image is directly input into the gradient residual dense block of the feature branch. However, the gradient map generated based on the visible light image using the Sobel operator is contaminated with a large number of noise points. This noise contamination destroys the context and detail information in low-light images; the subsequent segmentation network's processing results based on inaccurate gradient information will inevitably have significant deviations, resulting in residual noise or artifacts. Furthermore, as described in step S2 of the comparison files, the aforementioned comparison files are based on Retinex theory for image enhancement. As mentioned earlier, the enhancement results will exhibit significant color differences. In addition, in smart home appliance application scenarios, lighting, and the dynamic changes of objects or people are very frequent. The Retine× algorithm needs to adapt to these changes in real time, and smart home appliances can easily encounter performance bottlenecks.

[0010] The information disclosed in this background section is only intended to enhance the understanding of the background technology of this application, and therefore may include prior art that is not known to those skilled in the art. Summary of the Invention

[0011] To address the aforementioned problems, the first aspect of this application provides an image processing apparatus.

[0012] In one or more embodiments of this application, an image processing apparatus includes: an acquisition unit configured to acquire a low-light image and an infrared image of a target scene; and a processing unit including: a low-light image enhancement algorithm model, comprising: a gradient reconstruction module including: a frequency domain processing branch configured to transform the gradient map of the low-light image from the spatial domain to the frequency domain to extract the phase spectrum and amplitude spectrum of the low-light image; and a spatial domain processing branch configured to fuse the gradient map of the low-light image with the infrared image in the channel dimension; the gradient reconstruction module adds the residual of the spatial domain processing branch to the amplitude spectrum output by the frequency domain processing branch to obtain an enhanced amplitude spectrum; and obtains an amplitude-enhanced gradient map of the low-light image through an inverse Fourier transform between the enhanced amplitude spectrum and the original phase spectrum; the low-light image enhancement algorithm model generates an enhanced visible light image based on the gradient features of the amplitude-enhanced gradient map of the low-light image and the image features of the low-light image through an encoder-decoder architecture.

[0013] The above technical solution has the following advantages or effects: It designs and proposes a low-light image enhancement algorithm model, introduces gradient prior knowledge containing rich detail texture information, and performs modal fusion with an infrared image that also has detailed features, reconstructs gradient feature information that is closer to that of a normal-light image, and can obtain more complete image detail structure features, ensuring that no detail loss occurs in the entire low-light image enhancement process.

[0014] In one or more embodiments of this application, a low-light image enhancement algorithm model includes: an encoder module, the encoder module including: multiple encoder layers; the encoder layer including: a first gradient-guided calibration unit, configured to: extract image channel importance scores and gradient channel importance scores using a linear neural network based on gradient features of the gradient map of the amplitude-enhanced low-light image and image features of the low-light image; generate processed image features based on the image channel importance scores and image features; generate processed gradient features based on the gradient channel importance scores and gradient features; generate main fusion features based on the processed image features and processed gradient features, and residual connections with the image features; and generate enhanced image features of the encoder layer based on the main fusion features.

[0015] The above technical solution has the following advantages or effects: the first gradient-guided calibration unit in the encoder module uses gradient feature information to guide the image feature enhancement process; in the processing of the entire encoder layer, gradient feature information is integrated into the image feature enhancement step, ensuring that the enhanced image features of the final encoder layer can better reflect the details and structural information of the image.

[0016] In one or more embodiments of this application, a first gradient-guided calibration unit is configured to perform the following steps to extract image channel importance scores and gradient channel importance scores using a linear neural network based on gradient features of the gradient map of the amplitude-enhanced, low-light image and image features of the low-light image: performing layer normalization on the image features; performing layer normalization on the gradient features; stitching along the channel dimension to create a combined feature representation that combines image information and gradient information; and using a linear neural network to compress the stitched combined features at scale to extract the image feature channel importance scores of the image features and the gradient feature importance scores of the gradient features.

[0017] The above technical solution has the following advantages or effects: By performing layer normalization on image features and gradient features respectively, and concatenating them in the channel dimension, a combined feature representation combining image information and gradient information is created; then, a linear neural network is used to compress the concatenated combined features in scale, and the importance scores of image feature channels and gradient feature channels are extracted, thereby effectively fusing image and gradient information. Based on stable and accurate features, more representative channel importance scores are generated, resulting in enhanced image features of the encoder layer with higher quality.

[0018] In one or more embodiments of this application, a low-light image enhancement algorithm model includes: a decoder module, which includes: multiple decoder layers, each corresponding to an encoder layer; the decoder layer is configured to: fuse the image features output by the previous decoder layer and the enhanced image features of the corresponding encoder layer to generate fused image features; the decoder layer further includes: a second gradient-guided calibration unit, which is configured to: extract image channel importance scores and gradient channel importance scores using a linear neural network based on the gradient features of the gradient map of the amplitude-enhanced low-light image and the fused image features; generate processed fused image features based on the image channel importance scores and the fused image features; generate processed gradient features based on the gradient channel importance scores and gradient features; generate main fused features based on the processed fused image features and the processed gradient features, and the residual connection with the fused image features; generate enhanced fused image features based on the main fused features; the low-light image enhancement algorithm model generates an enhanced visible light image based on the enhanced image features generated by the decoder module.

[0019] The above technical solution has the following advantages or effects: By introducing a first gradient-guided calibration unit and a second gradient-guided calibration unit in the encoder layer and decoder layer respectively, the image feature enhancement process is guided by gradient feature information, ensuring the effective extraction and preservation of image details and structural information. The second gradient-guided calibration unit in the decoder layer further fuses image and gradient information based on skip connections, generating higher-quality enhanced image features. This dual-guided mechanism not only improves the stability and accuracy of feature extraction but also ensures that the final enhanced visible light image has higher clarity and detail richness under low-light conditions.

[0020] In one or more embodiments of this application, the second gradient-guided calibration unit is configured to perform the following steps to extract image channel importance scores and gradient channel importance scores using a linear neural network based on gradient features of the gradient map of the amplitude-enhanced, low-light image and fused image features: performing layer normalization on the fused image features; performing layer normalization on the gradient features; stitching along the channel dimension to create a combined feature representation that combines image information and gradient information; and using a linear neural network to compress the stitched combined features at scale to extract the image feature channel importance scores of the image features and the gradient feature importance scores of the gradient features.

[0021] The above technical solution has the following advantages or effects: the second gradient-guided calibration unit in the decoder layer extracts the channel importance score of the fused image features and gradient features through layer normalization processing, feature splicing and linear neural network compression, thereby generating more representative enhanced image features and ensuring that the final enhanced image features can better reflect the details and structural information of the image.

[0022] In one or more embodiments of this application, the low-light image enhancement algorithm model is trained based on a dataset, which includes low-light image samples, normal-light image samples, and infrared image samples; the low-light image samples and normal-light image samples are in one-to-one correspondence, and the normal-light image samples and low-light image samples are in one-to-one correspondence; the loss function of the gradient reconstruction module is established based on the gradient maps of the low-light images and the normal-light images generated by the gradient reconstruction module with amplitude enhancement.

[0023] The above technical solution has the following advantages or effects: by establishing the loss function of the gradient reconstruction module based on the gradient map of the amplitude-enhanced low-light image and the gradient map of the normal-light image generated by the gradient reconstruction module, the gradient map of the amplitude-enhanced low-light image is closest to the gradient of the normal-light image, thereby increasing the effect of image enhancement.

[0024] In one or more embodiments of this application, the low-light image enhancement algorithm model is trained based on a dataset, which includes low-light image samples, normal-light image samples, and infrared image samples; the low-light image samples and normal-light image samples are in one-to-one correspondence, and the normal-light image samples and low-light image samples are in one-to-one correspondence; the loss function of the low-light image enhancement algorithm model is established based on the enhanced visible light image and the normal-light image.

[0025] The above technical solution has the following advantages or effects: by using a dataset containing low-light image samples, normal-light image samples, and infrared image samples for training, the model is able to learn effective image enhancement features under various lighting conditions. In particular, by establishing a loss function based on enhanced visible light images and normal-light images, the model can directly optimize the difference between the enhanced image and the target image, thereby improving the quality and accuracy of the enhanced image. This method not only utilizes multimodal data (such as infrared images) to supplement the information in low-light images, but also ensures that the model generates high-quality enhanced images under low-light conditions through effective loss function design.

[0026] In one or more embodiments of this application, the loss function can be used to calculate one or more of pixel consistency loss, perceptual consistency loss, amplitude spectrum consistency loss, edge consistency loss, and gradient consistency loss.

[0027] The above technical solution has the following advantages or effects: By comprehensively considering multiple loss functions, including pixel consistency loss, perceptual consistency loss, amplitude spectrum consistency loss, edge consistency loss, and gradient consistency loss, the quality of the enhanced image can be comprehensively evaluated and optimized. This combination of multiple loss functions not only ensures the accuracy of the enhanced image at the pixel level, but also considers the image's perceptual quality, frequency domain characteristics, edge details, and gradient information, thereby generating more realistic and detailed enhanced images. Through this comprehensive optimization strategy, the model can generate high-quality enhanced visible light images under low-light conditions, significantly improving image clarity and visual effects.

[0028] In one or more embodiments of this application, the linear neural network includes global average pooling, a first linear layer, a ReLU activation function, a second linear layer, and a Sig moid activation function.

[0029] The above technical solution has the following advantages or effects: by using a linear neural network composed of global average pooling, a first linear layer, a ReLU activation function, a second linear layer, and a Sig moid activation function, feature information can be effectively extracted and compressed. Global average pooling compresses spatial dimension information into channel dimension information. The first linear layer and the ReLU activation function introduce nonlinearity, increasing the complexity of feature representation. The second linear layer and the Sig moid activation function compress the features to between 0 and 1, generating channel importance scores. This design is not only simple and efficient, but also accurately evaluates and adjusts the importance of image and gradient features, thereby generating more representative enhanced image features and improving the image enhancement effect.

[0030] A second aspect of this application provides an air conditioner, including an indoor unit comprising: an infrared imaging device capable of sampling an infrared image of an air-conditioned room; a camera device capable of sampling a visible light image of the air-conditioned room, the visible light image including a low-light image; and a processor comprising: a low-light image enhancement algorithm model, including: a gradient reconstruction module, comprising: a frequency domain processing branch configured to transform the gradient map of the low-light image from the spatial domain to the frequency domain to extract the phase spectrum and amplitude spectrum of the low-light image; and a spatial domain processing branch configured to fuse the gradient map of the low-light image with the infrared image in the channel dimension; gradient reconstruction... The module adds the residual of the spatial domain processing branch to the amplitude spectrum output by the frequency domain processing branch to obtain the enhanced amplitude spectrum; through the inverse Fourier transform between the enhanced amplitude spectrum and the original phase spectrum, the gradient map of the amplitude-enhanced low-light image is obtained; the low-light image enhancement algorithm model generates an enhanced visible light image through an encoder-decoder architecture based on the gradient features of the amplitude-enhanced low-light image gradient map and the image features of the low-light image; the controller is configured to perform one or more of face recognition, emotion recognition, gender recognition, age recognition, and vital sign recognition based on the enhanced visible light image, and control the operation of at least the indoor unit based on the recognition results.

[0031] The above technical solution has the following advantages or technical effects: The air conditioner provided in this application is particularly suitable for use in low light conditions. In addition to different nighttime use environments, it also includes cinemas, recording studios, museums, laboratories or wards, etc., so that facial recognition, gesture recognition, emotion recognition, gender recognition, age recognition and vital sign recognition can be accurately realized without adjusting the light, and corresponding control can be performed. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a schematic block diagram of the structure of an image processing apparatus provided in some embodiments of the present invention;

[0034] Figure 2 This is a schematic block diagram of the structure of an image processing apparatus provided in some embodiments of the present invention;

[0035] Figure 3 This is a schematic block diagram of the structure of the processing unit in an image processing apparatus provided in some embodiments of the present invention;

[0036] Figure 4 This is a network architecture diagram of the gradient reconstruction module in an image processing apparatus provided in some embodiments of the present invention;

[0037] Figure 5 This is a schematic block diagram of the encoder module in an image processing apparatus provided in some embodiments of the present invention;

[0038] Figure 6 This is a schematic block diagram of the decoder module in an image processing apparatus provided in some embodiments of the present invention;

[0039] Figure 7 This is a schematic diagram of the encoder layer in an image processing apparatus provided in some embodiments of the present invention;

[0040] Figure 8 This is a schematic diagram of the decoder layer in an image processing apparatus provided in some embodiments of the present invention;

[0041] Figure 9 This is a network architecture diagram of the first gradient-guided calibration unit in an image processing apparatus provided in some embodiments of the present invention;

[0042] Figure 10 This is a network architecture diagram of the second gradient-guided calibration unit in an image processing apparatus provided in some embodiments of the present invention;

[0043] Figure 11 This is a flowchart of a first gradient-guided calibration unit in an image processing apparatus provided in some embodiments of the present invention;

[0044] Figure 12 A flowchart of a second gradient-guided calibration unit in an image processing apparatus provided in some embodiments of the present invention;

[0045] Figure 13 This is a partial training flowchart of the gradient reconstruction module in an image processing apparatus provided in some embodiments of the present invention;

[0046] Figure 14 This is a network architecture diagram of a low-light image enhancement algorithm model in an image processing apparatus provided in some embodiments of the present invention;

[0047] Figure 15 These are schematic block diagrams of the structure of an air conditioner provided in some embodiments of the present invention;

[0048] In the diagram: 1. Image processing device; 10. Acquisition unit; 20. Processing unit; 201. Central processing unit; 202. Read-only memory; 203. Random access memory; 204. Input / output interface; 205. Bus; 30. Low-light image enhancement algorithm model; 301. Gradient reconstruction module; 302. Encoder module; 303. Decoder module; 401. Frequency domain processing branch; 402. Spatial domain processing branch; 501. Encoder layer; 502. Decoder layer; 601. First gradient guidance calibration unit; 602. Second gradient guidance calibration unit; 100. Air conditioner; 71. Processor; 72. Controller. Detailed Implementation

[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0050] In the description of this application, it should be understood that the terms "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0051] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "multiple" means two or more.

[0052] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0053] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" the second feature includes the first feature directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0054] The following disclosure provides many different embodiments or examples for implementing various structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the invention. Furthermore, reference numerals and / or letters may be repeated in different examples; such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed. In addition, examples of various specific processes and materials are provided in this invention, but those skilled in the art will recognize the application of other processes and / or the use of other materials.

[0055] Figure 1 The illustration shows the functional configuration of the image processing apparatus 11 disclosed in this application.

[0056] In one or more embodiments of this application, the image processing apparatus 11 includes an acquisition unit 10 and a processing unit 20.

[0057] In one or more embodiments of this application, the acquisition unit 10 may be integrated into the image processing apparatus 11.

[0058] In one or more embodiments of this application, the acquisition unit 10 may also be connected to the image processing device 11 via wired or wireless communication.

[0059] In one or more embodiments of this application, the acquisition unit 10 is configured to acquire an image of a target scene. The image includes an infrared image and a visible light image, and the visible light image includes a low-light image.

[0060] In one or more embodiments of this application, infrared images and visible light images are acquired by an infrared imaging device and a camera device, respectively.

[0061] Infrared imaging devices are equipment that use infrared radiation to create images. All objects emit infrared radiation of different wavelengths depending on their temperature; the higher the temperature, the greater the radiation intensity and the shorter the wavelength. Infrared imaging devices detect infrared wavelength signals and convert them into different electrical signals to capture thermal energy, which is then further converted into images (or videos) for observation and analysis. The images output by infrared imaging devices are usually called infrared images or thermal images, where different colors represent different temperatures. For example, red or white can represent heat sources, while blue can represent cooler areas.

[0062] A camera device, on the other hand, is an imaging device that captures visible light waves to create an image. It converts light signals into image signals, thereby recording the corresponding scene. Camera devices are the most common type of video recording equipment.

[0063] In one or more embodiments of this application, the infrared imaging device and the camera device have a viewing angle that covers the same target scene. The viewing angles of the infrared imaging device and the camera device can be fixed or adjustable.

[0064] like Figure 2 As shown, the processing unit 20 includes a central processing unit 201, a read-only memory 202 (ROM), and a random access memory 203 (RAM). The central processing unit 201 executes various processing tasks according to a program stored in the read-only memory 202 or a program loaded from a storage unit into the random access memory 203. Furthermore, the random access memory 203 appropriately stores data required by the central processing unit 201 to execute these tasks. The central processing unit 201, the read-only memory 202, and the random access memory 203 are interconnected via a bus 205. An input / output interface 204 is also connected to the bus 205.

[0065] like Figure 3 As shown, the processing unit 20 is equipped with a low-light image enhancement algorithm model 30. This model, based on image processing technology, improves the quality of images captured under low-light conditions. By extracting and enhancing the gradient features of the low-light image and combining information from the infrared image, it generates a higher-quality visible light image. The image processing device 1 is particularly suitable for use with smart home appliances to improve their performance under low-light conditions, such as baby monitors, scene lighting, and smart refrigerators.

[0066] In one or more embodiments of this application, such as Figure 3 and Figure 4 As shown, the low-light image enhancement algorithm model 30 includes a gradient reconstruction module 301.

[0067] The gradient reconstruction module 301 is a key part of the low-light image enhancement algorithm model 30. It extracts and enhances the gradient features of low-light images through frequency domain and spatial domain processing methods.

[0068] In one or more embodiments of this application, the gradient reconstruction module 301 includes a frequency domain processing branch 401 and a spatial domain processing branch 402.

[0069] In one or more embodiments of this application, the frequency domain processing branch 401 is configured to process the gradient map I of the low-light image. gradient_low The spatial domain is transformed to the frequency domain (via Fourier transform, FFT) to extract the phase and amplitude spectra of the low-light image. The phase spectrum contains structural information of the low-light image, while the amplitude spectrum contains intensity information.

[0070] In one or more embodiments of this application, the spatial domain processing branch 402 is configured to fuse the gradient map of the low-light image with the infrared image in the channel dimension, and to use the thermal radiation information in the infrared image to supplement and enhance the details of the low-light image.

[0071] In one or more embodiments of this application, the gradient reconstruction module 301 adds the residual of the spatial domain processing branch 402 to the amplitude spectrum output by the frequency domain processing branch 401 to obtain an enhanced amplitude spectrum; through the inverse Fourier transform (1FFT) between the enhanced amplitude spectrum and the original phase spectrum, the gradient map I of the amplitude-enhanced low-light image is obtained. gradient_enhance .

[0072] In one or more embodiments of this application, the low-light image enhancement algorithm model 30 generates an enhanced visible light image based on the gradient features of the gradient map of the low-light image with amplitude enhancement and the image features of the low-light image, through an encoder-decoder architecture. The encoder extracts the features of the image, and the decoder reconstructs the enhanced image from the features.

[0073] The low-light image enhancement algorithm model 30 provided in this application introduces gradient prior knowledge containing rich detail texture information and performs modal fusion with an infrared image that also has detailed features to reconstruct gradient feature information that is closer to that of a normal-light image. This results in more complete image detail structure features and ensures that no detail loss occurs during the entire low-light image enhancement process.

[0074] The encoder-decoder architecture can adopt the U-Net architecture or other common architecture designs. The specific design of the encoder-decoder will be described in detail below.

[0075] like Figure 3 and Figure 5 As shown, in one or more embodiments of this application, the encoder-decoder architecture includes an encoder module 302, which includes a plurality of encoder layers 501.

[0076] like Figure 3 and Figure 6 As shown, in one or more embodiments of this application, the encoder-decoder architecture includes a decoder module 303, which includes multiple decoder layers 502.

[0077] In one or more embodiments of this application, multiple encoder layers 501 and multiple decoder layers 502 are symmetrically designed. The multiple encoder layers 501 progressively downsample the image to extract features, while the multiple decoder layers 502 progressively upsample the image to reconstruct it. The multiple decoder layers 502 employ skip connections, which pass the features output by the encoder layers 501 to the corresponding decoder layers 502, helping the decoder layers 502 recover detailed information.

[0078] In one or more embodiments of this application, encoder layer 501 includes a first gradient-guided calibration unit 601.

[0079] like Figure 7 and Figure 9 As shown, the first gradient-guided calibration unit 601 is configured as follows:

[0080] Based on the gradient features of the gradient map of the amplitude-enhanced low-light image and the image features of the low-light image, a linear neural network is used to extract the image channel importance score and the gradient channel importance score.

[0081] Image features are generated based on image channel importance scores and image features;

[0082] Gradient features are generated based on gradient channel importance scores and gradient features after processing;

[0083] The main fusion features are generated based on the processed image features and the processed gradient features, as well as the residual connections with the image features.

[0084] Enhanced image features of encoder layer 501 are generated based on the main fusion features.

[0085] The first gradient-guided calibration unit 601 in encoder module 302 uses gradient feature information to guide the image feature enhancement process; throughout the processing of encoder layer 501, gradient feature information is integrated into the image feature enhancement steps to ensure that the enhanced image features of the final encoder layer 501 can better reflect the details and structural information of the image.

[0086] In one or more embodiments of this application, the first gradient-guided calibration unit 601 is configured to perform the following steps to extract image channel importance scores and gradient channel importance scores using a linear neural network based on gradient features of the gradient map of the amplitude-enhanced, low-light image and image features of the low-light image:

[0087] Perform layer normalization on image features;

[0088] Perform layer normalization on the gradient features;

[0089] The data is stitched along the channel dimension to create a combined feature representation that combines image information and gradient information;

[0090] A linear neural network is used to compress the stitched combined features at scale to extract the image feature channel importance scores of image features and the gradient feature importance scores of gradient features.

[0091] By performing layer normalization on image features and gradient features separately and concatenating them along the channel dimension, a combined feature representation combining image and gradient information is created. Then, a linear neural network is used to compress the concatenated combined features at scale, extracting the channel importance scores of image features and gradient features. This effectively fuses image and gradient information, generating more representative channel importance scores based on stable and accurate features, resulting in enhanced image features of encoder layer 501 with higher quality.

[0092] like Figure 8 and Figure 10 As shown, the decoder layer 502, based on skip connections, fuses the image features output by the previous decoder layer with the enhanced image features of the corresponding encoder layer 501 to generate fused image features; the decoder layer 502 also includes a second gradient-guided calibration unit 602. The architecture of the second gradient-guided calibration unit 602 is similar to that of the first gradient-guided calibration unit 601.

[0093] The second gradient-guided calibration unit 602 is configured as follows:

[0094] Based on gradient features of gradient maps of amplitude-enhanced low-light images and fused image features, a linear neural network is used to extract image channel importance scores and gradient channel importance scores.

[0095] The fused image features are generated based on the importance scores of image channels and the fused image features.

[0096] Gradient features are generated based on gradient channel importance scores and gradient features after processing;

[0097] The main fusion features are generated based on the processed fused image features and the processed gradient features, as well as the residual connections with the fused image features.

[0098] Generate enhanced fused image features based on key fusion features;

[0099] The low-light image enhancement algorithm model 30 generates an enhanced visible light image based on the enhanced image features generated by the decoder module 303.

[0100] By introducing a first gradient-guided calibration unit 601 and a second gradient-guided calibration unit 602 in the encoder layer 501 and decoder layer 502 respectively, the image feature enhancement process is guided by gradient feature information, ensuring the effective extraction and preservation of image details and structural information. The second gradient-guided calibration unit 602 in decoder layer 502 further fuses image and gradient information based on skip connections, generating higher-quality enhanced image features. This dual-guided mechanism not only improves the stability and accuracy of feature extraction but also ensures that the final enhanced visible light image has higher clarity and detail richness under low-light conditions.

[0101] Similarly, the second gradient-guided calibration unit 602 is configured to perform the following steps to extract image channel importance scores and gradient channel importance scores using a linear neural network based on gradient features of the gradient map of the amplitude-enhanced, low-light image and fused image features: performing layer normalization on the fused image features; performing layer normalization on the gradient features; stitching along the channel dimension to create a combined feature representation that combines image information and gradient information; and using a linear neural network to compress the stitched combined features at scale to extract the image feature channel importance scores of the image features and the gradient feature importance scores of the gradient features.

[0102] The second gradient-guided calibration unit 602 in the decoder layer 502 extracts the channel importance score of the fused image features and gradient features through layer normalization, feature splicing and linear neural network compression, thereby generating more representative enhanced image features and ensuring that the final enhanced image features can better reflect the details and structural information of the image.

[0103] In one or more embodiments of this application, the low-light image enhancement algorithm model 30 is trained based on a dataset, which includes low-light image samples, normal-light image samples, and infrared image samples; the low-light image samples and normal-light image samples are in one-to-one correspondence, and the normal-light image samples and low-light image samples are in one-to-one correspondence.

[0104] In one or more embodiments of this application, the loss function of the gradient reconstruction module 301 is established based on the gradient map of the amplitude-enhanced low-light image and the gradient map of the normal-light image generated by the gradient reconstruction module 301.

[0105] In one or more embodiments of this application, a loss function is established for a low-light image enhancement algorithm model 30 based on enhanced visible light images and normal lighting images.

[0106] By training the model on a dataset containing samples of low-light, normal-light, and infrared images, the model is able to learn effective image enhancement features under various lighting conditions. Specifically, by establishing a loss function based on enhanced visible light and normal-light images, the model can directly optimize the difference between the enhanced and target images, thereby improving the quality and accuracy of the enhanced images. This approach not only utilizes multimodal data (such as infrared images) to supplement the information in low-light images but also ensures, through effective loss function design, that the model generates high-quality enhanced images under low-light conditions.

[0107] In one or more embodiments of this application, the loss function can be used to calculate one or more of pixel consistency loss, perceptual consistency loss, amplitude spectrum consistency loss, edge consistency loss, and gradient consistency loss.

[0108] By comprehensively considering multiple loss functions, including pixel consistency loss, perceptual consistency loss, amplitude spectrum consistency loss, edge consistency loss, and gradient consistency loss, the quality of enhanced images can be fully evaluated and optimized. This combination of multiple loss functions not only ensures the accuracy of the enhanced image at the pixel level but also considers the image's perceptual quality, frequency domain characteristics, edge details, and gradient information, thereby generating more realistic and detailed enhanced images. Through this comprehensive optimization strategy, the model can generate high-quality enhanced visible light images under low-light conditions, significantly improving image clarity and visual effects.

[0109] In one or more embodiments of this application, the linear neural network includes global average pooling, a first linear layer, a ReLU activation function, a second linear layer, and a Sig moid activation function.

[0110] By using a linear neural network consisting of global average pooling, a first linear layer, a ReLU activation function, a second linear layer, and a Sigma-moid activation function, feature information can be effectively extracted and compressed. Global average pooling compresses spatial dimension information into channel dimension information. The first linear layer and the ReLU activation function introduce non-linearity, increasing the complexity of feature representation. The second linear layer and the Sigma-moid activation function compress the features to a range of 0 to 1, generating channel importance scores. This design is not only concise and efficient but also accurately evaluates and adjusts the importance of image and gradient features, thereby generating more representative enhanced image features and improving the image enhancement effect.

[0111] In one or more embodiments of this application, the image processing apparatus 1 executes a low-light image enhancement algorithm guided by infrared information, the network architecture of which is as follows: Figure 14 As shown.

[0112] In one or more embodiments of this application, the basic algorithm for low-light image enhancement based on infrared information guidance is divided into multiple stages.

[0113] The following sections will introduce each of the multiple stages separately.

[0114] Phase 1: Generating the dataset.

[0115] In one or more embodiments of this application, image data samples in the dataset are acquired by visible light imaging devices and infrared imaging devices.

[0116] In one or more embodiments of this application, the imaging resolution of the visible light imaging device is 1024×768 pixels.

[0117] In one or more embodiments of this application, the imaging resolution of the infrared imaging device is 640 × 512 pixels.

[0118] In one or more embodiments of this application, the dataset includes visible light images and infrared images.

[0119] In one or more embodiments of this application, the visible light image further includes low-light image samples and normal-light image samples; the low-light image samples and normal-light image samples correspond one-to-one, with each low-light image sample corresponding to one normal-light image sample. The content of the low-light image samples and normal-light image samples is the same, but their brightness differs. Low-light image samples refer to images taken in low-light environments, while normal-light image samples refer to images taken in well-lit environments.

[0120] In one or more embodiments of this application, a data pair is formed by a one-to-one correspondence between low-light image samples and normal-light image samples.

[0121] In one or more embodiments of this application, low-light image samples and normal-light image samples can be distinguished by brightness values. For example, when the average brightness value is lower than a preset brightness threshold, it is considered a low-light image sample; while when the average brightness value is within a certain preset range, it is considered a normal-light image sample.

[0122] In one or more embodiments of this application, the infrared image includes an infrared image sample.

[0123] In one or more embodiments of this application, infrared image samples are acquired simultaneously when acquiring low-light image samples; the low-light image samples and infrared image samples correspond one-to-one, with each low-light image sample corresponding to one infrared image sample.

[0124] In one or more embodiments of this application, a data pair is formed by a one-to-one correspondence between low-light image samples and infrared image samples.

[0125] In one or more embodiments of this application, the dataset is processed by manual data screening to remove data pairs with large deviations.

[0126] In one or more embodiments of this application, an infrared image sample is expanded to 1024x768 pixels and aligned with a visible light image sample using a bilinear interpolation method.

[0127] Bilinear interpolation is used to generate new pixel values ​​when an image is scaled or warped. It achieves smooth image transformation by calculating the value of the target pixel after considering the values ​​of its four nearest neighbors.

[0128] In one or more embodiments of this application, the imaging resolution of the infrared imaging device is 640×512 pixels, which is different from the imaging resolution of the visible light imaging device. In order to align the infrared image samples with the visible light image samples, the infrared image samples are adjusted to the same resolution as the visible light image samples.

[0129] The bilinear interpolation method is a mature algorithm in the prior art and is not the focus of this application, so it will not be described in detail here.

[0130] The final dataset is constructed using multiple filtered data pairs, and then divided into training and test sets. The training set is used for algorithm training and tuning, while the test set is used for performance evaluation. The ratio of training to test sets is determined based on specific needs, ensuring data independence and representativeness to more accurately reflect the algorithm's actual performance.

[0131] Phase 2: Constructing a low-light image enhancement algorithm model 30.

[0132] In one or more embodiments of this application, the low-light image enhancement algorithm model 30 is a novel low-light image enhancement algorithm model based on infrared image combination.

[0133] In one or more embodiments of this application, the low-light image enhancement algorithm model 30 includes: a gradient reconstruction module 301, an encoder module 302, and a decoder module 303.

[0134] In one or more embodiments of this application, encoder module 302 includes a first gradient-guided calibration unit 601.

[0135] In one or more embodiments of this application, the decoder module 303 includes a second gradient-guided calibration unit 602.

[0136] First, a gradient map of the low-light image is generated. A gradient map of a low-light image is an image generated by calculating the gradients of pixel intensity changes in a low-light image. The gradient map of a low-light image highlights edge and detail information; specifically, the horizontal and vertical gradients of the low-light image can be calculated using convolution operations with the Sobel operator, and further, the gradient magnitude and direction are calculated using the square root of the sum of the squares of the horizontal and vertical gradients and the arctangent.

[0137] The gradient maps of low-light images are contaminated with numerous noise points, which destroys the context and detail information within the image. Due to inaccurate gradient priors, residual noise or artifacts may remain in the enhanced low-light images. This application proposes a hybrid interactive method combining spatial and frequency domains to compensate for the damaged gradient maps of low-light images. The spatial domain of a low-light image refers to its representation in its original pixel coordinate system; while in the frequency domain, the low-light image is represented as a superposition of different frequency components, reflecting different structures and features within the image. On one hand, the spatial domain of a low-light image contains structural integrity and rich semantic information, forming the basis for visual understanding. On the other hand, the phase spectrum in the frequency domain of a low-light image primarily captures its structural essence, while the amplitude spectrum encodes semantic nuances including brightness levels and noise components. Furthermore, infrared low-light images are inherently insensitive to light and can clearly reflect the edge contours of objects, including a wealth of structural information. This structural information can assist in the reconstruction of the low-light image gradient.

[0138] In one or more embodiments of this application, the gradient reconstruction module 301 is used to adjust the amplitude spectrum of the image using global feature information while maintaining the structural integrity of the image. This method effectively reduces noise and enhances the semantic importance of the gradient image, thereby providing more robust information for the low-light image enhancement process.

[0139] Figure 4 The network architecture of the gradient reconstruction module 301 is shown below. In one or more embodiments of this application, the gradient reconstruction module 301 includes a frequency domain processing branch 401 and a spatial domain processing branch 402. The frequency domain processing branch 401 transforms the gradient map of the low-light image from the spatial domain to the frequency domain, extracting the phase spectrum and amplitude spectrum of the low-light image. The spatial domain processing branch 402 fuses the gradient map of the low-light image with the infrared image in the channel dimension to extract rich texture and structural information. The gradient reconstruction module 301 adds the residual from the spatial domain processing branch 402 to the amplitude spectrum to obtain an enhanced amplitude spectrum. Through the inverse Fourier transform between the enhanced amplitude spectrum and the original phase spectrum, the gradient map of the low-light image with enhanced amplitude is obtained.

[0140] In one or more embodiments of this application, the spatial domain processing branch 402 includes a 1×1 convolutional layer with a Leaky ReLU activation function to increase the dimension of the gradient map of the low-light image; after the 1×1 convolutional layer with a Leaky ReLU activation function, grouped convolution is used: a 5×5 convolutional layer with four groups and a Leaky ReLU activation function is set. In this case, the input channels are divided into four groups, and each group performs a 5×5 convolution operation independently to extract spatial information and maintain a certain expressive power while reducing the number of parameters and computational complexity. Finally, the dimension is reduced by a 1×1 convolutional layer, and the residual of the amplitude spectrum is obtained by a sig moid activation function.

[0141] The above process can be represented by the following formula:

[0142] A, P = FFT(I) gradient_low ),

[0143] A res =Function(Concat(I gradient_low I infrared )),

[0144] I gradient_enhance =IFFT(A+A) res ,P),

[0145] in:

[0146] A and P are the amplitude and phase spectra of the gradient map of the low-light image, respectively, and FFT(*) represents the Fourier transform; Igradient_low Represents the gradient map of a low-light image; A, P = FFT(I gradient_low ) indicates that a Fourier transform is performed on the gradient map of the low-light image to obtain the frequency domain representation of the low-light image. The frequency domain representation of the low-light image includes the amplitude spectrum and the phase spectrum.

[0147] I infrared Represents an infrared image, Concat(I gradient_low I infrared ) represents the gradient map I of a low-light image. gradient_low and I infrared Perform concatenation along the channel dimension; Function(Concat(I gradient_low I infrared This indicates that a series of convolution operations are performed on the stitched image, including dimensionality enhancement, spatial information extraction, and dimensionality reduction, ultimately yielding the residual A of the amplitude spectrum. res A res The residual between the amplitude of the gradient map of a low-light image and the ideal amplitude reflects the insufficiency or deviation of the gradient map of the low-light image in the amplitude spectrum.

[0148] IFFT stands for Inverse Fourier Transform, I gradient_enhance Gradient plot representing an enhanced low-light image.

[0149] The gradient reconstruction module 301 effectively utilizes a limited number of convolutional layers to fully leverage the inherent semantic and structural information in the original image, minimizing information loss during denoising and enhancement processes and ensuring high-quality results.

[0150] Encoder module 302 is used to progressively compress gradient features and extract high-level abstract features. In one or more embodiments of this application, encoder module 302 includes three encoder layers 501. Each encoder layer processes features at a different scale. Extracting features at different scales allows encoder module 302 to capture different levels of contextual information in the image. Shallower encoder layers focus more on local details, while deeper encoder layers can capture broader contextual information; for example, the first layer extracts low-level features, such as edges; the second layer extracts mid-level features, such as shapes; and the third layer extracts high-level features, such as contextual information. Therefore, with each encoder layer 501, the scale of the feature map changes: the width becomes half of its original value, the height becomes half of its original value, and the number of channels doubles.

[0151] Each encoder layer 501 includes a first gradient guided calibration unit 601.

[0152] Image features X obtained after a 3×3 convolution operation on a low-light image inThe gradient map of the amplitude-enhanced, low-light image is fed into encoder layer 501 as one input; the gradient feature G is obtained after a 3×3 convolution operation. in It is fed into encoder layer 501 as another input.

[0153] In the first gradient-guided calibration unit 601, the following is performed: Figure 11 The steps are shown.

[0154] Step S101: Process image features X in Layer normalization is performed, and the gradient feature G is also processed. in Perform layer normalization.

[0155] Step S102: Concatenate along the channel dimension to create a combined feature representation that combines image information and gradient information.

[0156] Step S103: Use a linear neural network to compress the stitched combined features at different scales to extract image features X. in Image feature channel importance scores and gradient features G in Gradient feature channel importance score.

[0157] The forward propagation process of a linear neural network includes global average pooling, a first linear layer, a ReLU activation function, a second linear layer, and a Sigmoid activation function.

[0158] Global average pooling performs a spatial averaging on each channel, converting the concatenated combined features into a scalar.

[0159] The first linear layer maps the feature vector after global flat pooling to a new feature space, resulting in a new vector. The ReLU activation function performs element-wise operations on the output of the first linear layer to learn complex feature representations and increase non-linearity.

[0160] The second linear layer and the Sigmoid function perform element-wise operations on the output of the second linear layer, compressing the importance scores of the channels to the range [0, 1]. The score of each channel represents the importance of that channel.

[0161] Based on the output of the Sigmoid function, image features X are further extracted. in Image feature channel importance scores and gradient features G in The gradient feature channel importance scores (e.g., through segmentation operations, which will be explained below) effectively identify channels that contain information that is crucial for image enhancement.

[0162] In one or more embodiments of this application, the loss function and optimization algorithm used in the backpropagation process of the linear neural network can be algorithms known in the prior art, such as using the cross-entropy loss function as the loss function and stochastic gradient descent, Adam, etc. as the optimization algorithm. The true labels of the samples used to train the linear neural network can be generated using existing prediction algorithm models, such as SHAP, LIME, or other known prediction models, or can be generated based on pre-configured rules.

[0163] The above-mentioned steps of the first gradient-guided calibration unit 601 can be expressed by the following formula:

[0164] U = Concat(LayerNorm(X) in G in ))

[0165] σ u =Linear(ReLU(Linear(GAP(U))))

[0166] Where LayerNorm represents layer normalization, Concat represents concatenating the normalized features along the channel dimension, U represents the combined features after concatenation, GAP represents global average pooling, Linear represents a linear layer, ReLU is the ReLU activation function, and σ u This is the output of the second linear layer.

[0167] Step S104: Compare the importance scores of image feature channels with image feature X. in Element-wise multiplication is performed to obtain the processed image features; the gradient feature channel importance scores are then compared with the gradient feature G. in The processed gradient features are obtained by performing element-wise multiplication.

[0168] Step S105: Add the processed image features and the processed gradient features element-wise. That is, fuse the processed image features and the processed gradient features by adding them element-wise, combining the enhanced image features with the fine spatial details captured by the gradient features to obtain a more comprehensive representation.

[0169] Step S106: Add the processed image features and the processed gradient features element by element to the image feature X. in Residual connections are established, and their outputs are primarily fused features S. For example, residual connections can be established through element-wise addition. This residual connection preserves the original image features, allowing the linear neural network to learn residual enhancements rather than the full transformation. The primary fused feature S combines image and gradient information and emphasizes channels crucial for effective enhancement.

[0170] The above-mentioned steps of the first gradient-guided calibration unit 601 can be expressed by the following formula:

[0171] σ1, σ2 = Split(Sigmoid(σ u ))

[0172] S=(1+σ1)×X in +σ2×G in

[0173] In the above formula, σ u For the output of the second linear layer, Sigmoid(σ) u The first step () applies the Sigmoid activation function to the output of the second linear layer, mapping the input to the (0, 1) interval. The second step () is a segmentation operation (also called a partitioning operation), which divides the input into two parts, σ1 and σ2, based on the output of the Sigmoid function. σ1 represents the importance score of the image feature channels, and σ2 represents the importance score of the gradient feature channels. The importance score of each channel reflects its contribution. The segmentation operation can allocate the calculated "channel importance scores" according to their original sources, namely image features and gradient features, thereby distinguishing their contributions.

[0174] Further, the importance score σ1 of the image feature channel is compared with the image feature X. in The processed image features are obtained by performing element-wise multiplication, i.e., σ1×X. in The gradient feature channel importance score σ² is compared with the gradient feature G. in Element-wise multiplication yields the processed gradient feature, i.e., σ²×G. in The processed image features and the processed gradient features are added element-wise, resulting in σ1×X. in +σ2×G in The result of element-wise summing the processed image features and the processed gradient features is compared with the image feature X. in Establishing a residual connection yields σ1×X in +σ2×G in +X in Thus obtain

[0175] S=(1+σ1)×X in +σ2×G in

[0176] Among them, S is the main fusion feature.

[0177] Step S107: Further perform a three-layer convolution operation on the main fusion features.

[0178] The main fusion feature S is first normalized, then passed through a 3×3 convolution, a 1×1 convolution, a LeakyReLU activation function, and another 1×1 convolution. It is then identity-connected to the main fusion feature S to obtain the output X of the first gradient-guided calibration unit 601. out To enrich the texture information in the image, the output X of the first gradient-guided calibration unit 601 out This refers to the enhanced image features corresponding to encoder layer 501.

[0179] Corresponding to encoder module 302, the decoder progressively expands the low-dimensional feature representation into a high-dimensional image. Decoder module 303 includes three decoder layers 502. Each decoder layer 502 corresponds to one encoder layer 501. In each decoder layer 502, the enhanced image feature maps in the corresponding encoder layer 501 are directly connected via skip connections.

[0180] Each decoder layer 502 includes a second gradient-guided calibration unit 602. The network architecture of the second gradient-guided calibration unit 602 is basically the same as that of the first gradient-guided calibration unit 601.

[0181] In one or more embodiments of this application, in each decoder layer 502, the following is performed: Figure 12 The steps are shown.

[0182] For the first decoder layer 502, it is to process the enhanced image features X corresponding to the encoder layer 501. out and gradient features G in As the input to the second gradient-guided calibration unit 602, the operation of the second gradient-guided calibration unit 602 for the first decoder layer 502 is similar to that of the first gradient-guided calibration unit 601, and will not be repeated here.

[0183] For other decoder layers 502, the previous layer is executed first, and the image features output by the decoder layer and the enhanced image features X of the corresponding encoder layer 501 are processed. out The fusion is performed on the fused features, and the output is used as one input to the second gradient-guided calibration unit 602 in the decoder layer 502. This is denoted as the image feature of the second gradient-guided calibration unit 602 in the encoder layer 501, i.e., the fused image feature X′. in Gradient feature G in It serves as another input to the second gradient-guided calibration unit 602 in the decoder layer 502.

[0184] In the second gradient-guided calibration unit 602 of the decoder layer 502, the following steps are performed:

[0185] Step S201: Analyze the fused image features X′in Layer normalization is performed, and the gradient feature G is also processed. in Perform layer normalization.

[0186] Step S202: Concatenate along the channel dimension to create a combined feature representation that combines image information and gradient information;

[0187] Step S203: Use a linear neural network to compress the stitched combined features at scale to extract image features X′. in Image feature channel importance scores and gradient features G in Gradient feature channel importance score.

[0188] The forward propagation process of a linear neural network includes global average pooling, a first linear layer, a ReLU activation function, a second linear layer, and a Sigmoid activation function.

[0189] Global average pooling performs a spatial averaging on each channel, converting the concatenated combined features into a scalar.

[0190] The first linear layer maps the feature vector after global flat pooling to a new feature space, resulting in a new vector. The ReLU activation function performs element-wise operations on the output of the first linear layer to learn complex feature representations and increase non-linearity.

[0191] The second linear layer and the Sigmoid function perform element-wise operations on the output of the second linear layer, compressing the importance scores of the channels to the range [0, 1]. The score of each channel represents the importance of that channel.

[0192] Based on the output of the Sigmoid function, image features X are further extracted. in Image feature channel importance scores and gradient features G in The gradient feature channel importance scores (e.g., through segmentation operations, which will be explained below) effectively identify channels that contain information that is crucial for image enhancement.

[0193] In one or more embodiments of this application, the loss function and optimization algorithm used in the backpropagation process of the linear neural network can be algorithms known in the prior art, such as using the cross-entropy loss function as the loss function and stochastic gradient descent, Adam, etc. as the optimization algorithm. The true labels of the samples used to train the linear neural network can be generated using existing prediction algorithm models, such as SHAP, LIME, or other known prediction models, or can be generated based on pre-configured rules.

[0194] The above-mentioned steps of the second gradient guided calibration unit 602 can be expressed by the following formula:

[0195] U′=Concat(LayerNorm(X′ in G in ))

[0196] σ′ u =Linear(ReLU(Linear(GAP(U′))))

[0197] Where LayerNorm represents layer normalization, Concat represents concatenating the normalized features along the channel dimension, U′ represents the combined features after concatenation; GAP represents global average pooling; Linear represents a linear layer; ReLU is the ReLU activation function; σ′ u This is the output of the second linear layer.

[0198] Step S204: Compare the importance scores of image feature channels with image feature X′ in Element-wise multiplication is performed to obtain the processed image features; the gradient feature channel importance scores are then compared with the gradient feature G. in The processed gradient features are obtained by performing element-wise multiplication.

[0199] Step S205: Add the processed image features and the processed gradient features element-wise. That is, fuse the processed image features and the processed gradient features by adding them element-wise, combining the enhanced image features with the fine spatial details captured by the gradient features to obtain a more comprehensive representation.

[0200] Step S206: Add the processed image features and the processed gradient features element by element to the image feature X′. in Residual connections are established, and their outputs are primarily fused features S′. For example, residual connections can be established through element-wise addition. This residual connection preserves the original image features, allowing the linear neural network to learn residual enhancements rather than the full transformation. The primary fused feature s combines image and gradient information and emphasizes channels crucial for effective enhancement.

[0201] The above-mentioned steps of the second gradient guided calibration unit 602 can be expressed by the following formula:

[0202] σ′1,σ′2=Split(Sigmoid(σ′ u ))

[0203] S′=(1+σ′1)×X′ in +σ′2×G in

[0204] In the above formula, σ′ uThe output of the second linear layer, Sigmoid(σ′) u The first step () applies the Sigmoid activation function to the output of the second linear layer, mapping the input to the (0, 1) interval. The second step () is the splitting operation, which divides the input into two parts σ′1 and σ′2 based on the output of the Sigmoid function. σ′1 represents the importance score of the image feature channels, and σ′2 represents the importance score of the gradient feature channels. For example, the splitting operation can make the sum of σ′1 and σ′2 equal to 1. It can also be attention-based splitting, statistical feature-based splitting, learning-based splitting, or even equal division.

[0205] Further, the image feature channel importance score σ′1 is compared with the image feature X′. in The processed image features are obtained by performing element-wise multiplication, i.e., σ′1×X′. in The gradient feature channel importance score σ′2 is compared with the gradient feature G. in Element-wise multiplication yields the processed gradient feature, i.e., σ′2×G. in The processed image features and the processed gradient features are added element-wise, resulting in σ′1×X′. in +σ′2×G in The result of element-wise summing of the processed image features and the processed gradient features is compared with the image feature X′. in Establishing a residual connection yields σ′1×X′ in +σ′2×G in +X′ in Thus obtain

[0206] S′=(1+σ′1)×X in +σ′2×G in

[0207] Among them, S′ is the main fusion feature.

[0208] Step S207: Further perform a three-layer convolution operation on the main fused features.

[0209] The main fusion feature S′ is first normalized, then passed through a 3×3 convolution, a 1×1 convolution, a Leaky ReLU activation function, and another 1×1 convolution. It is then identity-connected with the main fusion feature s to obtain the output X′ of the second gradient-guided calibration unit 602. out To enrich the texture information in the image, the output X′ of the second gradient-guided calibration unit 602 is... out This corresponds to the enhanced image features of decoder layer 502.

[0210] Enhancing image features in the decoder can help restore image details and remove noise, thereby generating clearer and higher-quality images.

[0211] The following describes the training process of the low-light image enhancement algorithm model 30, specifically including: Figure 13 The steps are shown.

[0212] Step S301: Select data pairs consisting of low-light image samples and infrared image samples from the dataset; select data pairs consisting of low-light image samples and normal-light image samples from the dataset.

[0213] Step S302: Generate the gradient map of the low-light image.

[0214] For example, the gradient map of a low-light image can be generated using the Sobel operator.

[0215] Step S303: Input the gradient map of the low-light image and the infrared image into the gradient reconstruction module 301; calculate the loss using the amplitude-enhanced gradient map of the low-light image and the gradient map of the normal-light image generated by the gradient reconstruction module 301.

[0216] The loss function is:

[0217]

[0218] Among them, I gradient_enhance Gradient maps representing amplitude enhancement in low-light images; I gradient_gt This represents the gradient map calculated using the same Sobel operator for a normally lit image; H represents the image height; W represents the image width. It is a normalization factor that ensures the value of the loss function does not change with the image size; L gradient By measuring the average Euclidean distance between the gradient map representing the amplitude-enhanced low-light image and the normal-light image, the gradient map of the amplitude-enhanced low-light image is made as close as possible to the gradient of the normal-light image when minimizing the loss function, thereby increasing the image enhancement effect.

[0219] The gradient map of the amplitude-enhanced low-light image is used to extract gradient features G0 through a 3×3 convolution, and the gradient features G0 of the low-light image are also extracted through a 3×3 convolution. Gradient features G0 and G0 are then input into encoder module 302 and decoder module 303, respectively, and finally, a three-channel enhanced visible light image I is recovered through a single 3×3 convolution layer. enhance .

[0220] The overall loss function of the low-light image enhancement algorithm model 30 includes:

[0221]

[0222] L vgg =||φ(I enhance )-φ(I gt )||1,

[0223]

[0224] L total =L pixel +L vgg +L amphase +λ1L edge +λ2L gradient ,

[0225] in:

[0226] L pixel Pixel consistency loss is used to measure the enhancement of visible light image I. enhance and normal lighting image sample I gt Pixel-level differences between them ε is the normalization factor, H is the image height, W is the image width, 3 is the number of channels (the number of channels can be adjusted according to actual needs), and ε is a very small positive number to prevent division by zero.

[0227] L vgg Perceptual consistency loss is used to measure the enhancement of visible light image I. enhance The difference between the image sample and the image sample under normal lighting in the high-level feature space; φ(*) represents the output of certain layers of the pre-trained VGG-19 network. For example, the output of certain layers of the pre-trained VGG-19 network refers to the feature map extracted from a specific layer of the VGG-19 network; ||(*)||1 represents the L1 norm, which calculates the absolute difference between feature maps; the VGG-19 network is a deep convolutional neural network, widely used in image classification and feature extraction tasks. A specific layer can be a convolutional layer in the VGG-19 network. The output of the convolutional layer, as a feature map, contains high-level features of the image and can capture the structural and semantic information of the image;

[0228] L amphase Amplitude spectral consistency loss is used to measure the enhancement of visible light image I. enhance The difference in amplitude spectrum between the sample and the image under normal illumination in the frequency domain, where Amp(*) represents the amplitude spectrum obtained by Fourier transform; the remaining parameters are defined in the same way as the pixel consistency loss.

[0229] L edge Represents edge consistency loss, used to measure the enhancement of visible light image I. enhance The difference in edge features between the sample and the normally illuminated image. This represents the edge feature map obtained from the image using the Laplacian operator; the remaining parameters are defined in the same way as the pixel consistency loss.

[0230] L gradient This represents the gradient consistency loss, used to measure the enhancement of visible light images I. enhance The difference in gradient between L and normal illumination image samples gradient The formula is the same as the loss function formula in step S, the difference being that it is used to calculate the enhanced visible light image I. enhance The squared difference between the gradient of the image and the gradient of the normally illuminated image sample.

[0231] L total This represents the overall loss function, which comprehensively considers consistency at the pixel level, perceptual level, frequency domain, edge level, and gradient level to improve the image enhancement effect. λ1 and λ2 are used to adjust the weights of edge consistency loss and gradient consistency loss, and can be adjusted according to actual needs; for example, they can be set to 0.1 and 10 respectively.

[0232] In one or more embodiments of this application, the loss function may be selected from one or more parts as needed.

[0233] In one or more embodiments of this application, where the primary concern is pixel consistency of the image, it is desirable to enhance the visible light image I. enhance When aiming to approximate normal lighting image samples at the pixel level, only pixel consistency loss L can be selected. pixel That is, L total =L pixel .

[0234] Similarly, in one or more embodiments of this application, only perceptual consistency loss, amplitude spectrum consistency loss, edge consistency loss, and gradient consistency loss may be selected.

[0235] In one or more embodiments of this application, multiple loss functions can be selected and combined as needed. The weights of the loss functions can be tuned experimentally.

[0236] After calculating the overall loss, the low-light image enhancement algorithm model 30 is trained and updated using the Adam optimizer.

[0237] A second aspect of this application provides an air conditioner 100, such as... Figure 15 As shown. Air conditioner 100 includes an indoor unit. The indoor unit includes an infrared imaging device and a camera device. The infrared imaging device can sample infrared images of the air-conditioned room, and the camera device can sample visible light images of the air-conditioned room. The visible light images include low-light images.

[0238] The air conditioner 100 also includes a processor 71, which includes a low-light image enhancement algorithm model 30, comprising: a gradient reconstruction module 301, which includes: a frequency domain processing branch 401 configured to transform the gradient map of the low-light image from the spatial domain to the frequency domain to extract the phase spectrum and amplitude spectrum of the low-light image; and a spatial domain processing branch 402 configured to fuse the gradient map of the low-light image with the infrared image in the channel dimension; the gradient reconstruction module 301 adds the residual of the spatial domain processing branch 402 to the amplitude spectrum output by the frequency domain processing branch 401 to obtain an enhanced amplitude spectrum; and obtains the amplitude-enhanced gradient map of the low-light image through the inverse Fourier transform between the enhanced amplitude spectrum and the original phase spectrum.

[0239] The low-light image enhancement algorithm model 30 generates enhanced visible light images based on the gradient features of the gradient map of the low-light image and the image features of the low-light image through an encoder-decoder architecture.

[0240] The controller 72 is configured to perform one or more of the following based on enhanced visible light images: face recognition, gesture recognition, emotion recognition, gender recognition, age recognition, and vital sign recognition, and to control the operation of at least the indoor unit based on the recognition results.

[0241] Controlling the operation of indoor or outdoor units based on the results of facial recognition, gesture recognition, emotion recognition, gender recognition, age recognition, and vital sign recognition can employ algorithms disclosed in the prior art, which are not the focus of this application and will not be elaborated upon here.

[0242] The processor 71 can also be set in the indoor unit's wired controller, central controller, or cloud server.

[0243] The air conditioner 100 provided in this application is particularly suitable for use in low light conditions, including not only different nighttime use environments, but also cinemas, recording studios, museums, laboratories or hospital wards, so that the above-mentioned identification can be achieved and the corresponding control can be executed without adjusting the light.

[0244] In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0245] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An image processing apparatus, comprising: The acquisition unit is configured to acquire low-light images and infrared images of the target scene; Its characteristic is that it further includes: The processing unit includes: Low-light image enhancement algorithm models, which include: The gradient reconstruction module includes: A frequency domain processing branch is configured to transform the gradient map of the low-light image from the spatial domain to the frequency domain to extract the phase spectrum and amplitude spectrum of the low-light image; and The spatial domain processing branch is configured to fuse the gradient map of the low-light image with the infrared image in the channel dimension. The gradient reconstruction module adds the residual of the spatial domain processing branch to the amplitude spectrum output by the frequency domain processing branch to obtain an enhanced amplitude spectrum; and obtains the gradient map of the low-light image with enhanced amplitude through the inverse Fourier transform between the enhanced amplitude spectrum and the original phase spectrum. The low-light image enhancement algorithm model generates an enhanced visible light image based on the gradient features of the gradient map of the amplitude-enhanced low-light image and the image features of the low-light image through an encoder-decoder architecture.

2. The image processing apparatus according to claim 1, characterized in that: The low-light image enhancement algorithm model includes: The encoder module includes: Multiple encoder layers; the encoder layer includes: The first gradient-guided calibration unit is configured as follows: Based on the gradient features of the gradient map of the amplitude-enhanced low-light image and the image features of the low-light image, a linear neural network is used to extract image channel importance scores and gradient channel importance scores. The processed image features are generated based on the image channel importance scores and image features; The processed gradient features are generated based on the gradient channel importance scores and gradient features; The main fusion features are generated based on the processed image features and the processed gradient features, as well as the residual connections with the image features; Enhanced image features of the encoder layer are generated based on the main fusion features.

3. The image processing apparatus according to claim 2, characterized in that: The first gradient-guided calibration unit is configured to perform the following steps to extract image channel importance scores and gradient channel importance scores using a linear neural network based on gradient features of the gradient map of the amplitude-enhanced, low-light image and image features of the low-light image: The image features are subjected to layer normalization processing; The gradient features are then subjected to layer normalization. The data is stitched along the channel dimension to create a combined feature representation that combines image information and gradient information; A linear neural network is used to compress the stitched combined features at scale to extract the image feature channel importance scores of image features and the gradient feature importance scores of gradient features.

4. The image processing apparatus according to claim 2, characterized in that: The low-light image enhancement algorithm model includes: Decoder module, the decoder module comprising: Multiple decoder layers, each corresponding to an encoder layer; each decoder layer is configured to fuse image features output from the previous decoder layer and enhanced image features from the corresponding encoder layer to generate fused image features; the decoder layer further includes: The second gradient-guided calibration unit is configured as follows: Based on the gradient features of the gradient map of the amplitude-enhanced low-light image and the fused image features, a linear neural network is used to extract image channel importance scores and gradient channel importance scores. Based on the image channel importance scores and the fused image features, fused image features are generated after processing; The processed gradient features are generated based on the gradient channel importance scores and gradient features; The main fusion features are generated based on the processed fused image features and the processed gradient features, as well as the residual connections with the fused image features; Enhanced fused image features are generated based on the main fusion features; The low-light image enhancement algorithm model generates an enhanced visible light image based on the enhanced image features generated by the decoder module.

5. The image processing apparatus according to claim 4, characterized in that: The second gradient-guided calibration unit is configured to perform the following steps to extract image channel importance scores and gradient channel importance scores using a linear neural network, based on gradient features of the gradient map of the amplitude-enhanced, low-light image and the fused image features: The fused image features are subjected to layer normalization processing; The gradient features are then subjected to layer normalization. The data is stitched along the channel dimension to create a combined feature representation that combines image information and gradient information; A linear neural network is used to compress the stitched combined features at scale to extract the image feature channel importance scores of image features and the gradient feature importance scores of gradient features.

6. The image processing apparatus according to any one of claims 1 to 5, characterized in that: The low-light image enhancement algorithm model is trained based on a dataset, which includes low-light image samples, normal-light image samples, and infrared image samples; the low-light image samples and the normal-light image samples are in one-to-one correspondence. The loss function of the gradient reconstruction module is established based on the gradient maps of the amplitude-enhanced low-light image and the gradient map of the normal-light image generated by the gradient reconstruction module.

7. The image processing apparatus according to any one of claims 1 to 5, characterized in that: The low-light image enhancement algorithm model is trained based on a dataset, which includes low-light image samples, normal-light image samples, and infrared image samples; the low-light image samples and the normal-light image samples are in one-to-one correspondence. The loss function of the low-light image enhancement algorithm model is established based on the enhanced visible light image and the normal lighting image.

8. The image processing apparatus according to claim 7, characterized in that: The loss function can be used to calculate one or more of the following: pixel consistency loss, perceptual consistency loss, amplitude spectrum consistency loss, edge consistency loss, and gradient consistency loss.

9. The image processing apparatus according to claim 2 or 4, characterized in that: The linear neural network includes global average pooling, a first linear layer, a ReLU activation function, a second linear layer, and a Sigmoid activation function.

10. Air conditioners, including: Indoor unit, which includes: An infrared imaging device, which can sample infrared images of an air-conditioned room; A camera device capable of sampling visible light images of an air-conditioned room, the visible light images including low-light images; Its characteristic is that it further includes: Processor, the processor comprising: Low-light image enhancement algorithm models, which include: The gradient reconstruction module includes: A frequency domain processing branch is configured to transform the gradient map of the low-light image from the spatial domain to the frequency domain to extract the phase spectrum and amplitude spectrum of the low-light image; and The spatial domain processing branch is configured to fuse the gradient map of the low-light image with the infrared image in the channel dimension. The gradient reconstruction module adds the residual of the spatial domain processing branch to the amplitude spectrum output by the frequency domain processing branch to obtain an enhanced amplitude spectrum; and obtains the gradient map of the low-light image with enhanced amplitude through the inverse Fourier transform between the enhanced amplitude spectrum and the original phase spectrum. The low-light image enhancement algorithm model generates an enhanced visible light image based on the gradient features of the gradient map of the amplitude-enhanced low-light image and the image features of the low-light image through an encoder-decoder architecture. The controller is configured to perform one or more of the following based on the enhanced visible light image: face recognition, emotion recognition, gender recognition, age recognition, and vital sign recognition, and to control the operation of at least the indoor unit based on the recognition results.

Images