Image processing method and device, electronic equipment and storage medium

Abstract

The invention provides an image processing method and device, electronic equipment and a storage medium. An infrared image, collected by an infrared sensor, of a target object is obtained, and a visible light image, collected by a visible light sensor, of the target object is obtained. And obtaining color features of the visible light image. The color features are used for reflecting color information of the target object. And obtaining high-frequency features of the infrared image, wherein the high-frequency features are used for reflecting detail information and texture information of the target object. And obtaining thermal radiation characteristics of the infrared image according to the high-frequency characteristics. The heat radiation characteristics are used for reflecting detail information, texture information and heat distribution information of the target object. And fusing the thermal radiation feature and the color feature to obtain a fused feature. And generating a fusion image of the target object according to the fusion features. The advantages of the infrared image and the advantages of the visible light image are combined, so that the quality of the image can be improved, for example, the quality of the image in a low-illumination environment is improved.

Description

Technical Field

[0001] This application belongs to the field of image technology, specifically relating to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, people typically use image sensors to acquire images. However, the amount of information in images acquired by different types of image sensors usually varies greatly, and the amount of information obtained by a single image sensor is limited, making it difficult to meet people's practical application needs for images.

[0003] Image fusion technology has emerged to address this issue, aiming to improve the visual quality of images and make the structures, edges, and details contained in the images clearer and more complete.

[0004] Image fusion technology combines images acquired by two or more image sensors to create a new image. The fused image boasts advantages such as rich detail and high contrast, making it widely applicable in fields like medical diagnosis, target recognition and tracking, video surveillance, intelligent transportation, and national defense, thus possessing significant social application value. Summary of the Invention

[0005] This application discloses an image processing method, apparatus, electronic device, and storage medium.

[0006] In a first aspect, this application discloses an image processing method, the method comprising:

[0007] Acquire an infrared image of the target object collected by an infrared sensor, and acquire a visible light image of the target object collected by a visible light sensor;

[0008] Obtain the color features of the visible light image; the color features are used to represent the color information of the target object;

[0009] The high-frequency features of the infrared image are obtained, and the high-frequency features are used to reflect the detailed information and texture information of the target object;

[0010] The thermal radiation features of the infrared image are obtained based on the high-frequency features; the thermal radiation features are used to reflect the detailed information, texture information and heat distribution information of the target object.

[0011] The thermal radiation feature and the color feature are fused together to obtain the fused feature;

[0012] A fused image of the target object is generated based on the fusion features.

[0013] In one optional implementation, obtaining the color features of the visible light image includes:

[0014] The visible light image is encoded to obtain its image features;

[0015] Obtain a color feature extraction network; the color feature extraction network includes N cascaded convolutional layers, the kernel size of the N convolutional layers decreases sequentially, and N is a positive integer greater than or equal to 2;

[0016] Based on the first convolutional layer in the color feature extraction network, feature extraction is performed on the image features to obtain the first convolutional color features;

[0017] Based on the second convolutional layer in the color feature extraction network, the first convolutional color features are extracted to obtain the second convolutional color features;

[0018] ...and so on...

[0019] Based on the Nth convolutional layer in the color feature extraction network, feature extraction is performed on the color features of the (N-1)th convolution to obtain the color features of the Nth convolution;

[0020] The color features of the visible light image are obtained based on the first convolutional color features, the second convolutional color features, ... and the Nth convolutional color features.

[0021] In one optional implementation, acquiring the high-frequency features of the infrared image includes:

[0022] The infrared image is subjected to wavelet transform processing to obtain the wavelet-transformed features;

[0023] A high-frequency feature extraction network is obtained; the high-frequency feature extraction network includes N cascaded convolutional layers, the kernel size of the N convolutional layers decreases sequentially, and N is a positive integer greater than or equal to 2;

[0024] Based on the first convolutional layer in the high-frequency feature extraction network, feature extraction is performed on the wavelet-transformed features to obtain the first convolutional high-frequency features;

[0025] Based on the second convolutional layer in the high-frequency feature extraction network, feature extraction is performed on the high-frequency features of the first convolution to obtain the high-frequency features of the second convolution;

[0026] ...and so on...

[0027] Based on the Nth convolutional layer in the high-frequency feature extraction network, feature extraction is performed on the high-frequency features of the (N-1)th convolution to obtain the high-frequency features of the Nth convolution.

[0028] The high-frequency features of the infrared image are obtained based on the first convolutional high-frequency features, the second convolutional high-frequency features, ... and the Nth convolutional high-frequency features.

[0029] In an optional implementation, obtaining the thermal radiation features of the infrared image based on the high-frequency features includes:

[0030] Extract the first thermal radiation feature of the infrared image from the first high-frequency feature of the first convolution;

[0031] The second thermal radiation feature of the infrared image is extracted from the second convolutional high-frequency feature;

[0032] ...and so on...

[0033] Extract the Nth thermal radiation feature of the infrared image from the Nth high-frequency features of the Nth convolution;

[0034] The thermal radiation characteristics of the infrared image are obtained based on the first thermal radiation characteristic, the second thermal radiation characteristic, ... and the Nth thermal radiation characteristic.

[0035] In an optional implementation, fusing the thermal radiation feature and the color feature to obtain the fused feature includes:

[0036] The Nth convolutional color feature is concatenated with the Nth thermal radiation feature to obtain the Nth concatenated feature;

[0037] The Nth splicing feature is spliced ​​with the (N-1)th thermal radiation feature to obtain the (N-1)th splicing feature;

[0038] ...and so on...

[0039] The second splicing feature is spliced ​​with the first thermal radiation feature to obtain the first splicing feature;

[0040] The fusion feature is obtained based on the first splicing feature.

[0041] In an optional implementation, obtaining the thermal radiation features of the infrared image based on the high-frequency features includes:

[0042] The high-frequency features are input into the thermal radiation feature extraction network so that the thermal radiation feature extraction network can extract features from the high-frequency features to obtain the thermal radiation features of the infrared image.

[0043] The thermal radiation feature extraction network includes: a thermal radiation feature extraction block, a multilayer perceptron, a first addition layer, a convolutional layer, a max pooling layer, an activation function, a first multiplication layer, and a second addition layer.

[0044] The input of the thermal radiation feature extraction network includes: the input of the thermal radiation feature extraction block;

[0045] The output of the thermal radiation feature extraction block is connected to the input of the multilayer perceptron, the input of the first addition layer, and the input of the second addition layer, respectively.

[0046] The output of the multilayer perceptron is connected to the input of the first additive layer;

[0047] The output of the first additive layer is connected to the input of the convolutional layer.

[0048] The output of the convolutional layer is connected to the input of the max pooling layer and the input of the first multiplication layer, respectively.

[0049] The output of the max pooling layer is connected to the input of the activation function;

[0050] The output of the activation function is connected to the input of the first multiplication layer;

[0051] The output of the first multiplication layer is connected to the input of the second addition layer;

[0052] The output of the thermal radiation feature extraction network includes the output of the second additive layer;

[0053] The thermal radiation feature extraction block includes: a first normalization layer, a first linear layer, a depthwise separable convolutional layer, a two-dimensional selective scanning network, a second normalization layer, a second linear layer, a second multiplication layer, a third linear layer, and a third addition layer.

[0054] The input of the thermal radiation feature extraction block includes: the input of the first normalization layer;

[0055] The output of the first normalization layer is connected to the input of the first linear layer, the input of the second linear layer, and the input of the third addition layer, respectively.

[0056] The output of the first linear layer is connected to the input of the depth-separable convolutional layer;

[0057] The output of the depth-separable convolutional layer is connected to the input of a two-dimensional selective scanning network;

[0058] The output of the two-dimensional selective scanning network is connected to the input of the second normalization layer;

[0059] The output of the second normalization layer is connected to the input of the second multiplication layer;

[0060] The output of the second multiplication layer is connected to the input of the third linear layer.

[0061] The output of the third linear layer is connected to the input of the third additive layer;

[0062] The output of the thermal radiation feature extraction block includes the output of the third additive layer.

[0063] In an optional implementation, the method further includes:

[0064] The fused image of the target object is subjected to at least one of the following processing methods to obtain the processed image of the target object. The processing methods include white balance processing, demosaic processing, and color correction processing.

[0065] Secondly, this application discloses an image processing apparatus, the apparatus comprising:

[0066] The first acquisition module is used to acquire an infrared image of the target object collected by an infrared sensor and a visible light image of the target object collected by a visible light sensor.

[0067] The second acquisition module is used to acquire the color features of the visible light image; the color features are used to reflect the color information of the target object.

[0068] The third acquisition module is used to acquire the high-frequency features of the infrared image, and the high-frequency features are used to reflect the detailed information and texture information of the target object;

[0069] The fourth acquisition module is used to acquire the thermal radiation features of the infrared image based on the high-frequency features; the thermal radiation features are used to reflect the detailed information, texture information and heat distribution information of the target object;

[0070] The fusion module is used to fuse the thermal radiation feature and the color feature to obtain a fused feature;

[0071] The generation module is used to generate a fused image of the target object based on the fusion features.

[0072] In one optional implementation, the second acquisition module includes:

[0073] Encoding unit, used to encode the visible light image to obtain the image features of the visible light image;

[0074] The first acquisition unit is used to acquire a color feature extraction network; the color feature extraction network includes N cascaded convolutional layers, the kernel size of the N convolutional layers decreases sequentially, and N is a positive integer greater than or equal to 2;

[0075] The first extraction unit is used to extract features from the image features based on the first convolutional layer in the color feature extraction network to obtain the first convolutional color feature; to extract features from the first convolutional color feature based on the second convolutional layer in the color feature extraction network to obtain the second convolutional color feature; ... and so on... to extract features from the (N-1)th convolutional color feature based on the Nth convolutional layer in the color feature extraction network to obtain the Nth convolutional color feature;

[0076] The second acquisition unit is used to acquire the color features of the visible light image based on the first convolutional color features, the second convolutional color features, ... and the Nth convolutional color features.

[0077] In one optional implementation, the third acquisition module includes:

[0078] The transformation unit is used to perform wavelet transform processing on the infrared image to obtain the wavelet transformed features;

[0079] The third acquisition unit is used to acquire the high-frequency feature extraction network; the high-frequency feature extraction network includes N cascaded convolutional layers, the kernel size of the N convolutional layers decreases sequentially, and N is a positive integer greater than or equal to 2;

[0080] The second extraction unit is used to extract features from the wavelet-transformed features based on the first convolutional layer in the high-frequency feature extraction network to obtain the first convolutional high-frequency features; to extract features from the first convolutional high-frequency features based on the second convolutional layer in the high-frequency feature extraction network to obtain the second convolutional high-frequency features; ... and so on... to extract features from the (N-1)th convolutional high-frequency features based on the Nth convolutional layer in the high-frequency feature extraction network to obtain the Nth convolutional high-frequency features;

[0081] The fourth acquisition unit is used to acquire the high-frequency features of the infrared image based on the first convolutional high-frequency features, the second convolutional high-frequency features, ... and the Nth convolutional high-frequency features.

[0082] In one optional implementation, the fourth acquisition module includes:

[0083] The third extraction unit is used to extract the first thermal radiation feature of the infrared image from the first convolutional high-frequency feature; extract the second thermal radiation feature of the infrared image from the second convolutional high-frequency feature; ... and so on ... extract the Nth thermal radiation feature of the infrared image from the Nth convolutional high-frequency feature;

[0084] The fifth acquisition unit is used to acquire the thermal radiation features of the infrared image based on the first thermal radiation feature, the second thermal radiation feature, ... and the Nth thermal radiation feature.

[0085] In one optional implementation, the fusion module includes:

[0086] A splicing unit is used to splice the Nth convolutional color feature with the Nth thermal radiation feature to obtain the Nth spliced ​​feature; splice the Nth spliced ​​feature with the (N-1)th thermal radiation feature to obtain the (N-1)th spliced ​​feature; ... and so on... splice the 2nd spliced ​​feature with the 1st thermal radiation feature to obtain the 1st spliced ​​feature;

[0087] The sixth acquisition unit is used to acquire the fusion feature based on the first splicing feature.

[0088] In one optional implementation, the fourth acquisition module includes:

[0089] An input unit is used to input the high-frequency features into a thermal radiation feature extraction network, so that the thermal radiation feature extraction network can extract features from the high-frequency features to obtain the thermal radiation features of the infrared image.

[0090] The thermal radiation feature extraction network includes: a thermal radiation feature extraction block, a multilayer perceptron, a first addition layer, a convolutional layer, a max pooling layer, an activation function, a first multiplication layer, and a second addition layer.

[0091] The input of the thermal radiation feature extraction network includes: the input of the thermal radiation feature extraction block;

[0092] The output of the thermal radiation feature extraction block is connected to the input of the multilayer perceptron, the input of the first addition layer, and the input of the second addition layer, respectively.

[0093] The output of the multilayer perceptron is connected to the input of the first additive layer;

[0094] The output of the first additive layer is connected to the input of the convolutional layer.

[0095] The output of the convolutional layer is connected to the input of the max pooling layer and the input of the first multiplication layer, respectively.

[0096] The output of the max pooling layer is connected to the input of the activation function;

[0097] The output of the activation function is connected to the input of the first multiplication layer;

[0098] The output of the first multiplication layer is connected to the input of the second addition layer;

[0099] The output of the thermal radiation feature extraction network includes the output of the second additive layer;

[0100] The thermal radiation feature extraction block includes: a first normalization layer, a first linear layer, a depthwise separable convolutional layer, a two-dimensional selective scanning network, a second normalization layer, a second linear layer, a second multiplication layer, a third linear layer, and a third addition layer.

[0101] The input of the thermal radiation feature extraction block includes: the input of the first normalization layer;

[0102] The output of the first normalization layer is connected to the input of the first linear layer, the input of the second linear layer, and the input of the third addition layer, respectively.

[0103] The output of the first linear layer is connected to the input of the depth-separable convolutional layer;

[0104] The output of the depth-separable convolutional layer is connected to the input of a two-dimensional selective scanning network;

[0105] The output of the two-dimensional selective scanning network is connected to the input of the second normalization layer;

[0106] The output of the second normalization layer is connected to the input of the second multiplication layer;

[0107] The output of the second multiplication layer is connected to the input of the third linear layer.

[0108] The output of the third linear layer is connected to the input of the third additive layer;

[0109] The output of the thermal radiation feature extraction block includes the output of the third additive layer.

[0110] In an optional implementation, the apparatus further includes:

[0111] The processing module is used to perform at least one of the following processing on the fused image of the target object to obtain the processed image of the target object, wherein the multiple processing includes: white balance processing, demosaic processing and color correction processing.

[0112] Thirdly, this application discloses an electronic device comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to perform the method as described in the first aspect.

[0113] Fourthly, this application discloses a non-transitory computer-readable storage medium that, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform the method described in the first aspect.

[0114] Fifthly, this application discloses a computer program product in which, when the instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to perform the method described in the first aspect.

[0115] The technical solution provided in this application has at least the following technical effects or advantages:

[0116] In this application, an infrared image of the target object is acquired by an infrared sensor, and a visible light image of the target object is acquired by a visible light sensor. Color features of the visible light image are acquired. These color features are used to represent the color information of the target object. High-frequency features of the infrared image are acquired, and these high-frequency features are used to represent the detail and texture information of the target object. Thermal radiation features of the infrared image are acquired based on the high-frequency features. These thermal radiation features are used to represent the detail, texture, and heat distribution information of the target object. The thermal radiation features and color features are fused to obtain a fused feature. A fused image of the target object is generated based on the fused feature.

[0117] This application combines the advantages of infrared images' stable photosensitive characteristics under low-light conditions (noise suppression characteristics) with the high-precision advantages of visible light images in color restoration (color preservation ability). It can generate a fused image that not only retains the detail information of the infrared image but also the color information of the visible light image. It can effectively suppress noise and restore image details under low-light conditions, and restore the true colors of the image. This makes the colors of the fused image realistic and undistorted, which can improve the visibility of the fused image and improve the quality and adaptability of the fused image under low-light conditions. The fused image is clearer and more natural in visual perception.

[0118] Thermal radiation features can provide information on the temperature distribution or heat distribution of a target object, as well as edge and texture information. This can help highlight the details of the target object, especially in low-light environments. Thermal radiation features are advantageous in restoring the edges, textures, and other details of an image, making the target object more obvious and clear.

[0119] As can be seen, this application improves the reconstruction effect of image details in low-light environments by combining the advantages of infrared images and visible light images, solving the problem that existing technologies cannot fully recover details in low-light environments. At the same time, it can reduce noise interference and effectively suppress noise, thereby improving the quality of image reconstruction in low-light environments.

[0120] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0121] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings.

[0122] In the attached diagram:

[0123] Figure 1 A flowchart illustrating the steps of an image processing method according to this application is shown.

[0124] Figure 2 A schematic diagram of a method for fusing infrared images with visible light images according to this application is shown.

[0125] Figure 3 A schematic diagram of a method for fusing infrared images with visible light images according to this application is shown.

[0126] Figure 4 A schematic diagram of a feature fusion architecture according to this application is shown.

[0127] Figure 5 A schematic diagram of a method for fusing infrared images with visible light images according to this application is shown.

[0128] Figure 6 A schematic diagram of a thermal radiation feature extraction network according to this application is shown.

[0129] Figure 7 A schematic diagram of a thermal radiation feature extraction block according to this application is shown.

[0130] Figure 8 A schematic diagram of the structure of an image processing apparatus according to this application is shown.

[0131] Figure 9 A schematic diagram of the structure of an electronic device according to this application is shown.

[0132] Figure 10 A schematic diagram of a storage medium according to this application is shown. Detailed Implementation

[0133] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0134] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application shall have the ordinary meaning as understood by one of ordinary skill in the art to which this application pertains.

[0135] For image fusion from multiple image sensors acquired under low light conditions, current methods include data-driven fusion and model-driven fusion.

[0136] Low illumination can be understood as light intensity below 0.1 lux (such as in environments with no moonlight at night or far from city lights).

[0137] Lux is a unit of luminance. When an object is uniformly illuminated and produces a luminous flux of 1 lumen over an area of ​​1 square meter, its luminance is 1 lux.

[0138] While both data-driven and model-driven fusion methods can fuse images acquired by multiple image sensors under low-light conditions, some problems still exist, such as:

[0139] Images captured under low-light conditions often lose a lot of detail due to excessive noise, making it difficult to even recognize some details.

[0140] Therefore, image fusion of multiple images acquired by different image sensors under low light conditions presents a significant challenge, especially in situations where light is scarce, current image fusion methods struggle to effectively recover image details.

[0141] In addition, the inventors discovered that infrared images can provide stable light sensitivity but lack color information, while visible light images (such as RAW images) can retain rich color data, but their high-frequency details are usually lost in low-light environments.

[0142] In view of this, in order to give full play to the advantages of both, this application proposes a new image fusion method that fuses infrared images and visible light images. By combining the advantages of infrared images and visible light images, the quality of the images can be improved, for example, the quality of images in low-light environments.

[0143] For example, this application combines the advantages of infrared images' stable photosensitive characteristics under low light conditions (noise suppression characteristics) with the high-precision advantages of visible light images in color restoration (color preservation ability). It can generate a fused image that not only retains the structural details of the infrared image but also the color information of the visible light image. It can effectively suppress noise and restore image details under low light conditions, and restore the true colors of the image. This makes the colors of the fused image realistic and undistorted, which can improve the visibility of the fused image. It can also improve the quality and adaptability of the fused image under low light conditions, making the fused image clearer and more natural in visual perception.

[0144] As can be seen, this application improves the reconstruction effect of high-frequency details in images under low-light conditions, especially the recovery effect of high-frequency information such as edges and textures, solving the problem that existing technologies cannot fully recover details under low-light conditions. At the same time, it can reduce noise interference and effectively suppress noise, thereby improving the quality of images recovered under low-light conditions.

[0145] The solution proposed in this application can also be widely used in many fields such as surveillance, autonomous driving, and nighttime photography.

[0146] This application achieves excellent results on multiple low-light image datasets, especially in terms of evaluation metrics such as PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity).

[0147] The scheme of this application will be described below with reference to the accompanying drawings.

[0148] Figure 1 A flowchart of the steps of an image processing method is shown. Figure 2 A schematic diagram of a method for fusing infrared images with visible light images is shown, wherein the method includes:

[0149] In step S101, the infrared image of the target object collected by the infrared sensor is acquired, and the visible light image of the target object collected by the visible light sensor is acquired.

[0150] Infrared sensors include: thermal infrared sensors, quantum infrared sensors, or focal plane array infrared sensors, etc.

[0151] Visible light sensors include: CCD (Charge-Coupled Device) sensors, CMOS (Complementary Metal-Oxide-Semiconductor) sensors, multispectral image sensors, and RGB (Red, Green, Blue) image sensors, etc.

[0152] In one embodiment of this application, the target object is located in a low-light environment. Thus, this step may obtain an infrared image of the target object collected by an infrared sensor in a low-light environment, and may also obtain a visible light image of the target object collected by a visible light sensor in a low-light environment.

[0153] Low-light environments refer to environments with low light intensity. In scenarios involving image acquisition or video recording, low-light environments include those where the light intensity is insufficient for a standard camera or image sensor to function properly, or environments where the light intensity is insufficient for a standard camera or image sensor to obtain clear or high-quality images.

[0154] Low-light environments include environments with light intensity less than a specific lux, which may include 10 lux, 7 lux, 5 lux, 3 lux, 2 lux, 1 lux, 0.5 lux, or 0.1 lux, etc. The specific value can be determined according to the actual situation, and this application does not limit it.

[0155] For example, a low-light environment can be a moonlight environment (about 0.1-1 lux) or a starlight environment (about 0.001-0.01 lux).

[0156] Visible light images include RAW images, which are unprocessed images, such as raw images captured directly by a camera or image sensor.

[0157] The target object includes a specific object or scene. Objects can include buildings, cars, bicycles, trees, people, or animals. Scenes can include squares, rivers, lawns, mountains, lakes, or forests.

[0158] The infrared image of the target object acquired by the infrared sensor is based on the same viewing angle as the visible light image of the target object acquired by the visible light sensor.

[0159] For example, the infrared sensor can be located at the same position as the visible light sensor, and the infrared sensor can be in the same direction of acquisition as the visible light sensor, so that both can acquire images of the target object from the same perspective.

[0160] In step S102, the color features of the visible light image are acquired. The color features are used to represent the color information of the target object.

[0161] The color features of a visible light image include: the frequency and distribution of each color in the visible light image, the statistical values ​​such as the mean and variance of each color in the visible light image (used to describe the overall characteristics of the color), and / or the spatial layout and interrelationship of each color in the visible light image (color consistency and local color distribution, etc.), which can reflect the color information of the target object.

[0162] Before obtaining the color features of a visible light image, color correction and / or noise reduction processing can be performed on the visible light image to preserve the color information of the visible light image as much as possible and avoid interference.

[0163] In step S103, high-frequency features of the infrared image are acquired. These high-frequency features are used to reflect the detailed information and texture information of the target object.

[0164] High-frequency features of infrared images include: texture information (e.g., recurring patterns or structures, such as the texture of leaves or skin), edge information (e.g., areas of significant brightness or color change, such as the outline or boundary of an object), detail information (e.g., minute changes in the image, such as scratches or blemishes on the surface of an object), and / or local variation information (e.g., local areas of rapid brightness or color change). These features reveal the detail and texture information of the target object.

[0165] Using the high-frequency features of infrared images can improve the contrast and clarity of the fused image, making the details in the fused image clearer and more complete.

[0166] In step S104, the thermal radiation features of the infrared image are obtained based on the high-frequency features. The thermal radiation features are used to reflect the detailed information, texture information, and heat distribution information of the target object.

[0167] Infrared images possess thermodynamic characteristics, such as thermal radiation features. The imaging principle of infrared images is based on the difference in thermal radiation distribution between the foreground target (such as the target object) and the background itself; therefore, infrared images contain rich thermal radiation information.

[0168] Extracting thermodynamic features (such as thermal radiation features) from infrared images during image restoration and using them together with high-frequency and color features for image restoration can indeed improve the quality of the restored image.

[0169] The thermal radiation characteristics of infrared images can reveal the temperature distribution of a target object, as well as its texture, edge, detail, and / or local variation information.

[0170] This technology supplements image information from multiple dimensions, enabling the restored image to contain more comprehensive details. By incorporating the thermal radiation characteristics of infrared images, it can supplement the temperature information of objects missing due to insufficient lighting, thereby more accurately reconstructing the physical state of the target object.

[0171] The thermal radiation characteristics of infrared images can highlight the differences in thermal radiation of target objects, making them more prominent in the image. By combining high-frequency and color features, not only can the edges and textures of target objects be clearly presented, but the contrast between the target object and the background can also be further enhanced through thermal radiation information, making the target stand out even more.

[0172] In low-light conditions, images are easily affected by noise. However, the thermal radiation characteristics of infrared images, reflecting the thermal radiation properties of objects, are relatively stable and not easily affected by changes in lighting. The stability of the thermal radiation characteristics of infrared images can be used to suppress noise and reduce misjudgments and interference caused by insufficient lighting.

[0173] The thermal radiation characteristics of infrared images can help distinguish noise from real signals in an image. Since noise usually does not have clear thermal radiation characteristics, while the real signals of an object have certain thermal radiation patterns, comparison can help to more accurately locate and suppress noise sources.

[0174] In step S105, the thermal radiation features and color features are fused to obtain the fused features.

[0175] In one embodiment, when fusing thermal radiation features and color features to obtain a fused feature, if the dimensions of the thermal radiation features and the color features are the same, the thermal radiation features and color features can be concatenated to obtain the fused feature. For example, assuming the thermal radiation features are an N*N two-dimensional matrix and the color features are also an N*N two-dimensional matrix, the thermal radiation features of the N*N two-dimensional matrix and the color features of the N*N two-dimensional matrix are concatenated to obtain an N*2N two-dimensional matrix, which is then used as the fused feature.

[0176] Alternatively, in another embodiment, when the dimensions of the thermal radiation feature and the color feature are different, the dimensions of the thermal radiation feature and / or the color feature can be transformed to make the dimensions of the thermal radiation feature and the color feature the same. Then, the thermal radiation feature and the color feature with the same dimensions are spliced ​​together to obtain the fused feature.

[0177] Alternatively, the thermal radiation features and color features can be fused together using the embodiments shown later to obtain fused features.

[0178] In step S106, a fused image of the target object is generated based on the fusion features.

[0179] You can refer to existing image generation methods to generate fused images of target objects based on fusion features, which will not be detailed here. This application does not limit the method of generating images based on image features.

[0180] In this application, an infrared image of the target object is acquired by an infrared sensor, and a visible light image of the target object is acquired by a visible light sensor. Color features of the visible light image are acquired. These color features are used to represent the color information of the target object. High-frequency features of the infrared image are acquired, and these high-frequency features are used to represent the detail and texture information of the target object. Thermal radiation features of the infrared image are acquired based on the high-frequency features. These thermal radiation features are used to represent the detail, texture, and heat distribution information of the target object. The thermal radiation features and color features are fused to obtain a fused feature. A fused image of the target object is generated based on the fused feature.

[0181] This application combines the advantages of infrared images' stable photosensitive characteristics under low-light conditions (noise suppression characteristics) with the high-precision advantages of visible light images in color restoration (color preservation ability). It can generate a fused image that not only retains the detail information of the infrared image but also the color information of the visible light image. It can effectively suppress noise and restore image details under low-light conditions, and restore the true colors of the image. This makes the colors of the fused image realistic and undistorted, which can improve the visibility of the fused image and improve the quality and adaptability of the fused image under low-light conditions. The fused image is clearer and more natural in visual perception.

[0182] Thermal radiation features can provide information on the temperature distribution or heat distribution of a target object, as well as edge and texture information. This can help highlight the details of the target object, especially in low-light environments. Thermal radiation features are advantageous in restoring the edges, textures, and other details of an image, making the target object more obvious and clear.

[0183] As can be seen, this application improves the reconstruction effect of image details in low-light environments by combining the advantages of infrared images and visible light images, solving the problem that existing technologies cannot fully recover details in low-light environments. At the same time, it can reduce noise interference and effectively suppress noise, thereby improving the quality of image reconstruction in low-light environments.

[0184] Furthermore, in another embodiment of this application, at least one of a variety of processing is performed on the fused image of the target object to obtain the processed image of the target object. The various processing includes white balance processing, demosaic processing, and color correction processing, etc.

[0185] Applying white balance processing, demosaic processing, and / or color correction to the fused image of the target object can further optimize the brightness of the fused image, further suppress noise in the fused image, further enhance the richness of details in the fused image, further enhance the realism of colors and brightness in the fused image, further enhance the visual realism and naturalness of the fused image, further enhance the visual quality of the fused image, and further enhance the flexibility and adaptability of image processing, thereby enabling excellent image quality enhancement effects in various devices and application scenarios.

[0186] In one embodiment of this application, a white balance processing can be performed on the image features of the fused image of the target object using a convolutional layer, a fully connected layer, and a white balance matrix.

[0187] The white balance matrix includes:

[0188] By applying white balance processing to the fused image of the target object, color deviations can be corrected, making the colors in the fused image closer to their true colors under natural light. This results in a more natural color appearance in the fused image. When color deviations are corrected, the image is more comfortable to view, and adjusting the color balance of the fused image indirectly improves contrast, making details in both dark and bright areas clearer.

[0189] Demosaicing can be performed on the fused image of the target object using a demosaicing matrix to restore the color information in the visualized image to RGB format, thus restoring a visualized color image. Through demosaicing, the missing color components of each pixel can be estimated, thereby recovering a complete RGB color image. This effectively reduces color artifacts, making the image look more natural and clearer, and better preserving image details.

[0190] Color correction matrices can be used to perform color correction processing on the fused image of the target object. Color correction processing can adjust the overall tone of the image, making it more consistent with human perception of natural scenes. It can improve image contrast, making highlights brighter and shadows darker, enhancing the image's depth; it can make colors more vibrant, making the image more attractive; and it can maintain color consistency, making the image look more natural.

[0191] Furthermore, based on Figure 2The diagram shown illustrates a method for fusing infrared and visible light images. Figure 3 A schematic diagram of another method for fusing infrared and visible light images is shown. Figure 3 In this process, the fused features can be processed sequentially with white balance, depigmentation, and color correction, and a fused image can be generated based on the processed image.

[0192] To fuse visible light and infrared images and obtain fused features, see [link to documentation]. Figure 4 The diagram illustrates a feature fusion architecture of this application. Figure 4 In this architecture, the feature fusion architecture includes a color feature extraction network, a high-frequency feature extraction network, and a thermal radiation feature extraction network.

[0193] The color feature extraction network consists of N cascaded convolutional layers. The kernel size of the N convolutional layers in the color feature extraction network can be decreased sequentially, where N is a positive integer greater than or equal to 2.

[0194] The high-frequency feature extraction network also includes N cascaded convolutional layers, and the kernel size of the N convolutional layers in the high-frequency feature extraction network can be decreased sequentially.

[0195] Thus, see Figure 5 In one embodiment of this application, the acquisition of color features of a visible light image can be achieved through the following process:

[0196] Encode the visible light image to obtain its image features.

[0197] Visible light images can be encoded using CNNs (Convolutional Neural Networks), ResNets (Residual Networks), or VGGs (Visual Geometry Groups) to obtain their image features. These features can be features matrices, etc.

[0198] Then, the color feature extraction network is obtained. The color feature extraction network consists of N cascaded convolutional layers, with the kernel size of the N convolutional layers decreasing sequentially, where N is a positive integer greater than or equal to 2.

[0199] Image features are extracted based on the first convolutional layer in the color feature extraction network to obtain the first convolutional color features.

[0200] The second convolutional layer in the color feature extraction network extracts the color features of the first convolutional layer, thus obtaining the color features of the second convolutional layer.

[0201] ...and so on...

[0202] The color features of the Nth convolutional layer are extracted from the color features of the (N-1)th convolutional layer in the color feature extraction network to obtain the color features of the Nth convolutional layer.

[0203] The color features of the visible light image are obtained based on the color features of the first convolution, the color features of the second convolution, ... and the color features of the Nth convolution.

[0204] For example, the color features of the first convolution, the color features of the second convolution, ... and the color features of the Nth convolution are all determined as color features of the visible light image.

[0205] The color feature extraction network provided in this application includes a convolutional network, which mainly consists of convolutional layers. Convolutional layers are a lightweight network architecture with few parameters, low computational complexity, and high accuracy. Therefore, compared to traditional network architectures, the color feature extraction network used in this application can be considered a lightweight color feature extraction network. Lightweight color feature extraction networks have fewer parameters and lower computational complexity, enabling efficient image denoising and effective recovery of high-frequency details in images, while consuming fewer computational and memory resources. This improves the fusion speed and saves time while enhancing the quality of the fused image. It avoids excessive computational burden, allowing the lightweight color feature extraction network provided in this application to adapt to different hardware resources. This enables the lightweight color feature extraction network to run efficiently even on devices with limited computing power (devices with limited hardware performance) to achieve image fusion processing, solving the problem of excessively high computational resource requirements of traditional network architectures.

[0206] Accordingly, in another embodiment of this application, the acquisition of high-frequency features of an infrared image can be achieved through the following process:

[0207] Wavelet transform can be performed on infrared images to obtain the wavelet-transformed features.

[0208] Wavelet transform can separate high-frequency and low-frequency features in infrared images, thereby extracting high-frequency features more effectively and reducing or avoiding the impact of low-frequency noise.

[0209] Obtain a high-frequency feature extraction network. The high-frequency feature extraction network consists of N cascaded convolutional layers, with the kernel size of the N convolutional layers decreasing sequentially, where N is a positive integer greater than or equal to 2.

[0210] The first convolutional layer in the high-frequency feature extraction network is used to extract features from the wavelet transform features, thus obtaining the first convolutional high-frequency features.

[0211] The high-frequency features of the first convolution are extracted by the second convolutional layer in the high-frequency feature extraction network, thus obtaining the high-frequency features of the second convolution.

[0212] ...and so on...

[0213] Based on the Nth convolutional layer in the high-frequency feature extraction network, the high-frequency features of the Nth convolution are extracted to obtain the high-frequency features of the Nth convolution.

[0214] Based on the high-frequency features of the first convolution, the high-frequency features of the second convolution, ... and the high-frequency features of the Nth convolution, the high-frequency features of the infrared image are obtained. For example, the high-frequency features of the first convolution, the high-frequency features of the second convolution, ... and the high-frequency features of the Nth convolution can all be determined as the high-frequency features of the infrared image.

[0215] The high-frequency feature extraction network provided in this application includes a convolutional network, which mainly consists of convolutional layers. Convolutional layers are a lightweight network architecture with few parameters, low computational complexity, and high accuracy. Therefore, compared to traditional network architectures, the high-frequency feature extraction network used in this application can be considered a lightweight high-frequency feature extraction network. This lightweight network has fewer parameters and lower computational complexity, enabling efficient image denoising and effective recovery of high-frequency details while consuming less computational and memory resources. It improves the fusion speed and saves time while enhancing the quality of the fused image. This avoids excessive computational burden, allowing the lightweight high-frequency feature extraction network provided in this application to adapt to different hardware resources. It enables efficient operation even on devices with limited computing power (devices with limited hardware performance) to achieve image fusion processing, solving the problem of excessive computational resource requirements of traditional network architectures.

[0216] Accordingly, when obtaining the thermal radiation characteristics of an infrared image based on high-frequency features, the following process can be used:

[0217] The first thermal radiation feature of the infrared image is extracted from the first high-frequency feature of the first convolution.

[0218] The second thermal radiation feature of the infrared image is extracted from the high-frequency features of the second convolution.

[0219] ...and so on...

[0220] Extract the Nth thermal radiation feature from the Nth high-frequency feature of the infrared image.

[0221] The thermal radiation characteristics of the infrared image are obtained based on the first thermal radiation characteristic, the second thermal radiation characteristic, ..., and the Nth thermal radiation characteristic. For example, the first thermal radiation characteristic, the second thermal radiation characteristic, ..., and the Nth thermal radiation characteristic are all determined as the thermal radiation characteristics of the infrared image.

[0222] Furthermore, when fusing thermal radiation features and color features to obtain the fused features, the following process can be used:

[0223] The Nth convolutional color feature is concatenated with the Nth thermal radiation feature to obtain the Nth concatenated feature.

[0224] The Nth splicing feature is spliced ​​with the (N-1)th thermal radiation feature to obtain the (N-1)th splicing feature.

[0225] ...and so on...

[0226] The second splicing feature is spliced ​​with the first thermal radiation feature to obtain the first splicing feature.

[0227] The fusion feature is obtained based on the first splicing feature. For example, the first splicing feature can be directly determined as the fusion feature.

[0228] Secondly, see Figure 6 This application provides a schematic diagram of a thermal radiation feature extraction network.

[0229] The thermal radiation feature extraction network includes: a thermal radiation feature extraction block (VSSblock), a multilayer perceptron (MLP), a first addition layer, a convolutional layer, a max pooling layer, an activation function (SigMoid), a first multiplication layer, and a second addition layer.

[0230] The input of the thermal radiation feature extraction network includes the input of the thermal radiation feature extraction block.

[0231] The output of the thermal radiation feature extraction block is connected to the input of the multilayer perceptron, the input of the first additive layer, and the input of the second additive layer, respectively.

[0232] The output of the multilayer perceptron is connected to the input of the first additive layer.

[0233] The output of the first additive layer is connected to the input of the convolutional layer.

[0234] The output of the convolutional layer is connected to the input of the max pooling layer and the input of the first multiplication layer, respectively.

[0235] The output of the max pooling layer is connected to the input of the activation function.

[0236] The output of the activation function is connected to the input of the first multiplication layer.

[0237] The output of the first multiplication layer is connected to the input of the second addition layer.

[0238] The output of the thermal radiation feature extraction network includes the output of the second additive layer.

[0239] The thermal radiation feature extraction block is used to perform preliminary processing on the input high-frequency features to extract features related to thermal radiation. The expressive power of the features can be enhanced through normalization, linear transformation, and convolution operations (see subsequent descriptions).

[0240] Multilayer perceptrons (MLPs) are used to further process the output features of the thermal radiation feature extraction block, enhancing the expressive power of the features through nonlinear transformations. A MLP can include multiple fully connected layers, enabling it to learn complex feature mapping relationships.

[0241] The first additive layer is used to perform an addition operation on the output features of the multilayer perceptron and the output features of the thermal radiation feature extraction block, thereby achieving feature fusion. This helps to preserve the original feature information while introducing the nonlinear transformation of the multilayer perceptron.

[0242] Convolutional layers are used to perform convolution operations on the fused features, extract local features, and enhance the local perception capability of the features.

[0243] Max pooling layers are used to downsample the output features of convolutional layers, extracting the maximum value of the features, reducing the spatial dimensionality of the features, and preserving important information. Max pooling helps reduce computation while retaining the salient information of the features.

[0244] Activation functions are used to enable networks to learn complex feature mapping relationships. The Sigmoid function can map input features to the (0, 1) interval.

[0245] The first multiplication layer is used to multiply the output features of the activation function element-wise with the output features of the convolutional layer, achieving weighted fusion of features. This can enhance the weights of some features while suppressing the weights of others.

[0246] The second addition layer is used to add the output features of the first multiplication layer to the output features of the thermal radiation feature extraction block, further fusing the features and helping to retain the original feature information, while introducing the weighted features of the multiplication layer.

[0247] Furthermore, based on Figure 6 The illustrated embodiment can be found in [reference]. Figure 7This application provides a schematic diagram of a thermal radiation feature extraction block.

[0248] The thermal radiation feature extraction block includes: a first normalization layer (LN), a first linear layer, a depthwise separable convolution layer (DWConv), a two-dimensional selective scan network (SS2D), a second normalization layer, a second linear layer, a second multiplication layer, a third linear layer, and a third addition layer.

[0249] The input of the thermal radiation feature extraction block includes: the input of the first normalization layer.

[0250] The output of the first normalization layer is connected to the input of the first linear layer, the input of the second linear layer, and the input of the third addition layer, respectively.

[0251] The output of the first linear layer is connected to the input of the depth-separable convolutional layer.

[0252] The output of the depth-separable convolutional layer is connected to the input of the two-dimensional selective scanning network.

[0253] The output of the two-dimensional selective scanning network is connected to the input of the second normalization layer.

[0254] The output of the second normalization layer is connected to the input of the second multiplication layer.

[0255] The output of the second multiplication layer is connected to the input of the third linear layer.

[0256] The output of the third linear layer is connected to the input of the third additive layer.

[0257] The output of the thermal radiation feature extraction block includes the output of the third additive layer.

[0258] The first normalization layer is used to normalize the high-frequency features of the input.

[0259] The first linear layer is used to perform a linear transformation on the normalized features, increasing the diversity of features and allowing adjustment of feature scale and offset.

[0260] Depthwise separable convolutional layers are used to process the output features of the first linear layer through depthwise separable convolution operations (decomposing standard convolution into depthwise convolution and pointwise convolution).

[0261] Two-dimensional selective scanning networks are used to perform two-dimensional convolution operations on the features output by depth-separable convolutional layers through selective scanning, capturing features in different directions and enhancing the global perception of features by dynamically selecting the scanning direction.

[0262] The second normalization layer is used to normalize the output of the two-dimensional selective scanning network.

[0263] The second linear layer is used to perform a linear transformation on the normalized features, thereby enhancing the expressive power of the features.

[0264] The second multiplication layer is used to perform element-wise multiplication of the output features of the second normalization layer and the output features of the second linear layer, achieving weighted feature fusion. This can enhance the weights of some features while suppressing the weights of others.

[0265] The third linear layer is used to perform linear transformations on the fused features, further enhancing the expressive power of the features.

[0266] The third addition is used to add the output features of the third linear layer to the output features of the first normalization layer to achieve feature fusion, which helps to preserve the original feature information, while introducing the linear transformation of the third linear layer.

[0267] Thus, when obtaining the thermal radiation features of an infrared image based on high-frequency features, the high-frequency features can be input into the thermal radiation feature extraction network so that the thermal radiation feature extraction network can extract the high-frequency features and obtain the thermal radiation features of the infrared image.

[0268] For example, the high-frequency features of the first convolution can be input into the thermal radiation feature extraction network so that the thermal radiation feature extraction network can extract features from the high-frequency features of the first convolution to obtain the first thermal radiation feature of the infrared image.

[0269] The high-frequency features of the second convolution are input into the thermal radiation feature extraction network so that the thermal radiation feature extraction network can extract features from the high-frequency features of the second convolution to obtain the second thermal radiation feature of the infrared image.

[0270] ...and so on...

[0271] The high-frequency features of the Nth convolution are input into the thermal radiation feature extraction network so that the thermal radiation feature extraction network can extract features from the high-frequency features of the Nth convolution to obtain the Nth thermal radiation feature of the infrared image.

[0272] By deeply integrating the state-space modeling mechanism of a two-dimensional selective scanning network with infrared physical characteristics, an intelligent enhancement scheme with thermodynamic sensing capabilities was constructed.

[0273] By leveraging the selective state-space mechanism of a two-dimensional selective scanning network, a dynamic thermal radiation feature propagation path is established in the spatiotemporal dimension, precisely controlling the feature interaction intensity of different temperature regions. The radiation characteristics of infrared images are transformed into physical parameters in the state space, enabling the network to implicitly learn the diffusion laws in the heat conduction equation, thereby achieving precise enhancement of thermal anomaly regions. The two-dimensional selective scanning network endows the system with the ability to memorize the historical temperature field evolution process, modeling thermal inertia and thermal relaxation phenomena through a time-continuous state transition function, maintaining the thermodynamic equilibrium of the scene while enhancing the instantaneous thermal signal. This state-space-based modeling approach is suitable for handling unsteady heat sources, can adaptively distinguish between environmental thermal noise and target thermal characteristics, and maintains the physical rationality of the enhancement results under complex thermal backgrounds.

[0274] It also introduces a temperature gradient sensing attention mechanism, which works in conjunction with the selective scanning strategy of the two-dimensional selective scanning network to achieve multi-scale feature focusing in the critical temperature region, ensuring the enhanced continuity from micro hot spots to macro thermal fields.

[0275] This application also provides an image processing apparatus for performing the image processing method described in any of the above embodiments, such as... Figure 8 As shown, the device includes:

[0276] The first acquisition module 11 is used to acquire the infrared image of the target object collected by the infrared sensor and the visible light image of the target object collected by the visible light sensor.

[0277] The second acquisition module 12 is used to acquire the color features of the visible light image; the color features are used to reflect the color information of the target object.

[0278] The third acquisition module 13 is used to acquire the high-frequency features of the infrared image, and the high-frequency features are used to reflect the detailed information and texture information of the target object;

[0279] The fourth acquisition module 14 is used to acquire the thermal radiation features of the infrared image based on the high-frequency features; the thermal radiation features are used to reflect the detailed information, texture information and heat distribution information of the target object;

[0280] The fusion module 15 is used to fuse the thermal radiation feature and the color feature to obtain a fused feature;

[0281] The generation module 16 is used to generate a fused image of the target object based on the fusion features.

[0282] In one optional implementation, the second acquisition module includes:

[0283] Encoding unit, used to encode the visible light image to obtain the image features of the visible light image;

[0284] The first acquisition unit is used to acquire a color feature extraction network; the color feature extraction network includes N cascaded convolutional layers, the kernel size of the N convolutional layers decreases sequentially, and N is a positive integer greater than or equal to 2;

[0285] The first extraction unit is used to extract features from the image features based on the first convolutional layer in the color feature extraction network to obtain the first convolutional color feature; to extract features from the first convolutional color feature based on the second convolutional layer in the color feature extraction network to obtain the second convolutional color feature; ... and so on... to extract features from the (N-1)th convolutional color feature based on the Nth convolutional layer in the color feature extraction network to obtain the Nth convolutional color feature;

[0286] The second acquisition unit is used to acquire the color features of the visible light image based on the first convolutional color features, the second convolutional color features, ... and the Nth convolutional color features.

[0287] In one optional implementation, the third acquisition module includes:

[0288] The transformation unit is used to perform wavelet transform processing on the infrared image to obtain the wavelet transformed features;

[0289] The third acquisition unit is used to acquire the high-frequency feature extraction network; the high-frequency feature extraction network includes N cascaded convolutional layers, the kernel size of the N convolutional layers decreases sequentially, and N is a positive integer greater than or equal to 2;

[0290] The second extraction unit is used to extract features from the wavelet-transformed features based on the first convolutional layer in the high-frequency feature extraction network to obtain the first convolutional high-frequency features; to extract features from the first convolutional high-frequency features based on the second convolutional layer in the high-frequency feature extraction network to obtain the second convolutional high-frequency features; ... and so on... to extract features from the (N-1)th convolutional high-frequency features based on the Nth convolutional layer in the high-frequency feature extraction network to obtain the Nth convolutional high-frequency features;

[0291] The fourth acquisition unit is used to acquire the high-frequency features of the infrared image based on the first convolutional high-frequency features, the second convolutional high-frequency features, ... and the Nth convolutional high-frequency features.

[0292] In one optional implementation, the fourth acquisition module includes:

[0293] The third extraction unit is used to extract the first thermal radiation feature of the infrared image from the first convolutional high-frequency feature; extract the second thermal radiation feature of the infrared image from the second convolutional high-frequency feature; ... and so on ... extract the Nth thermal radiation feature of the infrared image from the Nth convolutional high-frequency feature;

[0294] The fifth acquisition unit is used to acquire the thermal radiation features of the infrared image based on the first thermal radiation feature, the second thermal radiation feature, ... and the Nth thermal radiation feature.

[0295] In one optional implementation, the fusion module includes:

[0296] A splicing unit is used to splice the Nth convolutional color feature with the Nth thermal radiation feature to obtain the Nth spliced ​​feature; splice the Nth spliced ​​feature with the (N-1)th thermal radiation feature to obtain the (N-1)th spliced ​​feature; ... and so on... splice the 2nd spliced ​​feature with the 1st thermal radiation feature to obtain the 1st spliced ​​feature;

[0297] The sixth acquisition unit is used to acquire the fusion feature based on the first splicing feature.

[0298] In one optional implementation, the fourth acquisition module includes:

[0299] An input unit is used to input the high-frequency features into a thermal radiation feature extraction network, so that the thermal radiation feature extraction network can extract features from the high-frequency features to obtain the thermal radiation features of the infrared image.

[0300] The thermal radiation feature extraction network includes: a thermal radiation feature extraction block, a multilayer perceptron, a first addition layer, a convolutional layer, a max pooling layer, an activation function, a first multiplication layer, and a second addition layer.

[0301] The input of the thermal radiation feature extraction network includes: the input of the thermal radiation feature extraction block;

[0302] The output of the thermal radiation feature extraction block is connected to the input of the multilayer perceptron, the input of the first addition layer, and the input of the second addition layer, respectively.

[0303] The output of the multilayer perceptron is connected to the input of the first additive layer;

[0304] The output of the first additive layer is connected to the input of the convolutional layer.

[0305] The output of the convolutional layer is connected to the input of the max pooling layer and the input of the first multiplication layer, respectively.

[0306] The output of the max pooling layer is connected to the input of the activation function;

[0307] The output of the activation function is connected to the input of the first multiplication layer;

[0308] The output of the first multiplication layer is connected to the input of the second addition layer;

[0309] The output of the thermal radiation feature extraction network includes the output of the second additive layer;

[0310] The thermal radiation feature extraction block includes: a first normalization layer, a first linear layer, a depthwise separable convolutional layer, a two-dimensional selective scanning network, a second normalization layer, a second linear layer, a second multiplication layer, a third linear layer, and a third addition layer.

[0311] The input of the thermal radiation feature extraction block includes: the input of the first normalization layer;

[0312] The output of the first normalization layer is connected to the input of the first linear layer, the input of the second linear layer, and the input of the third addition layer, respectively.

[0313] The output of the first linear layer is connected to the input of the depth-separable convolutional layer;

[0314] The output of the depth-separable convolutional layer is connected to the input of a two-dimensional selective scanning network;

[0315] The output of the two-dimensional selective scanning network is connected to the input of the second normalization layer;

[0316] The output of the second normalization layer is connected to the input of the second multiplication layer;

[0317] The output of the second multiplication layer is connected to the input of the third linear layer.

[0318] The output of the third linear layer is connected to the input of the third additive layer;

[0319] The output of the thermal radiation feature extraction block includes the output of the third additive layer.

[0320] In an optional implementation, the apparatus further includes:

[0321] The processing module is used to perform at least one of the following processing on the fused image of the target object to obtain the processed image of the target object, wherein the multiple processing includes: white balance processing, demosaic processing and color correction processing.

[0322] In this application, an infrared image of the target object is acquired by an infrared sensor, and a visible light image of the target object is acquired by a visible light sensor. Color features of the visible light image are acquired. These color features are used to represent the color information of the target object. High-frequency features of the infrared image are acquired, and these high-frequency features are used to represent the detail and texture information of the target object. Thermal radiation features of the infrared image are acquired based on the high-frequency features. These thermal radiation features are used to represent the detail, texture, and heat distribution information of the target object. The thermal radiation features and color features are fused to obtain a fused feature. A fused image of the target object is generated based on the fused feature.

[0323] This application combines the advantages of infrared images' stable photosensitive characteristics under low-light conditions (noise suppression characteristics) with the high-precision advantages of visible light images in color restoration (color preservation ability). It can generate a fused image that not only retains the detail information of the infrared image but also the color information of the visible light image. It can effectively suppress noise and restore image details under low-light conditions, and restore the true colors of the image. This makes the colors of the fused image realistic and undistorted, which can improve the visibility of the fused image and improve the quality and adaptability of the fused image under low-light conditions. The fused image is clearer and more natural in visual perception.

[0324] Thermal radiation features can provide information on the temperature distribution or heat distribution of a target object, as well as edge and texture information. This can help highlight the details of the target object, especially in low-light environments. Thermal radiation features are advantageous in restoring the edges, textures, and other details of an image, making the target object more obvious and clear.

[0325] As can be seen, this application improves the reconstruction effect of image details in low-light environments by combining the advantages of infrared images and visible light images, solving the problem that existing technologies cannot fully recover details in low-light environments. At the same time, it can reduce noise interference and effectively suppress noise, thereby improving the quality of image reconstruction in low-light environments.

[0326] The image processing apparatus and the image processing method provided in the above embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0327] This application also provides an electronic device for performing the above image processing method. Please refer to... Figure 9 This illustrates a schematic diagram of an electronic device provided by some embodiments of this application. For example... Figure 9As shown, the electronic device 6 includes: a processor 600, a memory 601, a bus 602, and a communication interface 603. The processor 600, the communication interface 603, and the memory 601 are connected via the bus 602. The memory 601 stores a computer program that can run on the processor 600. When the processor 600 runs the computer program, it executes the image processing method provided in any of the foregoing embodiments of this application.

[0328] The memory 601 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between the virtual devices in the system is achieved through at least one communication interface 603 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0329] Bus 602 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 601 is used to store programs. After receiving an execution instruction, the processor 600 executes the program. The image processing method disclosed in any of the foregoing embodiments of this application can be applied to the processor 600, or implemented by the processor 600.

[0330] The processor 600 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed through integrated logic circuits in the hardware of the processor 600 or through software instructions. The processor 600 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 601. Processor 600 reads the information in memory 601 and, in conjunction with its hardware, completes the steps of the above method.

[0331] The electronic device provided in this application and the image processing method provided in this application are based on the same inventive concept and have the same beneficial effects as the methods they employ, operate, or implement.

[0332] This application also provides a computer-readable storage medium corresponding to the image processing method provided in the foregoing embodiments. Please refer to... Figure 10 The computer-readable storage medium shown is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the image processing method provided in any of the foregoing embodiments.

[0333] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0334] The computer-readable storage medium provided in the above embodiments of this application and the image processing method provided in this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application stored therein.

[0335] Although alternative embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make further changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0336] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of this application should be included within the scope of protection of this application.

Claims

1. An image processing method, characterized by, The method comprises: acquiring an infrared image of a target object collected by an infrared sensor, and acquiring a visible light image of the target object collected by a visible light sensor; acquiring a color feature of the visible light image; the color feature is used to reflect color information of the target object; acquiring a high-frequency feature of the infrared image; the high-frequency feature is used to reflect detail information and texture information of the target object; acquiring a thermal radiation feature of the infrared image according to the high-frequency feature; the thermal radiation feature is used to reflect the detail information, the texture information and heat distribution information of the target object; fusing the thermal radiation feature and the color feature to obtain a fusion feature; generating a fusion image of the target object according to the fusion feature.

2. The method of claim 1, wherein, The method comprises: encoding the visible light image to obtain an image feature of the visible light image; acquiring a color feature extraction network; the color feature extraction network comprises N cascaded convolution layers, convolution kernel sizes of the N convolution layers are reduced in turn, and N is a positive integer greater than or equal to 2; extracting a first convolution color feature from the image feature based on a first convolution layer in the color feature extraction network; extracting a second convolution color feature from the first convolution color feature based on a second convolution layer in the color feature extraction network; ... and so on... extracting an Nth convolution color feature from an (N-1)th convolution color feature based on an Nth convolution layer in the color feature extraction network; acquiring a color feature of the visible light image according to the first convolution color feature, the second convolution color feature,..., and the Nth convolution color feature.

3. The method of claim 2, wherein, The method comprises: performing wavelet transform on the infrared image to obtain a wavelet-transformed feature; acquiring a high-frequency feature extraction network; the high-frequency feature extraction network comprises N cascaded convolution layers, convolution kernel sizes of the N convolution layers are reduced in turn, and N is a positive integer greater than or equal to 2; extracting a first convolution high-frequency feature from the wavelet-transformed feature based on a first convolution layer in the high-frequency feature extraction network; extracting a second convolution high-frequency feature from the first convolution high-frequency feature based on a second convolution layer in the high-frequency feature extraction network; ... and so on... extracting an Nth convolution high-frequency feature from an (N-1)th convolution high-frequency feature based on an Nth convolution layer in the high-frequency feature extraction network; acquiring a high-frequency feature of the infrared image according to the first convolution high-frequency feature, the second convolution high-frequency feature,..., and the Nth convolution high-frequency feature.

4. The method of claim 3, wherein, The method comprises: extracting a first thermal radiation feature of the infrared image in the first convolution high-frequency feature; extracting a second thermal radiation feature of the infrared image in the second convolution high-frequency feature; ... and so on... extracting an Nth thermal radiation feature of the infrared image in an Nth convolution high-frequency feature; obtaining the thermal radiation feature of the infrared image according to the 1st thermal radiation feature, the 2nd thermal radiation feature, and the Nth thermal radiation feature.

5. The method of claim 4, wherein, The fusion of the thermal radiation feature and the color feature obtains a fusion feature, including: splicing the Nth convolution color feature and the Nth thermal radiation feature to obtain an Nth splicing feature; splicing the Nth splicing feature and the N-1th thermal radiation feature to obtain an N-1th splicing feature; ... and so on... splicing the 2nd splicing feature and the 1st thermal radiation feature to obtain a 1st splicing feature; obtaining the fusion feature according to the 1st splicing feature.

6. The method according to claim 1 or 4, characterized in that, The obtaining of the thermal radiation feature of the infrared image according to the high-frequency feature includes: inputting the high-frequency feature into a thermal radiation feature extraction network, so that the thermal radiation feature extraction network extracts the high-frequency feature to obtain the thermal radiation feature of the infrared image; wherein the thermal radiation feature extraction network includes a thermal radiation feature extraction block, a multi-layer perception, a first addition layer, a convolution layer, a maximum pooling layer, an activation function, a first multiplication layer, and a second addition layer; the input end of the thermal radiation feature extraction network includes the input end of the thermal radiation feature extraction block; the output end of the thermal radiation feature extraction block is connected with the input end of the multi-layer perception, the input end of the first addition layer, and the input end of the second addition layer respectively; the output end of the multi-layer perception is connected with the input end of the first addition layer; the output end of the first addition layer is connected with the input end of the convolution layer; the output end of the convolution layer is connected with the input end of the maximum pooling layer and the input end of the first multiplication layer respectively; the output end of the maximum pooling layer is connected with the input end of the activation function; the output end of the activation function is connected with the input end of the first multiplication layer; the output end of the first multiplication layer is connected with the input end of the second addition layer; the output end of the thermal radiation feature extraction network includes the output end of the second addition layer; the thermal radiation feature extraction block includes a first normalization layer, a first linear layer, a depth separable convolution layer, a two-dimensional selective scanning network, a second normalization layer, a second linear layer, a second multiplication layer, a third linear layer, and a third addition layer; the input end of the thermal radiation feature extraction block includes the input end of the first normalization layer; the output end of the first normalization layer is connected with the input end of the first linear layer, the input end of the second linear layer, and the input end of the third addition layer respectively; the output end of the first linear layer is connected with the input end of the depth separable convolution layer; the output end of the depth separable convolution layer is connected with the input end of the two-dimensional selective scanning network; the output end of the two-dimensional selective scanning network is connected with the input end of the second normalization layer; the output end of the second normalization layer is connected with the input end of the second multiplication layer; the output end of the second multiplication layer is connected with the input end of the third linear layer; the output end of the third linear layer is connected with the input end of the third addition layer; the output end of the thermal radiation feature extraction block includes the output end of the third addition layer.

7. The method of claim 1, wherein, The method further includes: The fusion image of the target object is subjected to at least one of a plurality of processes including white balance processing, demosaicing processing, and color correction processing, to obtain a processed image of the target object.

8. An image processing apparatus characterized by comprising: The device comprises: A first acquisition module is configured to acquire an infrared image of a target object collected by an infrared sensor and a visible light image of the target object collected by a visible light sensor. A second acquisition module is configured to acquire a color feature of the visible light image, the color feature being used to represent color information of the target object. A third acquisition module is configured to acquire a high-frequency feature of the infrared image, the high-frequency feature being used to represent detailed information and texture information of the target object. A fourth acquisition module is configured to acquire a thermal radiation feature of the infrared image according to the high-frequency feature, the thermal radiation feature being used to represent detailed information, texture information, and heat distribution information of the target object. A fusion module is configured to fuse the thermal radiation feature and the color feature to obtain a fusion feature. A generation module is configured to generate a fusion image of the target object according to the fusion feature.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the method of any one of claims 1 to 7 when executing the program.

10. A computer-readable storage medium, characterized in that, The computer program is stored on the computer readable storage medium and is executed by the processor to implement the method of any one of claims 1 to 7.

Images