An image processing method, apparatus, electronic device, and storage medium

Abstract

The application provides an image processing method and device, electronic equipment and storage medium, relates to the technical field of image processing, and includes the following steps: acquiring an RGB image and an IR image of a same target scene; calculating a weighting parameter by using edge features of the RGB image and edge features of the IR image, wherein the weighting parameter contains structure consistency information of the RGB image and the IR image; fusing the weighting parameter with an encoding feature map and a decoding feature map of the IR image to obtain a weighted feature map of the IR image; and fusing and restoring the weighted feature map of the IR image with a feature map extracted from the RGB image to output a processed RGB image. The application can solve the problem of inconsistent structures of the IR image and the RGB image and improve the noise reduction effect of the RGB image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus, electronic device and storage medium. Background Technology

[0002] Currently, considering the physical characteristics of near-infrared light, infrared (IR) image sensors can acquire clear IR images even in low-light conditions, and therefore can be used to guide and assist in noise reduction of RGB images.

[0003] However, IR and RGB images have some structural inconsistencies. The most typical are that some text cannot be imaged in IR images, and shadows caused by the infrared (IR) illumination lamp can appear in IR images. If these structural inconsistencies are not addressed, noise reduction of RGB images using IR images will have adverse effects. For example, areas where text is absent in the IR image will interfere with the output, making the text blurry, and shadows present in the IR image will be transferred to the output RGB image.

[0004] Currently, the publicly available network structures for IR-guided RGB denoising tasks cannot handle the structural inconsistencies between IR and RGB; the publicly available IR-guided RGB denoising models produce poor output results in regions where there are structural inconsistencies between IR and RGB; and most of the publicly available IR-guided RGB denoising models are based on guided filtering and optimization algorithms, which have high time complexity and cannot be applied to real-world scenarios. Summary of the Invention

[0005] In view of this, this application provides an image processing method, apparatus, electronic device, and storage medium to solve the technical problem that the structural inconsistency between IR images and RGB images affects the RGB noise reduction effect.

[0006] On one hand, embodiments of this application provide an image processing method, including:

[0007] Acquire RGB and IR images of the same target scene;

[0008] Using the edge features of the RGB image and the edge features of the IR image, a weighting parameter is calculated, and the weighting parameter includes structural consistency information of the RGB image and the IR image;

[0009] The weighting parameters are fused with the encoded feature map and the decoded feature map of the IR image to obtain the weighted feature map of the IR image;

[0010] The weighted feature map of the IR image is fused and restored with the feature map extracted from the RGB image to output the processed RGB image.

[0011] On the other hand, embodiments of this application provide an image processing apparatus, including:

[0012] The acquisition unit is used to acquire RGB and IR images of the same target scene;

[0013] The calculation unit is used to calculate weighting parameters using the edge features of the RGB image and the edge features of the IR image, wherein the weighting parameters include structural consistency information of the RGB image and the IR image;

[0014] A fusion unit is used to fuse the weighting parameters with the encoded feature map and the decoded feature map of the IR image to obtain the weighted feature map of the IR image;

[0015] The processing unit is used to fuse and restore the weighted feature map of the IR image with the feature map extracted from the RGB image, and output the processed RGB image.

[0016] On the other hand, embodiments of this application provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image processing method of embodiments of this application.

[0017] On the other hand, embodiments of this application provide a device-readable storage medium on which a computer program is stored, and the computer program is executed by a processor to implement the image processing method of embodiments of this application.

[0018] In the process of using IR images to guide RGB image denoising, this application can significantly reduce the adverse effects of structural inconsistencies between IR images and RGB images on the RGB image denoising results, thereby improving the RGB image denoising effect. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the noise reduction model according to an embodiment of this application;

[0021] Figure 2This is a schematic diagram of the structure of the fusion network according to an embodiment of this application;

[0022] Figure 3 This is a schematic diagram of the structure of the noise reduction model during the training process of an embodiment of this application;

[0023] Figure 4 This is a flowchart illustrating the training method of the noise reduction model according to an embodiment of this application;

[0024] Figure 5 This is a schematic flowchart illustrating the image processing method according to an embodiment of this application.

[0025] Figure 6 This is a schematic diagram illustrating the specific process of the image processing method according to an embodiment of this application;

[0026] Figure 7 This is a functional structure diagram of the image processing apparatus according to an embodiment of this application;

[0027] Figure 8 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0029] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0030] First, a brief introduction to the design concept of the embodiments of this application will be given.

[0031] There are some structural inconsistencies between RGB and IR images, including: some text cannot be imaged on the IR image; and IR lighting can create shadows on the IR image that are not present in the RGB image. If these structural inconsistencies are not addressed, fusing IR features into RGB features will result in poor image restoration in areas of inconsistency. Specifically, regions without text in the IR image will disrupt the noise reduction output, and shadows from the IR image will be transferred to the output RGB image.

[0032] To address the aforementioned issues, this application proposes a novel neural network-based denoising model. This model can explicitly process the structural inconsistencies between RGB and IR images, avoiding the impact of these inconsistencies on the RGB image denoising output and achieving better denoising results for RGB images.

[0033] In training the denoising model, a noise-free RGB image is required. First, edge detection is performed on both the noise-free RGB and IR images to obtain their edge images, which represent the structures of the RGB and IR images, respectively. Second, to obtain structural information from the noisy RGB image, the edge image calculated on the noise-free RGB image is used as a label, and the noisy RGB image is input to obtain its structural map. Similarly, to obtain structural information from the IR image, the edge image calculated on the noise-free RGB image is used as a label, and the IR image's structural map is obtained.

[0034] In the process of using IR images to guide RGB image denoising, this application can significantly reduce the adverse effects of structural inconsistencies between IR and RGB images on the RGB image denoising results and improve the RGB image denoising effect. In addition, by using a neural network to implement IR image-guided RGB image denoising, the powerful fitting ability of the neural network is fully utilized to recover the details of the RGB image, and a high-quality RGB image after denoising is obtained with low time complexity.

[0035] After introducing the application scenarios and design concepts of the embodiments of this application, the technical solutions provided by the embodiments of this application will be described below.

[0036] like Figure 1 As shown, the embodiments of this application first establish a noise reduction model, including: a first RGB image autoencoder, an RGB feature gradient learner, an IR image autoencoder, an IR feature gradient learner, a fusion network, and a second RGB image autoencoder;

[0037] The first RGB image autoencoder is used to obtain the first encoded feature map of a noisy RGB image; its main structure is a UNet, which contains an encoder and a decoder. The encoder progressively encodes RGB features from a lower-level feature space to a higher-level feature space, while the decoder progressively restores RGB features from a higher-level feature space to a lower-level feature space.

[0038] The RGB feature gradient learner consists of multiple small sub-networks, which learn the feature gradient map of the RGB image from the first encoded feature map of the noisy RGB image, representing the structure of the noisy RGB image.

[0039] An IR image autoencoder is used to acquire encoded and decoded feature maps of IR images. Its main structure is a UNet containing an encoder and a decoder. The encoder progressively encodes IR image features from lower-level feature spaces to higher-level feature spaces, while the decoder progressively recovers IR image features from higher-level feature spaces to lower-level feature spaces.

[0040] The IR feature gradient learner consists of multiple small sub-networks that learn the feature gradient map of the IR image from the encoded feature map of the IR image, representing the structure of the IR image.

[0041] The fusion network is used to calculate weights for the encoded and decoded feature maps of the IR image at corresponding locations based on the structural consistency between the noisy RGB image and the IR image, and to weight the encoded and decoded feature maps of the IR image to output a weighted feature map of the IR image.

[0042] like Figure 2 As shown, the fusion network includes: a first convolutional layer, a second convolutional layer, a third convolutional layer, an adder, and a multiplier.

[0043] The encoded feature map of the IR image is processed by the first convolutional layer to obtain the structure map of the IR image encoded features. In order to enable the first convolutional layer to extract the structure map of IR features better, a gradient convolution operator can be added to the convolutional layer. Therefore, the Sobel operator is used in the last layer of the first convolutional layer.

[0044] The second convolutional layer processes the decoded feature map of the IR image to obtain a structure map of the decoded IR image features. To enable the second convolutional layer to extract the structure map of the IR features better, a gradient convolution operator is added to the second convolutional layer. Therefore, the Sobel operator is used in the last layer of the second convolutional layer.

[0045] The structural feature map of the IR image is obtained by summing the structural map of the IR image's encoded features and the structural map of the IR image's decoded features using an adder.

[0046] The weighted parameters and the structural feature map of the IR image are multiplied by a multiplier to obtain the weighted IR structural feature map. The weighted parameters strengthen structures in the IR structural map that are consistent with RGB, and weaken structures that are inconsistent.

[0047] The weighted IR structure feature map is processed by the third convolutional layer to obtain the weighted feature map of the IR image.

[0048] The second RGB image autoencoder is used to re-encode the noisy RGB image. During the encoding process, a weighted feature map of the IR image is added, and the second encoded feature of the noisy RGB image is output. Then, the second encoded feature of the noisy RGB image is decoded to output the denoised RGB image.

[0049] When training the denoising model, it is necessary to introduce a noiseless RGB image and a noiseless RGB image autoencoder. The noiseless RGB image autoencoder includes an encoder and a decoder. The noiseless RGB image is encoded using the autoencoder to obtain its encoded feature map. The processing procedure of the denoising model after adding the noiseless RGB image autoencoder is as follows: Figure 3 As shown.

[0050] like Figure 4 As shown in the figure, this application provides a method for training a noise reduction model, including the following specific steps:

[0051] Step 101: Establish a training set, which includes multiple groups of image samples;

[0052] The image sample group includes: noisy RGB image samples, IR image samples, and noiseless RGB image samples; all three image samples are images of the same target scene.

[0053] Step 102: Convert the noise-free RGB image to RGB clean The input is fed into the encoder of the noiseless RGB image autoencoder to obtain the encoded feature map of the noiseless RGB image;

[0054] The encoded feature map of the noiseless RGB image includes:

[0055]

[0056] Step 103: Extract edges from the encoded feature map of the noiseless RGB image to obtain an edge image. Binarize the edge image to obtain the feature gradient map of the noiseless RGB image.

[0057] There are three ways to extract edges from the encoded feature map of a noiseless RGB image:

[0058] The first method: Use edge detection operators such as the Sobel operator and the Robert operator to extract edges from the image;

[0059] The second method: edge extraction using phase consistency methods;

[0060] The third method involves encoding the image using a neural network and then extracting edges from its features.

[0061] The binarization process is as follows: binarization to 1 indicates the existence of an edge, and binarization to 0 indicates the absence of an edge.

[0062] The feature gradient map of a noiseless RGB image includes: It is used to indicate whether or not a structure exists in a real RGB image.

[0063] Step 104: Extract noisy RGB image samples using GRB noise The input is fed into the encoder of the first RGB image autoencoder to obtain the first encoded feature map of the noisy RGB image sample;

[0064] The first encoded feature map of the noisy RGB image sample includes: n is the number of encoded feature maps.

[0065] Step 105: Input the first encoded feature map of the noisy RGB image sample into the RGB feature gradient learner, and output the feature gradient map of the noisy RGB image sample; calculate the first loss function value between the feature gradient map of the noisy RGB image sample and the feature gradient map of the noiseless RGB image;

[0066] The feature gradient maps of the noisy RGB image samples include: During the training of the RGB feature gradient learner, a loss function is constructed based on the feature gradient maps of noisy RGB image samples and noiseless RGB image samples. The goal is to learn a feature gradient map of the noisy RGB image that is similar to that of the noiseless RGB image. Figure 1 Therefore, the formula for calculating the first loss function value L1 is:

[0067]

[0068]

[0069] Where, p i Let g represent the i-th pixel of image P. i Let N represent the i-th pixel of image G, where N is the total number of pixels.

[0070] Step 106: Input the IR image samples into the IR image autoencoder, and output the encoded feature map and decoded feature map of the IR image samples; calculate the decoded feature map of the IR image samples and the second loss function value of the IR image samples;

[0071] The encoded feature maps of the IR image samples include: The decoded feature maps of IR image samples include:

[0072] The IR image autoencoder outputs a reconstructed IR image sample (NIR).out The autoencoder is required to encode IR image samples back, so the Charbonnier Loss is used here. The calculated values ​​of the decoded feature map and the second loss function L2 for the IR image samples are:

[0073]

[0074] Among them, IR i This represents the i-th pixel of the IR image sample IR. Indicates IR out The i-th pixel, N is the total number of pixels, and ε is the compensation factor.

[0075] Step 107: Input the encoded feature map of the IR image sample into the IR feature gradient learner, and output the feature gradient map of the IR image sample; calculate the third loss function value between the feature gradient map of the IR image sample and the feature gradient map of the noiseless RGB image;

[0076] In step 107, the reason for not directly using the encoded feature map of the IR image sample to extract the edge is that the IR image has a halo caused by the fill light, which will affect the edge extraction operator.

[0077] The feature gradient maps of the IR image samples include: During the training of the IR feature gradient learner, a loss function is constructed based on the feature gradient maps of IR image samples and noiseless RGB image samples. The goal is to learn a feature gradient map of the IR image samples that is similar to the feature gradient map of the noiseless RGB image samples. Figure 1 Therefore, the third loss function value L3 is:

[0078]

[0079] Step 108: Calculate the sample weighting parameters that include the structural consistency between the noisy RGB image samples and the IR image samples using their feature gradient maps.

[0080]

[0081] The feature gradient map is a binary gradient map, meaning the pixel value is either 0 or 1.

[0082] The function f(x, y) is:

[0083] f(x, y) = λ(1-x)(1-y) + xy

[0084] Where x and y are the independent variables of the function, and λ is the weight.

[0085] When neither the feature gradient map of the noisy RGB image sample nor the feature gradient map of the IR image sample has any edges (both values ​​are 0), the IR image sample information is utilized with a weight of λ in the subsequent denoising process of the noisy RGB image sample. When inconsistent edges exist between the feature gradient maps of the noisy RGB image sample and the IR image sample (one has a gradient, value 1, and the other has no gradient, value 0), the IR image sample information is not utilized in the subsequent denoising process of the noisy RGB image sample. When both the feature gradient maps of the noisy RGB image sample and the feature gradient maps of the IR image sample have edges and the edges are consistent (both are 1), the IR image sample information is utilized in the subsequent denoising process of the noisy RGB image sample. Preferably, λ is set to 1 / 3.

[0086] Step 109: Use a fusion network to fuse the sample weighting parameters with the encoded and decoded feature maps of the IR image samples, and output the weighted feature map of the IR image samples;

[0087] Weighted feature map of IR image samples for:

[0088]

[0089] Here, Fuse() represents the processing procedure of the fusion network.

[0090] This step can suppress the structural inconsistencies between noisy RGB image samples and IR image samples in the gradient feature map of IR image samples, and strengthen the consistent edges between noisy RGB image samples and IR image samples.

[0091] Step 110: Use the encoder of the second RGB image autoencoder to process the noisy RGB image samples. noise The encoding is then performed again, and a weighted feature map of the IR image samples is added during the encoding process to obtain a second encoded feature map of the noisy RGB image samples that fuses the IR image sample information.

[0092] The encoder of the second RGB image autoencoder includes: Then we have:

[0093]

[0094]

[0095] …

[0096]

[0097] The second encoded feature map of the noisy RGB image sample includes:

[0098] Step 111: Input the second encoded feature map of the noisy RGB image sample into the decoder of the second RGB autoencoder, and output the denoised RGB image sample. out ; Calculate the RGB values ​​of the denoised RGB image samples out and noiseless RGB image samples RGB clean The fourth loss function value;

[0099] Among them, the denoised RGB image samples RGB out and noiseless RGB image samples RGB clean The fourth loss function value L4 is:

[0100]

[0101] in, Represents RGB clean The i-th pixel, Represents RGB out The i-th pixel, where N is the total number of pixels.

[0102] Step 112: Calculate the total loss function value and update the parameters of the denoising model based on the total loss function value;

[0103] The total loss function value L is:

[0104] L = L1 + L2 + L3 + L4

[0105] During training, the parameters trained include: a noiseless RGB image autoencoder, a first RGB image autoencoder, an RGB feature gradient learner, an IR image autoencoder, an IR feature gradient learner, three convolutional layers of the fusion network, and various parameters from the second RGB image autoencoder. When the denoising model is used in image processing methods, only the first RGB image autoencoder, RGB feature gradient learner, IR image autoencoder, IR feature gradient learner, fusion network, and second RGB image autoencoder need to be used.

[0106] In practice, the total loss function value can measure the RGB values ​​of the denoised RGB image samples. out and noiseless RGB image samples RGB clean The smaller the total loss function value, the better the denoising effect of the denoising model, reflecting the degree of difference between the images. Based on this, in the embodiments of this application, the denoised RGB image samples corresponding to each image sample group can be calculated. out and noiseless RGB image samples RGB cleanThe total loss function value is calculated, and it is determined whether the total loss function value meets the preset requirements, such as being less than a preset threshold. If the total loss function value does not meet the preset requirements, the parameters of the denoising model are updated, and steps 102 to 111 are repeated until the total loss function value meets the preset requirements, thereby obtaining the final denoising model. In practical applications, the preset threshold can be flexibly set according to actual needs, and no specific limitation is made here.

[0107] In addition, a portion of the training set can be used as the test and validation set for the model to test and validate the model's parameters. If the validation results are unsatisfactory, a new set of images from the training set needs to be selected and the model retrained.

[0108] like Figure 5 As shown, the general flow of the image processing method provided in this application embodiment is as follows:

[0109] Step 201: Acquire RGB and IR images of the same target scene;

[0110] Step 202: Calculate weighting parameters using the edge features of the RGB image and the edge features of the IR image. The weighting parameters include structural consistency information of the RGB image and the IR image.

[0111] Step 203: Fuse the weighting parameters with the encoded feature map and decoded feature map of the IR image to obtain the weighted feature map of the IR image;

[0112] Step 204: Fuse and restore the weighted feature map of the IR image with the feature map extracted from the RGB image, and output the processed RGB image.

[0113] Specifically, when using the denoising model trained in the embodiments of this application, the image processing method provided in the embodiments of this application will be further described in detail, see [link to relevant documentation]. Figure 6 As shown, the specific flow of the image processing method provided in this application embodiment is as follows:

[0114] Step 301: Acquire RGB and IR images of the same target scene;

[0115] One approach is to simultaneously capture images of the same target scene using both an RGB camera and an IR camera, resulting in RGB and IR images. The RGB image is noisy and requires noise reduction processing using the IR image.

[0116] Step 302: Encode the RGB image using the encoder of the first RGB image autoencoder to obtain the first encoded feature map of the RGB image;

[0117] Step 303: Perform gradient learning on the first encoded feature map of the RGB image using the RGB feature gradient learner to obtain the feature gradient map of the RGB image;

[0118] Step 304: Encode and decode the IR image using an IR image autoencoder to obtain the encoded feature map and decoded feature map of the IR image;

[0119] Step 305: Perform gradient learning on the encoded feature map of the IR image using the IR feature gradient learner to obtain the feature gradient map of the IR image;

[0120] Step 306: Calculate the weighting parameters using the feature gradient maps of the RGB image and the IR image;

[0121] Step 307: Use a fusion network to fuse the weighting parameters with the encoded and decoded feature maps of the IR image, and output the weighted feature map of the IR image;

[0122] Step 308: Re-encode the RGB image using the encoder of the second RGB image autoencoder, and add the weighted feature map of the IR image during the encoding process to obtain the second encoded feature map of the RGB image that fuses the IR image information;

[0123] Step 309: Decode the second encoded feature map of the RGB image using the decoder of the second RGB image autoencoder to obtain the processed RGB image.

[0124] Based on the above embodiments, this application provides an image processing apparatus, see below. Figure 7 As shown, an image processing apparatus 400 provided in this application embodiment includes at least:

[0125] Acquisition unit 401 is used to acquire RGB and IR images of the same target scene;

[0126] The calculation unit 402 is used to calculate a weighting parameter using the edge features of the RGB image and the edge features of the IR image, wherein the weighting parameter includes structural consistency information of the RGB image and the IR image;

[0127] The fusion unit 403 is used to fuse the weighting parameters with the encoded feature map and the decoded feature map of the IR image to obtain the weighted feature map of the IR image;

[0128] The denoising unit 404 is used to fuse and restore the weighted feature map of the IR image with the feature map extracted from the RGB image, and output the processed RGB image.

[0129] It should be noted that since the principle of the image processing apparatus 400 provided in this application embodiment to solve the technical problem is similar to the image processing method provided in this application embodiment, the implementation of the image processing apparatus 400 provided in this application embodiment can refer to the implementation of the image processing provided in this application embodiment, and the repeated parts will not be described again.

[0130] Reference Figure 8 An example electronic device 500 for implementing the image processing method of embodiments of the present invention is described below.

[0131] like Figure 8 As shown, the electronic device 500 includes one or more processing devices 501, one or more storage devices 502, input devices 503, output devices 504, and image acquisition devices 505. These components are interconnected via a bus system 506 and / or other forms of connection mechanisms (not shown). It should be noted that... Figure 8 The components and structure of the electronic device 500 shown are merely exemplary and not limiting; the electronic device may have other components and structures as needed.

[0132] The processing device 501 may be a central processing unit (CPU), a graphics processing unit (GPU), or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 500 to perform desired functions.

[0133] Storage device 502 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and processing device 501 may execute the program instructions to implement the client functions (implemented by the processor) in the embodiments of the present invention described below, and / or other desired functions. Various applications and various data, such as various data used and / or generated by the applications, may also be stored in the computer-readable storage medium.

[0134] The input device 503 may be a device used by a user to input commands, and may include one or more of a keyboard, mouse, microphone, and touch screen.

[0135] The output device 504 can output various information (e.g., image samples or sound) to the outside (e.g., a user) and may include one or more of a display, a speaker, etc.

[0136] The image acquisition device 505 can capture images desired by the user. In this embodiment, the image acquisition device 505 includes an RGB camera and a near-infrared camera; the captured RGB and IR images are stored in the storage device 502 for use by other components.

[0137] For example, an example electronic device for implementing the image processing method according to the embodiments of this application can be implemented on a mobile terminal such as a smartphone or tablet computer.

[0138] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the image processing method provided in the foregoing method embodiments.

[0139] The computer program product of the image processing method and apparatus provided in this application includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0140] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0141] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interface; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0142] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0143] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0144] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. An image processing method, characterized in that, include: Acquire RGB and IR images of the same target scene; the RGB image is noisy. Using the edge features of the RGB image and the edge features of the IR image, a weighting parameter is calculated, and the weighting parameter includes structural consistency information of the RGB image and the IR image; The weighting parameters are fused with the encoded feature map and the decoded feature map of the IR image through a fusion network to obtain the weighted feature map of the IR image; the fusion network includes: a first convolutional layer, a second convolutional layer, a third convolutional layer, an adder, and a multiplier; The weighted feature map of the IR image is fused and restored with the feature map extracted from the RGB image to output the processed RGB image; Specifically, the weighting parameters are fused with the encoded and decoded feature maps of the IR image through a fusion network to obtain the weighted feature map of the IR image, including: The coded feature map of the IR image is processed by the first convolutional layer to obtain a structure map of the coded features of the IR image; The second convolutional layer processes the decoding feature map of the IR image to obtain a structure map of the decoding features of the IR image. The structure map of the IR image is obtained by summing the structure map of the encoded features of the IR image and the structure map of the decoded features of the IR image using the adder. The weighting parameters and the structural feature map of the IR image are multiplied by the multiplier to obtain the weighted structural feature map of the IR image. The weighted structural feature map is processed by the third convolutional layer to obtain the weighted feature map of the IR image.

2. The image processing method according to claim 1, characterized in that, Using the edge features of the RGB image and the edge features of the IR image, weighting parameters are calculated, including: The RGB image is encoded by a first autoencoder to obtain a first encoded feature map of the RGB image; The first feature gradient learner performs gradient learning on the first encoded feature map of the RGB image to obtain the feature gradient map of the RGB image. The IR image is encoded and decoded using a second autoencoder to obtain the encoded feature map and the decoded feature map of the IR image; The coded feature map of the IR image is obtained by performing gradient learning on the IR image through a second feature gradient learner. The weighting parameters are calculated using the feature gradient maps of the RGB image and the IR image.

3. The image processing method according to claim 2, characterized in that, The calculation of weighting parameters using the feature gradient maps of the RGB image and the IR image includes: The weighting parameters are calculated based on the following formula. : in, This is the feature gradient map of the RGB image. The feature gradient map of the IR image; function for: Where x and y are the independent variables of the function, For parameters.

4. The image processing method according to claim 2, characterized in that, The weighted feature map of the IR image is fused and restored with the feature map extracted from the RGB image to output the processed RGB image, including: The RGB image is re-encoded by the encoder of the third autoencoder, and the weighted feature map of the IR image is added during the encoding process to obtain the second encoded feature map of the RGB image; The second encoded feature map of the RGB image is decoded to obtain the processed RGB image.

5. The image processing method according to claim 4, characterized in that, The training process of the method includes: Obtain training samples, which include multiple image sample groups. The image sample groups include: noisy RGB image samples, IR image samples, and noiseless RGB image samples; the noiseless RGB image samples are noiseless RGB image samples. The noiseless RGB image samples are encoded by the encoder of the fourth autoencoder to obtain the encoded feature map of the noiseless RGB image samples. Edge extraction is performed on the encoded feature map of the noiseless RGB image samples to obtain the edge image. The edge image is binarized to obtain the feature gradient map of the noiseless RGB image samples. The noisy RGB image sample is encoded by the encoder of the first autoencoder to obtain the first encoded feature map of the noisy RGB image sample; the first feature gradient learner is used to learn the gradient of the first encoded feature map of the noisy RGB image sample to obtain the feature gradient map of the noisy RGB image sample; the first loss function value between the feature gradient map of the noisy RGB image sample and the feature gradient map of the noiseless RGB image sample is calculated. The IR image samples are encoded and decoded by the encoder of the second autoencoder to obtain the encoded feature map and decoded feature map of the IR image samples; the encoded feature map of the IR image samples is learned by the second feature gradient learner to obtain the feature gradient map of the IR image samples; the second loss function between the decoded feature map of the IR image samples and the IR image samples is calculated; the third loss function value between the feature gradient map of the IR image samples and the feature gradient map of the noiseless RGB image samples is calculated. The sample weighting parameters are calculated using the feature gradient maps of noisy RGB image samples and IR image samples. The fusion network is used to fuse the sample weighting parameters with the encoded and decoded feature maps of the IR image samples to obtain the weighted feature map of the IR image samples. The noisy RGB image samples are re-encoded by the encoder of the third autoencoder, and the weighted feature map of the IR image samples is added during the encoding process to obtain the second encoded feature map of the noisy RGB image samples. The second encoded feature map of the noisy RGB image sample is decoded by the decoder of the third autoencoder to obtain the processed RGB image sample. The fourth loss function value between the processed RGB image sample and the noiseless RGB image sample is calculated. The parameters of the first autoencoder, the second autoencoder, the third autoencoder, the first autoencoder, the first feature gradient learner, the second feature gradient learner, and the fusion network are updated based on the total loss function value, where the total loss function value is the sum of the first loss function value, the second loss function value, the third loss function value, and the fourth loss function value.

6. The image processing method according to claim 5, characterized in that, The formula for calculating the value of the first loss function is: Where L1 is the value of the first loss function. This is the gradient map of the j-th feature of a noisy RGB image sample; Let be the j-th feature gradient map of a noiseless RGB image sample, and n be the number of feature gradient maps; The formula for calculating the value of the third loss function is: Where L3 is the value of the third loss function. This is the gradient map of the j-th feature of the IR image sample; in, This represents the i-th pixel of image P. Let represent the i-th pixel of image G.

7. An image processing apparatus, characterized in that, include: An acquisition unit is used to acquire RGB and IR images of the same target scene; the RGB image is noisy. The calculation unit is used to calculate weighting parameters using the edge features of the RGB image and the edge features of the IR image, wherein the weighting parameters include structural consistency information of the RGB image and the IR image; A fusion unit is used to fuse the weighting parameters with the encoded feature map and decoded feature map of the IR image through a fusion network to obtain a weighted feature map of the IR image; the fusion network includes: a first convolutional layer, a second convolutional layer, a third convolutional layer, an adder, and a multiplier; The processing unit is used to fuse and restore the weighted feature map of the IR image with the feature map extracted from the RGB image, and output the processed RGB image; The fusion unit is configured to: process the encoded feature map of the IR image through the first convolutional layer to obtain a structure map of the encoded features of the IR image; process the decoded feature map of the IR image through the second convolutional layer to obtain a structure map of the decoded features of the IR image; sum the structure map of the encoded features of the IR image and the structure map of the decoded features of the IR image through the adder to obtain a structure map of the IR image; multiply the weighting parameters and the structure map of the IR image through the multiplier to obtain a weighted structure map of the IR image; and process the weighted structure map through the third convolutional layer to obtain a weighted feature map of the IR image.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the image processing method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the image processing method as described in any one of claims 1-6.

Images