Optimized training method for low-light image restoration models

Abstract

The present application proposes an optimization training method for a low-light image restoration model, including: obtaining a first training set for the current image restoration task, each group of sample data in the first training set includes a first training image and a second training image; performing counterfactual enhancement processing on the first training image of each group of sample data in the first training set to obtain a third training image; replacing the first training image with the third training image to obtain a second training set; using the third training image in the second training set as the input image and the second training image in the second training set as the control image of the output image, optimizing and training the initial low-light image restoration model until convergence, and performing frequency domain regularization processing on the intermediate image obtained by processing the residual block inside the encoder network of at least one scale in the model during each optimization training process. This technical solution improves the image restoration accuracy of the low-light image restoration model.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an optimization training method for a low-light image restoration model. Background Art

[0002] Currently, in many research fields such as image restoration, security monitoring, image intelligence analysis, and medical image analysis, images captured under unbalanced lighting conditions are a major challenge to human perception. It can be said that unbalanced lighting conditions interfere with downstream visual tasks in these fields.

[0003] In this regard, related technologies propose to restore the captured images through neural network models. However, existing neural network models for image restoration can only process images under specified lighting conditions, have poor adaptability to tasks with different lighting conditions, and only operate in the spatial domain, ignoring frequency domain information, which results in poor restored images.

[0004] Therefore, how to improve the ability of neural network models to restore images under lighting conditions has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The embodiment of the present application provides an optimization training method for a low-light image restoration model, aiming to solve the technical problem in the related art that the neural network model has insufficient ability to restore images under lighting conditions, resulting in poor image restoration effect.

[0006] In a first aspect, an embodiment of the present application provides an optimization training method for a low-light image restoration model, including:

[0007] Acquire a first training set for the current image restoration task, wherein each set of sample data in the first training set includes a first training image and a second training image, the first training image is used to describe the imaging content directly acquired by the image acquisition device, and the second training image is used to describe the actual situation of the imaging content in the actual scene;

[0008] Performing counterfactual enhancement processing on the first training image of each group of sample data in the first training set to obtain a third training image corresponding to the first training image;

[0009] replacing the first training image corresponding to the first training image in the first training set with the third training image to obtain a second training set;

[0010] The third training image in the second training set is used as an input image, and the second training image in the second training set is used as a control image of the output image, and the initial low-light image restoration model is optimized and trained until convergence to obtain a target low-light image restoration model suitable for the current image restoration task, wherein:

[0011] The initial low-light image restoration model is used to reflect the causal relationship between the input image, the lighting condition of the input image, the ideal lighting specific degradation kernel function and the output image,

[0012] In each optimization training process, for a residual block inside an encoder network of at least one scale in the model, a frequency domain regularization process is performed on an intermediate image obtained by processing the residual block.

[0013] In one embodiment of the present application, optionally, performing counterfactual enhancement processing on the first training image of each group of sample data in the first training set includes:

[0014] Decomposing the first training image into an amplitude component and a phase component by a Fourier transform function;

[0015] The amplitude components of the first training image are adjusted based on a plurality of first hyperparameters to obtain first enhanced images corresponding to the plurality of first hyperparameters, wherein:

[0016] ,

[0017] is the amplitude component of the first training image, is the first hyperparameter, for through the first enhanced image corresponding to the adjustment;

[0018] For each of the first enhanced images, based on the amplitude component and the phase component corresponding to the first enhanced image, a first real part and a first imaginary part corresponding to the first enhanced image are determined, wherein:

[0019] , ,

[0020] is the amplitude component corresponding to the first enhanced image, is the phase component corresponding to the first enhanced image, and are the first real part and the first imaginary part corresponding to the first enhanced image respectively;

[0021] The first real part and the first imaginary part are converted into the spatial domain for fusion to obtain a second enhanced image, wherein:

[0022] ,

[0023] is an imaginary unit, is the inverse function, is the second enhanced image;

[0024] The second enhanced images corresponding to all the first enhanced images are fused to obtain the third training image corresponding to the first training image.

[0025] In one embodiment of the present application, optionally, performing counterfactual enhancement processing on the first training image of each group of sample data in the first training set includes:

[0026] Calling a counterfactual enhancement processing plug-in to perform counterfactual enhancement processing on the first training image of each group of sample data in the first training set; or

[0027] In response to the acquisition of the first training set, a first hook function is called, wherein the first hook function is used to perform counterfactual enhancement processing on the first training image of each group of sample data in the first training set.

[0028] In one embodiment of the present application, optionally, performing frequency domain regularization processing on the intermediate image obtained by processing the residual block includes:

[0029] Performing a regularization transformation on the intermediate image obtained by processing the residual block based on a preset regularization function to obtain a fourth training image;

[0030] Decomposing the intermediate image and the fourth training image into an amplitude component and a phase component respectively by a Fourier transform function;

[0031] fusing the amplitude component of the intermediate image with the amplitude component of the fourth training image in the frequency domain to obtain an amplitude fusion component, and fusing the phase component of the intermediate image with the phase component of the fourth training image in the frequency domain to obtain a phase fusion component;

[0032] Based on the amplitude fusion component and the phase fusion component, a second real part and a second imaginary part are determined, wherein:

[0033] , ,

[0034] is the amplitude fusion component, is the phase fusion component, and are the second real part and the second imaginary part respectively;

[0035] The second real part and the second imaginary part are converted into the spatial domain for fusion to obtain a fifth training image, wherein:

[0036] ,

[0037] is an imaginary unit, is the inverse function is the fifth training image;

[0038] The intermediate image is replaced by the fifth training image, and the current optimization training is continued.

[0039] In one embodiment of the present application, optionally, fusing the amplitude component of the intermediate image and the amplitude component of the fourth training image in the frequency domain includes:

[0040] Performing a splicing operation on the amplitude component of the intermediate image and the amplitude component of the fourth training image to obtain a first splicing component;

[0041] Calling a convolution function to perform convolution processing on the first spliced ​​component;

[0042] The fusing the phase component of the intermediate image and the phase component of the fourth training image in the frequency domain comprises:

[0043] Performing a splicing operation on the phase component of the intermediate image and the phase component of the fourth training image to obtain a second splicing component;

[0044] A convolution function is called to perform convolution processing on the second spliced ​​component.

[0045] In one embodiment of the present application, optionally, performing frequency domain regularization processing on the intermediate image obtained by processing the residual block includes:

[0046] Calling a frequency domain regularization processing plug-in to perform frequency domain regularization processing on the intermediate image obtained by processing the residual block; or

[0047] In response to the acquisition of the intermediate image, a second hook function is called, wherein the second hook function is used to perform frequency domain regularization processing on the intermediate image obtained by processing the residual block.

[0048] In one embodiment of the present application, optionally, taking the third training image in the second training set as an input image and taking the second training image in the second training set as a reference image of an output image, and optimizing and training the initial low-light image restoration model until convergence, includes:

[0049] In each optimization training process, the third training image in the second training set is used as an input image, and a sixth training image is determined as an output image through the initial low-light image restoration model;

[0050] Based on the sixth training image and the second training image, the spatial domain loss and the frequency domain loss of the current sub-optimization training are determined, wherein:

[0051] ,

[0052] ,

[0053] and are respectively the spatial domain loss and the frequency domain loss of the current optimization training, is the number of the third training images, is the sixth training image, is the second training image;

[0054] Based on the spatial domain loss and the frequency domain loss of the current sub-optimization training, a dual-domain loss of the current sub-optimization training is determined, wherein:

[0055] ,

[0056] is the dual-domain loss of the current sub-optimization training, is the second hyperparameter;

[0057] If the dual-domain loss is within a preset convergence range, the optimization training process is terminated;

[0058] If the dual-domain loss is not within the preset convergence range, return to the step of determining the sixth training image as the output image through the initial low-light image restoration model and enter the next optimization training.

[0059] In one embodiment of the present application, optionally, the method further includes:

[0060] Acquire a test set of the current image restoration task, wherein each group of test data of the test set includes a first test image and a second test image, the first test image is used to describe the imaging content directly acquired by the image acquisition device, and the second test image is used to describe the real situation of the imaging content in the actual scene;

[0061] Performing counterfactual enhancement processing on the first test image to obtain a third test image;

[0062] Determining a fourth test image based on the third test image and the target low-light image restoration model;

[0063] Obtaining a similarity between the second test image and the fourth test image;

[0064] Based on the similarity, the image restoration accuracy of the target low-light image restoration model is determined.

[0065] In a second aspect, an embodiment of the present application provides an optimization training device for a low-light image restoration model, comprising:

[0066] A first training set acquisition unit, used to acquire a first training set of a current image restoration task, wherein each group of sample data of the first training set includes a first training image and a second training image, the first training image is used to describe the imaging content directly acquired by the image acquisition device, and the second training image is used to describe the real situation of the imaging content in an actual scene;

[0067] a counterfactual enhancement unit, configured to perform counterfactual enhancement processing on a first training image of each group of sample data in the first training set to obtain a third training image corresponding to the first training image;

[0068] A second training set acquisition unit, configured to replace the first training image corresponding to the first training image in the first training set with the third training image to obtain a second training set;

[0069] a model optimization training unit, configured to use the third training image in the second training set as an input image and the second training image in the second training set as a control image of an output image, optimize and train an initial low-light image restoration model until convergence, and obtain a target low-light image restoration model suitable for the current image restoration task, wherein the initial low-light image restoration model is used to reflect the causal relationship between the input image, the illumination condition of the input image, the ideal illumination specific degradation kernel function, and the output image;

[0070] The frequency domain regularization processing unit is used to perform frequency domain regularization processing on the intermediate image obtained by processing the residual block inside the encoder network of at least one scale in the model during each optimization training process of the model optimization training unit.

[0071] In one embodiment of the present application, optionally, the counterfactual enhancement unit is used to: decompose the first training image into an amplitude component and a phase component through a Fourier transform function; adjust the amplitude component of the first training image based on multiple first hyperparameters to obtain first enhanced images corresponding to each of the multiple first hyperparameters, wherein:

[0072] ,

[0073] is the amplitude component of the first training image, is the first hyperparameter, for through the first enhanced image corresponding to the adjustment; for each of the first enhanced images, determining the first real part and the first imaginary part corresponding to the first enhanced image based on the amplitude component and the phase component corresponding to the first enhanced image, wherein,

[0074] , ,

[0075] is the amplitude component corresponding to the first enhanced image, is the phase component corresponding to the first enhanced image, and are the first real part and the first imaginary part corresponding to the first enhanced image respectively; convert the first real part and the first imaginary part into the spatial domain for fusion to obtain a second enhanced image, wherein,

[0076] ,

[0077] is an imaginary unit, is the inverse function, The second enhanced image is the second enhanced image; and the second enhanced images corresponding to all the first enhanced images are fused to obtain the third training image corresponding to the first training image.

[0078] In one embodiment of the present application, optionally, the counterfactual enhancement unit includes:

[0079] The first execution unit is used to call the counterfactual enhancement processing plug-in to perform counterfactual enhancement processing on the first training image of each group of sample data in the first training set.

[0080] In one embodiment of the present application, optionally, the counterfactual enhancement unit includes:

[0081] The second execution unit is used to call a first hook function in response to obtaining the first training set, wherein the first hook function is used to perform counterfactual enhancement processing on the first training image of each group of sample data in the first training set.

[0082] In one embodiment of the present application, optionally, the frequency domain regularization processing unit is used to: perform a regularization transformation on the intermediate image obtained by processing the residual block based on a preset regularization function to obtain a fourth training image; decompose the intermediate image and the fourth training image into an amplitude component and a phase component respectively through a Fourier transform function; fuse the amplitude component of the intermediate image with the amplitude component of the fourth training image in the frequency domain to obtain an amplitude fusion component, and fuse the phase component of the intermediate image with the phase component of the fourth training image in the frequency domain to obtain a phase fusion component; determine the second real part and the second imaginary part based on the amplitude fusion component and the phase fusion component, wherein,

[0083] , ,

[0084] is the amplitude fusion component, is the phase fusion component, and are the second real part and the second imaginary part respectively; convert the second real part and the second imaginary part into the spatial domain for fusion to obtain a fifth training image, wherein,

[0085] ,

[0086] is an imaginary unit, is the inverse function, The fifth training image is used to replace the intermediate image, and the current optimization training is continued.

[0087] In one embodiment of the present application, optionally, the frequency domain regularization processing unit is used to: perform a splicing operation on the amplitude component of the intermediate image and the amplitude component of the fourth training image to obtain a first splicing component; call a convolution function to perform convolution processing on the first splicing component; and perform a splicing operation on the phase component of the intermediate image and the phase component of the fourth training image to obtain a second splicing component; call a convolution function to perform convolution processing on the second splicing component.

[0088] In one embodiment of the present application, optionally, the frequency domain regularization processing unit includes:

[0089] The third execution unit is used to call the frequency domain regularization processing plug-in to perform frequency domain regularization processing on the intermediate image obtained by processing the residual block.

[0090] In one embodiment of the present application, optionally, the frequency domain regularization processing unit includes:

[0091] The fourth execution unit is used to call the second hook function in response to the acquisition of the intermediate image, wherein the second hook function is used to perform frequency domain regularization processing on the intermediate image obtained by processing the residual block.

[0092] In one embodiment of the present application, optionally, the model optimization training unit is used to: in each optimization training process, use the third training image in the second training set as the input image, determine the sixth training image as the output image through the initial low-light image restoration model; based on the sixth training image and the second training image, determine the spatial domain loss and frequency domain loss of the current optimization training, wherein,

[0093] ,

[0094] ,

[0095] and are respectively the spatial domain loss and the frequency domain loss of the current optimization training, is the number of the third training images, is the sixth training image, is the second training image; based on the spatial domain loss and the frequency domain loss of the current sub-optimization training, determining the dual-domain loss of the current sub-optimization training, wherein,

[0096] ,

[0097] is the dual-domain loss of the current sub-optimization training, is the second hyperparameter; if the dual-domain loss is within the preset convergence range, the optimization training process is terminated; if the dual-domain loss is not within the preset convergence range, the process returns to the step of determining the sixth training image as the output image through the initial low-light image restoration model and enters the next optimization training.

[0098] In one embodiment of the present application, optionally, the device further includes:

[0099] A model testing unit is used to obtain a test set for a current image restoration task, wherein each group of test data in the test set includes a first test image and a second test image, the first test image is used to describe the imaging content directly acquired by the image acquisition device, and the second test image is used to describe the actual situation of the imaging content in the actual scene; the first test image is subjected to counterfactual enhancement processing to obtain a third test image; based on the third test image and the target low-light image restoration model, a fourth test image is determined; the similarity between the second test image and the fourth test image is obtained; based on the similarity, the image restoration accuracy of the target low-light image restoration model is determined.

[0100] In a third aspect, an embodiment of the present application provides an image processing method, using the target low-light image restoration model described in any one of the first aspects to perform light-adaptive image restoration processing for a current image restoration task, the method comprising:

[0101] Acquire a first target image of a current image restoration task, wherein the first target image is used to describe imaging content directly acquired by an image acquisition device;

[0102] Performing counterfactual enhancement processing on the first target image to obtain a second target image corresponding to the first target image;

[0103] The second target image is used as an input image of a target low-light image restoration model, and a third target image corresponding to the second target image is output through the target low-light image restoration model, where the third target image is used to describe the actual situation of the imaging content of the first target image in the actual scene, wherein:

[0104] During the process of the target low-light image restoration model performing light adaptation image restoration processing on the second target image, for the residual blocks inside the encoder network of at least one scale in the model, frequency domain regularization processing is performed on the intermediate image obtained by processing the residual blocks.

[0105] In a fourth aspect, an embodiment of the present application provides an image processing device, which uses the target low-light image restoration model described in any one of the first aspects to perform light-adaptive image restoration processing for a current image restoration task, and the device includes:

[0106] A target image acquisition unit, used to acquire a first target image of a current image restoration task, wherein the first target image is used to describe the imaging content directly acquired by the image acquisition device;

[0107] a counterfactual enhancement unit, configured to perform counterfactual enhancement processing on the first target image to obtain a second target image corresponding to the first target image;

[0108] a model processing unit, configured to use the second target image as an input image of a target low-light image restoration model, and output a third target image corresponding to the second target image through the target low-light image restoration model, wherein the third target image is used to describe a real situation of the imaging content of the first target image in an actual scene;

[0109] A frequency domain regularization processing unit is used to perform frequency domain regularization processing on an intermediate image obtained by processing a residual block inside an encoder network of at least one scale in the model during the process in which the model processing unit performs light adaptation image restoration processing on the second target image.

[0110] In a fifth aspect, an embodiment of the present application provides a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are configured to execute the method described in any one of the first and third aspects above.

[0111] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the method described in any one of the first and third aspects above.

[0112] The above technical solution aims to solve the technical problem in the related art that the neural network model has insufficient ability to restore images under lighting conditions, resulting in poor image restoration results. For a specific image restoration task, the captured images of the training set exclusive to the image restoration task are subjected to counterfactual enhancement processing, and then used as model input for model optimization training. At the same time, during each optimization training process, the intermediate image obtained by processing the residual block inside the encoder network of at least one scale in the model is subjected to frequency domain regularization processing. By adding counterfactual enhancement processing, the model's understanding of changes in lighting conditions can be promoted and the model's generalization ability can be improved. By adding frequency domain regularization processing to the intermediate images in the model optimization training, the frequency domain information of the image can be fully considered. In summary, the processing capability of the low-light image restoration model for image restoration tasks under different lighting conditions can be effectively improved, and the image restoration accuracy of the low-light image restoration model can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0113] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0114] Figure 1A flowchart of an optimization training method for a low-light image restoration model according to an embodiment of the present application is shown;

[0115] Figure 2 A schematic diagram of a lighting adaptive model according to an embodiment of the present application is shown;

[0116] Figure 3 A flowchart of an image processing method according to an embodiment of the present application is shown;

[0117] Figure 4 A block diagram of a computer device according to an embodiment of the present application is shown;

[0118] Figure 5 A block diagram of a computer device according to another embodiment of the present application is shown. DETAILED DESCRIPTION

[0119] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0120] Figure 1 A flowchart of an optimization training method for a low-light image restoration model according to an embodiment of the present application is shown.

[0121] like Figure 1 As shown, according to an embodiment of the present application, an optimization training method for a low-light image restoration model includes:

[0122] Step 102: Obtain a first training set for the current image restoration task, wherein each group of sample data in the first training set includes a first training image and a second training image.

[0123] The first training image is used to describe the imaging content directly acquired by the image acquisition device, and the second training image is used to describe the actual situation of the imaging content in the actual scene.

[0124] The current image restoration task is an image restoration task under specific lighting conditions. Obtaining a dedicated training set for the current image restoration task will help optimize and adjust the existing model through the training set to a model suitable for the image restoration task under the specific lighting conditions. In other words, the adaptability of the model to the current image restoration task and the specific lighting conditions can be improved, thereby facilitating providing high-quality image restoration results for the current image restoration task and the specific lighting conditions.

[0125] Step 104 , performing counterfactual enhancement processing on the first training image of each group of sample data in the first training set to obtain a third training image corresponding to the first training image.

[0126] Counterfactual Augmentation (CA) is used to change the factors related to lighting conditions while keeping the core features of the first training image unchanged, which can promote the model's understanding of changes in lighting conditions and improve the model's generalization ability.

[0127] In one possible design, the specific manner of counterfactual enhancement processing is as follows.

[0128] First, through the Fourier transform function The first training image is decomposed into an amplitude component and a phase component, wherein the amplitude component of the first training image , the phase component of the first training image .

[0129] Next, the amplitude components of the first training images are adjusted based on a plurality of first super parameters to obtain first enhanced images corresponding to the plurality of first super parameters.

[0130] ,

[0131] is the amplitude component of the first training image, is the first hyperparameter, for through The first enhanced image corresponding to the adjustment. In other words, using multiple first hyperparameters The amplitude component of the first training image is adjusted, and the phase is kept unchanged, so as to obtain a plurality of first enhanced images with different illumination intensities.

[0132] Further, for each of the first enhanced images, a first real part and a first imaginary part corresponding to the first enhanced image are determined based on an amplitude component and a phase component corresponding to the first enhanced image.

[0133] , ,

[0134] is the amplitude component corresponding to the first enhanced image, is the phase component corresponding to the first enhanced image, and The first real part and the first imaginary part are respectively corresponding to the first enhanced image, and then the first real part and the first imaginary part are converted into the spatial domain for fusion to obtain the second enhanced image.

[0135] ,

[0136] is an imaginary unit, is the inverse function, is the second enhanced image.

[0137] Finally, the second enhanced images corresponding to all the first enhanced images are fused to obtain the third training image corresponding to the first training image.

[0138] Optionally, in the fusion process, pixels at corresponding positions in all the first enhanced images are superimposed and averaged.

[0139] Step 106: Replace the first training image corresponding to the first training set with the third training image to obtain a second training set.

[0140] That is, the third training image after counterfactual enhancement replaces the original first training image as the input image of the initial low-light image restoration model.

[0141] Step 108, using the third training image in the second training set as the input image, and using the second training image in the second training set as the control image of the output image, optimizing and training the initial low-light image restoration model until convergence, and obtaining a target low-light image restoration model suitable for the current image restoration task, wherein in each optimization training process, for the residual block inside the encoder network of at least one scale in the model, the intermediate image obtained by processing the residual block is subjected to frequency domain regularization processing.

[0142] Among them, the initial low-light image restoration model is used to reflect the causal relationship between the input image, the lighting condition of the input image, the ideal lighting specific degradation kernel function and the output image. Specifically, before step 102, an initial low-light image restoration model for the current image restoration task is first constructed to analyze the true causal relationship between the input image, lighting information, ideal lighting specific degradation kernel function and restored image in the current image restoration task.

[0143] In step 108, the model includes encoder networks of multiple scales, and each encoder network of each scale has a residual block inside.

[0144] Optionally, frequency domain regularization processing is performed only on the intermediate image obtained by processing a specified number of residual blocks.

[0145] Optionally, the intermediate image obtained by processing the residual block of each encoder network is subjected to frequency domain regularization processing.

[0146] Specifically, the specific manner of performing frequency domain regularization processing on the intermediate image obtained by processing the residual block in step 108 is as follows.

[0147] First, the intermediate image obtained by processing the residual block is regularized based on a preset regularization function to obtain a fourth training image, that is, ,in, is the intermediate image obtained by residual block processing, is the fourth training image.

[0148] Then, through the Fourier transform function The intermediate image and the fourth training image are decomposed into an amplitude component and a phase component respectively.

[0149] Among them, the amplitude component of the intermediate image , the phase component of the intermediate image , the amplitude component of the fourth training image , the phase component of the fourth training image .

[0150] Furthermore, the amplitude component of the intermediate image and the amplitude component of the fourth training image are fused in the frequency domain to obtain an amplitude fusion component. Specifically, the amplitude component of the intermediate image and the amplitude component of the fourth training image are spliced ​​to obtain a first spliced ​​component, and a convolution function is called to perform convolution processing on the first spliced ​​component to obtain ,in, represents the convolution function, Represents a splicing operation, is the amplitude fusion component.

[0151] Similarly, the phase component of the intermediate image and the phase component of the fourth training image are fused in the frequency domain to obtain a phase fusion component. Specifically, the phase component of the intermediate image and the phase component of the fourth training image are spliced ​​to obtain a second spliced ​​component, and a convolution function is called to perform convolution processing on the second spliced ​​component to obtain ,in, represents the convolution function, Represents a splicing operation, is the phase fusion component.

[0152] Next, based on the amplitude fusion component and the phase fusion component, a second real part and a second imaginary part are determined, wherein the second real part , the second imaginary part .

[0153] Finally, the second real part and the second imaginary part are converted to the spatial domain for fusion to obtain the fifth training image , replace the intermediate image with the fifth training image, and continue the current optimization training. is an imaginary unit, is the inverse function.

[0154] In addition, during each optimization training process of step 108, the third training image in the second training set is used as the input image, the sixth training image is determined as the output image through the initial low-light image restoration model, and based on the sixth training image and the second training image, the spatial domain loss and frequency domain loss of the current optimization training are determined.

[0155] ,

[0156] ,

[0157] and are respectively the spatial domain loss and the frequency domain loss of the current optimization training, is the number of the third training images, is the sixth training image, is the second training image.

[0158] Next, based on the spatial domain loss and the frequency domain loss of the current sub-optimization training, a dual-domain loss of the current sub-optimization training is determined.

[0159] ,

[0160] is the dual-domain loss of the current sub-optimization training, is the second hyperparameter.

[0161] Therefore, if the dual-domain loss is within the preset convergence range, the optimization training process is terminated. Conversely, if the dual-domain loss is not within the preset convergence range, the process returns to the step of determining the sixth training image as the output image through the initial low-light image restoration model and enters the next optimization training.

[0162] The above technical solution aims to solve the technical problem in the related art that the neural network model has insufficient ability to restore images under lighting conditions, resulting in poor image restoration results. For a specific image restoration task, the captured images of the training set exclusive to the image restoration task are subjected to counterfactual enhancement processing, and then used as model input for model optimization training. At the same time, during each optimization training process, the intermediate image obtained by processing the residual block inside the encoder network of at least one scale in the model is subjected to frequency domain regularization processing. By adding counterfactual enhancement processing, the model's understanding of changes in lighting conditions can be promoted and the model's generalization ability can be improved. By adding frequency domain regularization processing to the intermediate images in the model optimization training, the frequency domain information of the image can be fully considered. In summary, the processing capability of the low-light image restoration model for image restoration tasks under different lighting conditions can be effectively improved, and the image restoration accuracy of the low-light image restoration model can be improved.

[0163] It should be added that after the initial low-light image restoration model is optimized and trained to convergence, the image restoration accuracy of the obtained target low-light image restoration model can also be tested.

[0164] Specifically, first, a test set for the current image restoration task can be obtained.

[0165] Each group of test data in the test set includes a first test image and a second test image, wherein the first test image is used to describe the imaging content directly acquired by the image acquisition device, and the second test image is used to describe the actual situation of the imaging content in an actual scene.

[0166] Next, the first test image is subjected to counterfactual enhancement processing to obtain a third test image, that is, images with different light intensities are obtained through counterfactual enhancement processing, which are used as input of the target low-light image restoration model.

[0167] At this point, based on the third test image and the target low-light image restoration model, a fourth test image is determined, and the similarity between the second test image and the fourth test image is obtained, and based on the similarity, the image restoration accuracy of the target low-light image restoration model is determined.

[0168] Optionally, the ratio of identical pixels in the second test image and the fourth test image is used as the similarity between the second test image and the fourth test image.

[0169] Optionally, the proportion of pixels in which the degree of similarity between the second test image and the fourth test image is greater than a specified threshold is used as the similarity between the second test image and the fourth test image.

[0170] Therefore, the processing capability of the target low-light image restoration model obtained through training and optimization for the current image restoration task can be effectively detected, thus achieving a reasonable evaluation of the model.

[0171] In addition, it should be added that the counterfactual enhancement processing in the above technical solution can be implemented in the form of a plug-in, that is, calling the counterfactual enhancement processing plug-in to perform counterfactual enhancement processing on the first training image of each group of sample data in the first training set.

[0172] Optionally, the counterfactual enhancement processing in the above technical solution is also integrated into a module outside the model, and is called by hook. Specifically, in the model optimization training, in response to the acquisition of the first training set, the first hook function is called, wherein the first hook function is used to perform counterfactual enhancement processing on the first training image of each group of sample data in the first training set.

[0173] Similarly, the frequency domain regularization processing in the above technical solution can be implemented in the form of a plug-in, that is, calling the frequency domain regularization processing plug-in to perform frequency domain regularization processing on the intermediate image obtained by processing the residual block.

[0174] Optionally, a frequency domain regularization processing module can also be integrated into the model.

[0175] Optionally, a frequency domain regularization processing module may be integrated outside the model, and the external frequency domain regularization processing module may be called through a hook method. Specifically, in response to the acquisition of the intermediate image, a second hook function is called, wherein the second hook function is used to perform frequency domain regularization processing on the intermediate image obtained by processing the residual block.

[0176] In summary, this technical solution can re-examine the learning paradigm of the illumination adaptation method from a causal perspective, and propose an illumination generalization adaptation method based on causal reasoning, which fully captures the causal effect of the illumination adaptation task through two important components. Specifically, on the one hand, a counterfactual-based image enhancement module is proposed to fully expand the available value range of the variable for adjustment, and on the other hand, a regularization fusion module based on the frequency domain is proposed to develop the generalization ability of regularization, and realize the target information fusion from the two aspects of style and structure. Among them, the counterfactual-based image enhancement module enhances the image in the frequency domain through Fourier transform, expands the available value range of the input image, and the regularization fusion module based on the frequency domain fuses the input image and the regularized input image in the frequency domain, realizing targeted information fusion from the two aspects of style and structure. So far, this technical solution adopts a plug-and-play approach, which can be applied to a variety of low-light image restoration model architectures, and can improve the generalization performance of a variety of low-light image restoration methods.

[0177] Figure 2A schematic diagram of a lighting adaptive model according to an embodiment of the present application is shown.

[0178] like Figure 2 As shown, the illumination adaptation model adopts an encoder-decoder framework.

[0179] For the input image, first, the illumination adaptive model calls the counterfactual-based image enhancement module to perform counterfactual enhancement processing on the captured image to obtain a counterfactual enhanced image. The captured image refers to the imaging content directly acquired by the image acquisition device in the current image restoration task.

[0180] Then, the spatial size is gradually reduced and the number of channels is expanded through three scale encoder networks.

[0181] Specifically, the encoder network of each scale consists of three parts: convolutional layer, residual block (Resblock) and frequency-domain-based regularization fusion module. The frequency-domain-based regularization fusion module can be integrated into the model or called outside the model in the form of a plug-in.

[0182] At the same time, based on the first two encoder networks in the model, the features of the downsampled image after passing through the shallow neural network (Shallow Layer) are introduced, and the features are spliced ​​with the features after passing through the frequency domain-based regularization fusion module, that is, the features are spliced ​​with the results of the intermediate image after frequency domain regularization processing.

[0183] After that, the concatenated features are passed through the decoder network of three scales to gradually restore the spatial size and adjust the number of channels. Furthermore, the decoder features are concatenated with the encoder features and the channel dimension is adjusted through a 1x1 convolution layer. Finally, the final 3x3 convolution layer and image-level residual connection are used to generate the final clear image.

[0184] Figure 3 A flowchart of an image processing method according to an embodiment of the present application is shown.

[0185] like Figure 3 As shown, using Figure 1 Examples and Figure 2 The method for performing image processing using the target low-light image restoration model shown in the embodiment includes:

[0186] Step 302: Acquire a first target image of the current image restoration task, wherein the first target image is used to describe the imaging content directly acquired by the image acquisition device.

[0187] Step 304: Perform counterfactual enhancement processing on the first target image to obtain a second target image corresponding to the first target image.

[0188] Step 306, using the second target image as the input image of the target low-light image restoration model, outputting a third target image corresponding to the second target image through the target low-light image restoration model, wherein the third target image is used to describe the actual situation of the imaging content of the first target image in the actual scene, and in the process of the target low-light image restoration model performing light adaptation image restoration processing on the second target image, for the residual block inside the encoder network of at least one scale within the model, frequency domain regularization processing is performed on the intermediate image obtained by processing the residual block.

[0189] Therefore, when restoring any target image of the current image restoration task, the target image can be subjected to counterfactual enhancement processing and then input into the target low-light image restoration model. Moreover, in the process of processing the input image in the target low-light image restoration model, the intermediate image obtained by processing at least one residual block in the model can be subjected to frequency domain regularization processing.

[0190] Therefore, by introducing the counterfactual-based image enhancement module and the frequency-domain-based regularization fusion module into the traditional low-light image restoration model, it can adapt to image restoration tasks under different conditions, improve the adaptability of the low-light image restoration model to different lighting conditions in the image restoration task, and realize image restoration under different light intensities. It has a positive effect on the development of multiple research fields such as image restoration, security monitoring, image intelligence analysis and medical image analysis.

[0191] The embodiment of the present application provides an optimization training device for a low-light image restoration model, including:

[0192] A first training set acquisition unit, used to acquire a first training set of a current image restoration task, wherein each group of sample data of the first training set includes a first training image and a second training image, the first training image is used to describe the imaging content directly acquired by the image acquisition device, and the second training image is used to describe the real situation of the imaging content in an actual scene;

[0193] a counterfactual enhancement unit, configured to perform counterfactual enhancement processing on a first training image of each group of sample data in the first training set to obtain a third training image corresponding to the first training image;

[0194] A second training set acquisition unit, configured to replace the first training image corresponding to the first training image in the first training set with the third training image to obtain a second training set;

[0195] a model optimization training unit, configured to use the third training image in the second training set as an input image and the second training image in the second training set as a control image of an output image, optimize and train an initial low-light image restoration model until convergence, and obtain a target low-light image restoration model suitable for the current image restoration task, wherein the initial low-light image restoration model is used to reflect the causal relationship between the input image, the illumination condition of the input image, the ideal illumination specific degradation kernel function, and the output image;

[0196] The frequency domain regularization processing unit is used to perform frequency domain regularization processing on the intermediate image obtained by processing the residual block inside the encoder network of at least one scale in the model during each optimization training process of the model optimization training unit.

[0197] In one embodiment of the present application, optionally, the counterfactual enhancement unit is used to: decompose the first training image into an amplitude component and a phase component through a Fourier transform function; adjust the amplitude component of the first training image based on multiple first hyperparameters to obtain first enhanced images corresponding to each of the multiple first hyperparameters, wherein:

[0198] ,

[0199] is the amplitude component of the first training image, is the first hyperparameter, for through the first enhanced image corresponding to the adjustment; for each of the first enhanced images, determining the first real part and the first imaginary part corresponding to the first enhanced image based on the amplitude component and the phase component corresponding to the first enhanced image, wherein,

[0200] , ,

[0201] is the amplitude component corresponding to the first enhanced image, is the phase component corresponding to the first enhanced image, and are the first real part and the first imaginary part corresponding to the first enhanced image respectively; the first real part and the first imaginary part are converted into the spatial domain for fusion to obtain a second enhanced image, wherein,

[0202] ,

[0203] is an imaginary unit, is the inverse function, The second enhanced image is the second enhanced image; and the second enhanced images corresponding to all the first enhanced images are fused to obtain the third training image corresponding to the first training image.

[0204] In one embodiment of the present application, optionally, the counterfactual enhancement unit includes:

[0205] The first execution unit is used to call the counterfactual enhancement processing plug-in to perform counterfactual enhancement processing on the first training image of each group of sample data in the first training set.

[0206] In one embodiment of the present application, optionally, the counterfactual enhancement unit includes:

[0207] The second execution unit is used to call a first hook function in response to obtaining the first training set, wherein the first hook function is used to perform counterfactual enhancement processing on the first training image of each group of sample data in the first training set.

[0208] In one embodiment of the present application, optionally, the frequency domain regularization processing unit is used to: perform a regularization transformation on the intermediate image obtained by processing the residual block based on a preset regularization function to obtain a fourth training image; decompose the intermediate image and the fourth training image into an amplitude component and a phase component respectively through a Fourier transform function; fuse the amplitude component of the intermediate image with the amplitude component of the fourth training image in the frequency domain to obtain an amplitude fusion component, and fuse the phase component of the intermediate image with the phase component of the fourth training image in the frequency domain to obtain a phase fusion component; determine the second real part and the second imaginary part based on the amplitude fusion component and the phase fusion component, wherein,

[0209] , ,

[0210] is the amplitude fusion component, is the phase fusion component, and are the second real part and the second imaginary part respectively; convert the second real part and the second imaginary part into the spatial domain for fusion to obtain a fifth training image, wherein,

[0211] ,

[0212] is an imaginary unit, is the inverse function, The fifth training image is used to replace the intermediate image, and the current optimization training is continued.

[0213] In one embodiment of the present application, optionally, the frequency domain regularization processing unit is used to: perform a splicing operation on the amplitude component of the intermediate image and the amplitude component of the fourth training image to obtain a first splicing component; call a convolution function to perform convolution processing on the first splicing component; and perform a splicing operation on the phase component of the intermediate image and the phase component of the fourth training image to obtain a second splicing component; call a convolution function to perform convolution processing on the second splicing component.

[0214] In one embodiment of the present application, optionally, the frequency domain regularization processing unit includes:

[0215] The third execution unit is used to call the frequency domain regularization processing plug-in to perform frequency domain regularization processing on the intermediate image obtained by processing the residual block.

[0216] In one embodiment of the present application, optionally, the frequency domain regularization processing unit includes:

[0217] The fourth execution unit is used to call the second hook function in response to the acquisition of the intermediate image, wherein the second hook function is used to perform frequency domain regularization processing on the intermediate image obtained by processing the residual block.

[0218] In one embodiment of the present application, optionally, the model optimization training unit is used to: in each optimization training process, use the third training image in the second training set as the input image, determine the sixth training image as the output image through the initial low-light image restoration model; based on the sixth training image and the second training image, determine the spatial domain loss and frequency domain loss of the current optimization training, wherein,

[0219] ,

[0220] ,

[0221] and are respectively the spatial domain loss and the frequency domain loss of the current optimization training, is the number of the third training images, is the sixth training image, is the second training image; based on the spatial domain loss and the frequency domain loss of the current sub-optimization training, determining the dual-domain loss of the current sub-optimization training, wherein,

[0222] ,

[0223] is the dual-domain loss of the current sub-optimization training, is the second hyperparameter; if the dual-domain loss is within the preset convergence range, the optimization training process is terminated; if the dual-domain loss is not within the preset convergence range, the process returns to the step of determining the sixth training image as the output image through the initial low-light image restoration model and enters the next optimization training.

[0224] In one embodiment of the present application, optionally, the device further includes:

[0225] A model testing unit is used to obtain a test set for a current image restoration task, wherein each group of test data in the test set includes a first test image and a second test image, the first test image is used to describe the imaging content directly acquired by the image acquisition device, and the second test image is used to describe the actual situation of the imaging content in the actual scene; the first test image is subjected to counterfactual enhancement processing to obtain a third test image; based on the third test image and the target low-light image restoration model, a fourth test image is determined; the similarity between the second test image and the fourth test image is obtained; based on the similarity, the image restoration accuracy of the target low-light image restoration model is determined.

[0226] The device uses Figure 1 and Figure 2 The solution described in any one of the embodiments therefore has all the above-mentioned technical effects and will not be repeated here.

[0227] The present application embodiment provides an image processing device, which uses the target low-light image restoration model described in any one of the above embodiments to perform light-adaptive image restoration processing for a current image restoration task, and the device includes:

[0228] A target image acquisition unit, used to acquire a first target image of a current image restoration task, wherein the first target image is used to describe the imaging content directly acquired by the image acquisition device;

[0229] a counterfactual enhancement unit, configured to perform counterfactual enhancement processing on the first target image to obtain a second target image corresponding to the first target image;

[0230] a model processing unit, configured to use the second target image as an input image of a target low-light image restoration model, and output a third target image corresponding to the second target image through the target low-light image restoration model, wherein the third target image is used to describe a real situation of the imaging content of the first target image in an actual scene;

[0231] A frequency domain regularization processing unit is used to perform frequency domain regularization processing on an intermediate image obtained by processing a residual block inside an encoder network of at least one scale in the model during the process in which the model processing unit performs light adaptation image restoration processing on the second target image.

[0232] The device uses Figure 3 The solution described in any one of the embodiments therefore has all the above-mentioned technical effects and will not be repeated here.

[0233] In addition, in one embodiment, the present application provides a computer device, which may be a server, and its internal structure diagram may be as shown in FIG. Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, the method described in any of the above embodiments can be implemented.

[0234] In one embodiment, the present application further provides a computer device, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, the method described in any of the above embodiments can be implemented.

[0235] Any of the above-mentioned computer devices in the embodiments of the present application may exist in various forms, including but not limited to:

[0236] (1) Mobile communication devices: These devices are characterized by their mobile communication functions and their main purpose is to provide voice and data communications. These terminals include: smart phones (such as iPhone), multimedia phones, functional phones, and low-end phones.

[0237] (2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, such as iPad.

[0238] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players (such as iPods), handheld game consoles, e-books, as well as smart toys, wearable devices, and portable car navigation devices.

[0239] (4) Server: A device that provides computing services. The server consists of a processor, hard disk, memory, system bus, etc. The server is similar to a general computer architecture, but because it needs to provide highly reliable services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0240] (5) Other electronic devices with data interaction functions.

[0241] In addition, an embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to perform the following steps:

[0242] Acquire a first training set for the current image restoration task, wherein each set of sample data in the first training set includes a first training image and a second training image, the first training image is used to describe the imaging content directly acquired by the image acquisition device, and the second training image is used to describe the actual situation of the imaging content in the actual scene;

[0243] Performing counterfactual enhancement processing on the first training image of each group of sample data in the first training set to obtain a third training image corresponding to the first training image;

[0244] replacing the first training image corresponding to the first training image in the first training set with the third training image to obtain a second training set;

[0245] Taking the third training image in the second training set as the input image and the second training image in the second training set as the control image of the output image, the initial low-light image restoration model is optimized and trained until convergence to obtain a target low-light image restoration model suitable for the current image restoration task, wherein the initial low-light image restoration model is used to reflect the causal relationship between the input image, the lighting condition of the input image, the ideal lighting specific degradation kernel function and the output image, and in each optimization training process, for the residual block inside the encoder network of at least one scale in the model, the intermediate image obtained by processing the residual block is subjected to frequency domain regularization processing.

[0246] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant description in the aforementioned method embodiment. In order to avoid repetition, they will not be described one by one here.

[0247] The technical solution of the present application is described in detail above in conjunction with the accompanying drawings. Through the technical solution of the present application, for a specific image restoration task, the captured images of the training set exclusive to the image restoration task can be subjected to counterfactual enhancement processing, and then used as model input for model optimization training. At the same time, in each optimization training process, the intermediate image obtained by processing the residual block inside the encoder network of at least one scale in the model is subjected to frequency domain regularization processing. By adding counterfactual enhancement processing, the model's understanding of changes in lighting conditions can be promoted and the model's generalization ability can be improved. By adding frequency domain regularization processing to the intermediate images in the model optimization training, the frequency domain information of the image can be fully considered. In summary, the processing capability of the low-light image restoration model for image restoration tasks under different lighting conditions can be effectively improved, and the image restoration accuracy of the low-light image restoration model is improved.

[0248] It should be understood that although the terms first, second, etc. may be used to describe training images in the embodiments of the present application, the training images should not be limited to these terms. These terms are only used to distinguish training images from each other. For example, without departing from the scope of the embodiments of the present application, the first training image may also be referred to as the second training image, and similarly, the second training image may also be referred to as the first training image.

[0249] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0250] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0251] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0252] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0253] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0254] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. An optimization training method for a low-light image restoration model, characterized in that: include: Acquire a first training set of the current image restoration task, wherein each set of sample data of the first training set includes a first training image and a second training image, the first training image is used to describe the imaging content directly acquired by the image acquisition device, and the second training image is used to describe the real situation of the imaging content in the actual scene; Performing counterfactual enhancement processing on the first training image of each group of sample data in the first training set to obtain a third training image corresponding to the first training image; replacing the first training image corresponding to the first training image in the first training set with the third training image to obtain a second training set; The third training image in the second training set is used as an input image, and the second training image in the second training set is used as a control image of the output image, and the initial low-light image restoration model is optimized and trained until convergence to obtain a target low-light image restoration model suitable for the current image restoration task, wherein: The initial low-light image restoration model is used to reflect the causal relationship between the input image, the lighting condition of the input image, the ideal lighting specific degradation kernel function and the output image, In each optimization training process, for a residual block inside an encoder network of at least one scale in the model, a frequency domain regularization process is performed on an intermediate image obtained by processing the residual block.

2. The method according to claim 1, characterized in that: The performing counterfactual enhancement processing on the first training image of each group of sample data in the first training set includes: Decomposing the first training image into an amplitude component and a phase component by a Fourier transform function; The amplitude components of the first training image are adjusted based on a plurality of first hyperparameters to obtain first enhanced images corresponding to the plurality of first hyperparameters, wherein: ， is the amplitude component of the first training image, is the first hyperparameter, is the amplitude component corresponding to the first enhanced image; For each of the first enhanced images, based on the amplitude component and the phase component corresponding to the first enhanced image, a first real part and a first imaginary part corresponding to the first enhanced image are determined, wherein: ， is the phase component corresponding to the first enhanced image, are the first real part and the first imaginary part corresponding to the first enhanced image respectively; The first real part and the first imaginary part are converted into the spatial domain for fusion to obtain a second enhanced image, wherein: ， is an imaginary unit, is the inverse function, is the second enhanced image; The second enhanced images corresponding to all the first enhanced images are fused to obtain the third training image corresponding to the first training image.

3. The method according to claim 2, characterized in that The performing frequency domain regularization processing on the intermediate image obtained by processing the residual block includes: Performing regularization transformation on the intermediate image obtained by processing the residual block based on a preset regularization function to obtain a fourth training image; Decomposing the intermediate image and the fourth training image into an amplitude component and a phase component respectively by a Fourier transform function; fusing the amplitude component of the intermediate image with the amplitude component of the fourth training image in the frequency domain to obtain an amplitude fusion component, and fusing the phase component of the intermediate image with the phase component of the fourth training image in the frequency domain to obtain a phase fusion component; Based on the amplitude fusion component and the phase fusion component, a second real part and a second imaginary part are determined, wherein: ， is the amplitude fusion component, is the phase fusion component, are the second real part and the second imaginary part respectively; The second real part and the second imaginary part are converted into the spatial domain for fusion to obtain a fifth training image, wherein: ， is an imaginary unit, is the inverse function, is the fifth training image; The intermediate image is replaced by the fifth training image, and the current optimization training is continued.

4. The method according to claim 3, characterized in that The fusing the amplitude component of the intermediate image and the amplitude component of the fourth training image in the frequency domain comprises: Performing a splicing operation on the amplitude component of the intermediate image and the amplitude component of the fourth training image to obtain a first splicing component; Calling a convolution function to perform convolution processing on the first spliced ​​component; The fusing the phase component of the intermediate image and the phase component of the fourth training image in the frequency domain comprises: Performing a splicing operation on the phase component of the intermediate image and the phase component of the fourth training image to obtain a second splicing component; A convolution function is called to perform convolution processing on the second spliced ​​component.

5. The method according to any one of claims 1 to 4, characterized in that The performing counterfactual enhancement processing on the first training image of each group of sample data in the first training set includes: Calling a counterfactual enhancement processing plug-in to perform counterfactual enhancement processing on the first training image of each group of sample data in the first training set; or In response to acquiring the first training set, calling a first hook function, wherein the first hook function is used to perform counterfactual enhancement processing on a first training image of each group of sample data in the first training set; as well as The performing frequency domain regularization processing on the intermediate image obtained by processing the residual block includes: Calling a frequency domain regularization processing plug-in to perform frequency domain regularization processing on the intermediate image obtained by processing the residual block; or In response to the acquisition of the intermediate image, a second hook function is called, wherein the second hook function is used to perform frequency domain regularization processing on the intermediate image obtained by processing the residual block.

6. The method according to claim 1, characterized in that The method of using the third training image in the second training set as an input image and the second training image in the second training set as a control image of an output image, and optimizing and training the initial low-light image restoration model until convergence, comprises: In each optimization training process, the third training image in the second training set is used as an input image, and a sixth training image is determined as an output image through the initial low-light image restoration model; Based on the sixth training image and the second training image, the spatial domain loss and the frequency domain loss of the current sub-optimization training are determined, wherein: ， ， and are respectively the spatial domain loss and the frequency domain loss of the current optimization training, is the number of the third training images, is the sixth training image, is the second training image; Based on the spatial domain loss and the frequency domain loss of the current sub-optimization training, a dual-domain loss of the current sub-optimization training is determined, wherein: ， is the dual-domain loss of the current sub-optimization training, is the second hyperparameter; If the dual-domain loss is within a preset convergence range, the optimization training process is terminated; If the dual-domain loss is not within the preset convergence range, return to the step of determining the sixth training image as the output image through the initial low-light image restoration model and enter the next optimization training.

7. The method according to claim 6, characterized in that Also includes: Acquire a test set of the current image restoration task, wherein each group of test data of the test set includes a first test image and a second test image, the first test image is used to describe the imaging content directly acquired by the image acquisition device, and the second test image is used to describe the real situation of the imaging content in the actual scene; Performing counterfactual enhancement processing on the first test image to obtain a third test image; Determining a fourth test image based on the third test image and the target low-light image restoration model; Obtaining a similarity between the second test image and the fourth test image; Based on the similarity, the image restoration accuracy of the target low-light image restoration model is determined.

8. An image processing method, characterized in that: Using the method according to any one of claims 1 to 7 to perform light adaptation image restoration processing for the current image restoration task, the method comprising: Acquire a first target image of a current image restoration task, wherein the first target image is used to describe imaging content directly acquired by an image acquisition device; Performing counterfactual enhancement processing on the first target image to obtain a second target image corresponding to the first target image; The second target image is used as an input image of a target low-light image restoration model, and a third target image corresponding to the second target image is output through the target low-light image restoration model, where the third target image is used to describe the actual situation of the imaging content of the first target image in the actual scene, wherein: During the process of the target low-light image restoration model performing light adaptation image restoration processing on the second target image, for the residual blocks inside the encoder network of at least one scale in the model, frequency domain regularization processing is performed on the intermediate image obtained by processing the residual blocks.

9. A computer device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are configured to enable the processor to execute the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: Computer executable instructions are stored, and the computer executable instructions are configured to execute the method according to any one of claims 1 to 8.

Images