Exposure correction method and device in video Internet of Things environment

Abstract

The invention provides an exposure correction method in a video Internet of Things environment, and the method comprises the steps: inputting a to-be-adjusted image into a pre-trained feature extraction model, so as to enable the feature extraction model to output first feature information corresponding to the to-be-adjusted image. And inputting the first feature information and preset second feature information into a pre-trained feature fusion model to enable the feature fusion model to output fusion features. And inputting the fusion feature into a pre-trained diffusion network model to enable the diffusion network model to adjust the exposure of the to-be-adjusted image to a normal value, and outputting an adjusted image. The feature extraction is performed on the to-be-adjusted image, the first feature information extracted from the to-be-adjusted image is fused with the preset second feature information, so that the texture information of the to-be-adjusted image is richer, the fusion feature is adjusted by using the diffusion network, and the accurately exposed adjusted image is output. Therefore, the existing information in the to-be-adjusted image is utilized to the maximum extent, and the output adjusted image keeps more detail information.

Description

Technical Field

[0001] The present application relates to the technical field of image parameter adjustment, and in particular to a method and device for exposure correction in a video Internet of Things environment. Background Art

[0002] With the rapid development of electronic devices, we often take photos in our daily lives. However, under suboptimal lighting conditions, images with abnormal exposure are often captured, resulting in unreasonable brightness and loss of structural information. Exposure correction aims to enhance images damaged by improper exposure to achieve satisfactory visual effects. Usually, photometric anomalies occur globally and are accompanied by texture loss. Existing methods use naive Fourier transform to attempt to capture global photometric anomalies and repair degraded images. However, naive Fourier transform can cause incomplete image decoupling, which in turn leads to problems with photometric and texture restoration. Summary of the Invention

[0003] In view of this, the purpose of this application is to propose an exposure correction method in a video Internet of Things environment.

[0004] Based on the above-mentioned purpose, the present application provides an exposure correction method in a video Internet of Things environment, including: inputting the image to be adjusted into a pre-trained feature extraction model, so that the feature extraction model outputs first feature information corresponding to the image to be adjusted; inputting the first feature information and the preset second feature information into a pre-trained feature fusion model, so that the feature fusion model outputs a fusion feature; wherein the second feature information is the feature information output by the feature extraction model corresponding to the preset normal exposure image; inputting the fusion feature into a pre-trained diffusion network model, so that the diffusion network model adjusts the exposure of the image to be adjusted to a normal value, and outputs the adjusted image.

[0005] In some embodiments, the first feature information includes original image features and downsampled image features; inputting the image to be adjusted into a pre-trained feature extraction model so that the feature extraction model outputs first feature information corresponding to the image to be adjusted specifically includes: downsampling the image to be adjusted at least once to obtain a downsampled image; performing feature extraction on the image to be adjusted to obtain the original image features; and performing feature extraction on the downsampled image to obtain the downsampled image features.

[0006] In some embodiments, feature extraction is performed on the image to be adjusted to obtain the original image features, specifically including: calculating the first amplitude information and the first phase information of the image to be adjusted; setting a first mask and a second mask, wherein the first mask and the second mask are determined by the following formula: X′=iFFT(A(x)*M+P(X)*(1-M)), wherein X′ represents the mask Fourier prior of the first mask and the second mask; A(x) represents the first amplitude information; P(X) represents the first phase information; M represents the first mask; (1-M) represents the second mask; calculating the second amplitude information based on the first mask and the first amplitude information; calculating the second phase information based on the second mask and the first phase information; performing inverse Fourier calculation on the second amplitude information and the second phase information to obtain the original image features.

[0007] In some embodiments, the first feature information includes a first amplitude feature and a first phase feature, and the second feature information includes a second amplitude feature and a second phase feature; inputting the first feature information and the preset second feature information into a pre-trained feature fusion model so that the feature fusion model outputs a fusion feature, specifically includes: fusing the first amplitude feature and the second amplitude feature based on a preset fusion algorithm to obtain a fused amplitude feature; fusing the first phase feature and the second phase feature based on a fusion algorithm to obtain a fused phase feature; and performing inverse Fourier calculation on the fused amplitude feature and the fused phase feature to obtain the fused feature.

[0008] In some embodiments, the diffusion network model includes a forward propagation network and a back propagation network; inputting the fused features into a pre-trained diffusion network model so that the diffusion network model adjusts the exposure of the image to be adjusted to a normal value and outputs an adjusted image, specifically includes: inputting the fused feature information into the forward propagation network so that the forward propagation network adds noise to the image to be adjusted according to the fused feature information to obtain a noisy image; inputting the noisy image into a back propagation network, and the back propagation network denoises the noisy image according to a preset noise reduction model to obtain the adjusted image.

[0009] The present application also provides an exposure correction device in a video Internet of Things environment, including: a feature extraction module, used to input the image to be adjusted into a pre-trained feature extraction model, so that the feature extraction model outputs first feature information corresponding to the image to be adjusted; a feature fusion module, used to input the first feature information and preset second feature information into a pre-trained feature fusion model, so that the feature fusion model outputs a fusion feature; wherein the second feature information is the feature information output by the feature extraction model corresponding to the preset normal exposure image; an image adjustment module, used to input the fusion feature into a pre-trained diffusion network model, so that the diffusion network model adjusts the exposure of the image to be adjusted to a normal value, and outputs the adjusted image.

[0010] In some embodiments, the first feature information includes original image features and downsampled image features; the feature extraction module specifically includes: a downsampling module, used to downsample the image to be adjusted at least once to obtain a downsampled image; a first image extraction module, used to extract features from the image to be adjusted to obtain the original image features; and a second image extraction module, used to extract features from the downsampled image to obtain the downsampled image features.

[0011] In some embodiments, the first image extraction module specifically includes: a feature calculation module, used to calculate the first amplitude information and first phase information of the image to be adjusted; a mask adjustment module, used to set the first mask and the second mask, wherein the first mask and the second mask are determined by the following formula: X′=iFFT(A(x)*M+P(X)*(1-M)) wherein X′ represents the mask Fourier prior of the first mask and the second mask; A(x) represents the first amplitude information; P(X) represents the first phase information; M represents the first mask; (1-M) represents the second mask; a feature adjustment module, used to calculate the second amplitude information based on the first mask and the first amplitude information; and calculate the second phase information based on the second mask and the first phase information; an inverse Fourier module, used to perform inverse Fourier calculation on the second amplitude information and the second phase information to obtain the original image features.

[0012] The present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above methods when executing the program.

[0013] The present application also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute any of the above methods.

[0014] From the above description, it can be seen that the exposure correction method provided by the present application in a video Internet of Things environment extracts features of the image to be adjusted, and fuses the first feature information extracted from the image to be adjusted with the preset second feature information, thereby enriching the texture information of the image to be adjusted, and then uses the diffusion network to adjust the fused features, and outputs an adjusted image with accurate exposure, thereby maximizing the use of existing information in the image to be adjusted, so that the output adjusted image retains more detail information. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in this application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are merely embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0016] Figure 1 A flowchart of an exposure correction method in a video IoT environment provided by an embodiment of the present application;

[0017] Figure 2 A flowchart of a first feature information extraction method provided in an embodiment of the present application;

[0018] Figure 3 A schematic diagram of a flow chart of a method for extracting features from an original image provided in an embodiment of the present application;

[0019] Figure 4 A flow chart of a fusion feature calculation method provided in an embodiment of the present application;

[0020] Figure 5 A flowchart of an image transmission method provided in an embodiment of the present application;

[0021] Figure 6 A flowchart of an image exposure correction model training method provided in another embodiment of the present application;

[0022] Figure 7 This is a schematic diagram of the structure of an exposure correction device in a video IoT environment provided by an embodiment of the present application;

[0023] Figure 8 A schematic structural diagram of an image exposure correction model provided in another embodiment of the present application;

[0024] Figure 9 This is a more specific schematic diagram of the hardware structure of an electronic device provided by this embodiment. DETAILED DESCRIPTION

[0025] In order to make the objectives, technical solutions and advantages of this application more clear, this application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0026] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should have the usual meanings understood by people with ordinary skills in the field to which this application belongs. The "first", "second" and similar words used in the embodiments of the present application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0027] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0028] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operation of the disclosed technical solution based on the prompt message.

[0029] As an optional but non-limiting implementation, in response to a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0030] It is understandable that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0031] This application proposes a method for exposure correction in a video IoT environment, such as Figure 1 As shown, including:

[0032] Step S1: input the image to be adjusted into a pre-trained feature extraction model, so that the feature extraction model outputs first feature information corresponding to the image to be adjusted.

[0033] In this embodiment, the image to be adjusted is input into a feature extraction model, thereby converting the image to be adjusted into first feature information of a specific size and specification, which facilitates subsequent processing.

[0034] Among them, the image to be adjusted is an image with abnormal exposure. The exposure of the image to be adjusted is adjusted through the feature extraction model, feature fusion model and diffusion network model, so as to output an adjusted image with normal exposure corresponding to the image to be adjusted.

[0035] The type of feature extraction model is not limited. For example, it can be one of a boundary feature extraction model, a Fourier feature extraction model, and a geometric parameter extraction model. A reasonable selection can be made according to actual application needs.

[0036] Step S2: Input the first feature information and the preset second feature information into a pre-trained feature fusion model so that the feature fusion model outputs a fusion feature. The second feature information is feature information output by the feature extraction model corresponding to the preset normal exposure image.

[0037] The second feature information is obtained by inputting a preset normal exposure image into a feature extraction model so that the feature extraction model extracts features from the normal exposure image to obtain a feature image.

[0038] The feature fusion model is used to fuse the first feature information with the second feature information, so that the fused features have clearer texture details and avoid loss of adjusted image details.

[0039] In some optional embodiments, the normally exposed images in the dataset can be divided into a 4:1 ratio (20% of the images are used as reference images) using a grayscale image similarity strategy. For each image to be adjusted, the cosine similarity between the VGG features of the grayscale images of the two images in the reference image pool is calculated one by one. Finally, the most similar normally exposed image is used as the reference image, and the second feature information obtained from the reference image through the feature extraction network is used as the input information of the fusion model. By selecting the normally exposed image with a high similarity to the image to be adjusted as the reference image to extract the second feature information, the texture details of the adjusted image are more accurate.

[0040] In step S3, the fused features are input into a pre-trained diffusion network model, so that the diffusion network model adjusts the exposure of the image to be adjusted to a normal value and outputs the adjusted image.

[0041] Among them, the diffusion network model adjusts the texture information of the image to be adjusted by fusing features, thereby avoiding the loss of texture information of the image to be adjusted.

[0042] In some embodiments, the first feature information includes original image features and downsampled image features.

[0043] like Figure 2 As shown, step S1 specifically includes:

[0044] Step S11 : down-sample the image to be adjusted at least once to obtain a down-sampled image.

[0045] Step S12: extracting features from the image to be adjusted to obtain features of the original image.

[0046] Step S13: extract features from the downsampled image to obtain downsampled image features.

[0047] In this embodiment, by downsampling the image to be adjusted and performing feature extraction on the image to be adjusted and the downsampled image respectively, information of different densities can be extracted respectively, thereby avoiding information loss.

[0048] The downsampling operation downsamples the image with the original resolution (C, H, W) by 2 times through convolution Conv(in_channel=C,out_channel=2*C,kernel_size=3,stride=2,padding=1), and the final feature resolution is (2*C,H / 2,W / 2).

[0049] In some embodiments, as Figure 3 As shown, step S12 specifically includes:

[0050] Step S101 : calculating first amplitude information and first phase information of an image to be adjusted.

[0051] Step S102: Set a first mask and a second mask, wherein the first mask and the second mask are determined by the following formula:

[0052] X′=iFFT(A(x)*M+P(X)*(1-M))

[0053] Where X′ represents the mask Fourier prior of the first mask and the second mask. A(x) represents the first amplitude information. P(x) represents the first phase information. M represents the first mask. (1-M) represents the second mask.

[0054] Step S103: Calculate second amplitude information according to the first mask and the first amplitude information, and calculate second phase information according to the second mask and the first phase information.

[0055] Step S104 , performing inverse Fourier transform calculation on the second amplitude information and the second phase information to obtain original image features.

[0056] In this embodiment, the feature extraction module converts the image to be adjusted into the frequency domain, and then calculates the first amplitude information and the first phase information according to the values ​​of the real part and the imaginary part of the image in the frequency domain.

[0057] The frequency domain conversion formula for converting the image to be adjusted into the frequency domain is implemented by the following formula:

[0058] Where (h, w) represents each pixel in the image to be adjusted; H represents the number of pixels in the height direction of the image to be adjusted; W represents the number of pixels in the width direction of the image to be adjusted; j represents the imaginary unit; and (u, v) are the pixel coordinates after the image is converted to the frequency domain.

[0059] The first amplitude information and the first phase information are calculated using the following formula:

[0060] Wherein, R represents the real part of the image calculated according to the frequency domain conversion formula; I represents the imaginary part of the image calculated according to the frequency domain conversion formula; A(X) represents the first amplitude information; and P(X) represents the first phase information.

[0061] Obtain mask amplitude information (A\cdot M) by element-by-element multiplication of the first mask M and the first amplitude information A(X);

[0062] The second mask (1-M) is multiplied element-by-element by the first phase information P(X) to obtain the mask phase information (P\cdot(1-M).

[0063] Then pass Conv(kernel=1,stride=1,padding=0)

[0064] ->LReLU->Conv(kernel=1,stride=1,padding=0)->LReLU processes (A\cdotM) and (P\cdot(1-M)) to obtain the second amplitude information A' and the second phase information P'.

[0065] Finally, the second amplitude information and the second phase information are converted back to the image domain through iFFT (inverse Fourier transform), which is defined as R_f(C,H,W).

[0066] For the spatial branch, R_s(C,H,W) is obtained through the sequence of Conv(kernel=3,stride=1,padding=1)->LReLU->Conv-3->LReLU

[0067] Optionally, R_f and R_s can be concat-ed on the channel to obtain R_c(2*C,H,W). Then, feature R_o is obtained by performing Conv(in_channel=2*C,out_channel=C,kernel=3,stride=1,padding=1). The output feature is R_o+R_in.

[0068] In some embodiments, the first characteristic information includes a first amplitude characteristic and a first phase characteristic, and the second characteristic information includes a second amplitude characteristic and a second phase characteristic.

[0069] like Figure 4 As shown, step S2 specifically includes:

[0070] Step S21 : fusing the first amplitude feature and the second amplitude feature based on a preset fusion algorithm to obtain a fused amplitude feature.

[0071] Step S22: fusing the first phase feature and the second phase feature based on a fusion algorithm to obtain a fused phase feature.

[0072] Step S23: Perform inverse Fourier transform on the fused amplitude feature and the fused phase feature to obtain a fused feature.

[0073] In this embodiment, the feature fusion model may be a Fourier cross-attention fusion network, where the cross-attention network is represented by the following formula:

[0074]

[0075] Where Q represents the first feature information, and K and V represent the second feature information. Q, K, and V are all 3x64x64. Finally, an inverse Fourier transform is performed on the corrected amplitude and phase to obtain the fused feature.

[0076] There is no limit to the type of fusion algorithm, and a reasonable selection can be made according to actual application needs. For example, it can be one of a linear fusion algorithm, a weighted fusion algorithm, and a typical association analysis algorithm.

[0077] Specifically, the image to be adjusted is represented as I_in; the preset properly exposed image is represented as I_ref; the first feature information is represented as R_in = E(I_in)(3, H / 8, W / 8); and the second feature information is represented as R_ref = E(I_ref)(3, H / 8, W / 8). Through FFT and Amp. / Pha. calculations, R_in and R_ref are converted to A_in, P_in, A_ref, R_ref(4, H / 8, W / 8).

[0078] The first amplitude feature, the second amplitude feature, the first phase feature and the second phase feature are respectively obtained by Conv(kernel=1, stride=1, padding=0)->LReLU

[0079] ->Conv(kernel=1,stride=1,padding=0) network is encoded according to the formula:

[0080]

[0081] The fused amplitude features and fused phase features are obtained respectively, and finally the fused features are output through inverse Fourier calculation.

[0082] In some embodiments, the diffusion network model includes a forward propagation network and a backpropagation network.

[0083] like Figure 5 As shown, step S3 specifically includes:

[0084] In step S31, the fused feature information is input into a forward propagation network, so that the forward propagation network adds noise to the image to be adjusted according to the fused feature information to obtain a noisy image.

[0085] In step S32, the noisy image is input into a back propagation network, and the back propagation network performs denoising on the noisy image according to a preset denoising model to obtain an adjusted image.

[0086] In this embodiment, the diffusion network model may be a Fourier diffusion network model, wherein the forward propagation network gradually adds noise signals to the image based on a Markov chain, and learns the distribution of the image by adding noise to the image. The forward propagation network is represented by the following formula:

[0087]

[0088] Among them, β is a predefined sequence used to control time sampling.

[0089] There is no limit to the types of noise reduction models, and they can be reasonably selected according to actual application needs.

[0090] In some optional embodiments, the back propagation model adopts a UNet model. The back propagation network is used to reduce the noise of the noisy image generated by the forward propagation network, thereby correcting the fused features, and then decoding the corrected fused features to obtain an adjusted image with normal exposure. The back propagation network is represented by the following formula:

[0091] In some optional embodiments, the diffusion network model further includes a decoder, which decodes the corrected fusion features within the diffusion network model so that the diffusion network model directly outputs an adjusted image corresponding to the image to be adjusted, thereby simplifying the image output step.

[0092] It can be seen from the above embodiments of the present application that the present application extracts features from the image to be adjusted, and fuses the first feature information extracted from the image to be adjusted with the preset second feature information, thereby enriching the texture information of the image to be adjusted, and then uses the diffusion network to adjust the fused features and output an adjusted image with accurate exposure, thereby maximizing the use of the existing information in the image to be adjusted and allowing the output adjusted image to retain more detail information.

[0093] In some optional embodiments, such as Figure 6 As shown, the training method of the image exposure correction model includes:

[0094] Step S10: Initialize the image exposure correction model, including the pre-trained Fourier feature extraction network, the Fourier cross attention fusion network, and the diffusion model network improved based on the mask Fourier prior.

[0095] In this example, an initialization image exposure correction model can be built based on the deep learning framework PyTorch.

[0096] Step S20: obtaining a large number of images captured by a camera in various scenes as training samples, where the samples consist of original images and their corresponding normally exposed images.

[0097] Image samples are collected by cameras. To ensure the robustness and accuracy of the image exposure correction model, a large number of image samples must be collected under different lighting conditions and in a wide range of scenes. Training provides pairs of samples with both abnormal and normal exposures, which is a form of supervised learning.

[0098] In step S30, the abnormal exposure image in the training sample is input into the Fourier feature extraction network, and a preset convolution operation and frequency domain processing based on a mask prior are performed on the abnormal exposure image, thereby mapping the abnormal exposure image to the Fourier coding space to obtain a feature image of a fixed scale.

[0099] In step S40, the feature image of the fixed scale is subjected to a Fourier feature extraction network, and the feature image of another normally exposed image is subjected to feature fusion using a Fourier cross attention fusion network to obtain fused feature information.

[0100] In step S50, the fused feature information is input as conditional information together with the noise sampled from the normal space into the Fourier diffusion model network, and the corrected features are obtained through the back propagation process of the diffusion model improved based on the masked Fourier prior.

[0101] Step S60: Input the corrected features into the decoder of the Fourier feature extraction network to obtain an exposure-corrected image.

[0102] Step S70: Based on the difference between the corrected image output by the image exposure correction network and the corresponding normally exposed image, the parameters of the Fourier feature extraction network, the Fourier attention network, and the Fourier diffusion model network are adjusted. When the number of iterations reaches a preset number, training is terminated, resulting in a trained image exposure correction model.

[0103] The model can be trained using a two-stage training method. The first stage does not involve the training of the diffusion model. This stage guides the training of the model through image reconstruction loss and frequency loss, where for the paired abnormal exposure image I in and normal exposure image I gt , the reconstruction loss is

[0104]

[0105] in and ε are the decoder and encoder described in S201 and S204. The frequency loss guides model training by controlling the effect of the attention mechanism:

[0106]

[0107] in This is the result obtained after the abnormal exposure image and the normal exposure image are fused through the Fourier cross attention network.

[0108] The second stage of training the diffusion model includes diffusion loss and consistency loss. Diffusion loss is

[0109]

[0110] In order to ensure the accuracy of image restoration, the output features The consistency loss is

[0111]

[0112] in is the characteristic of the normal exposure image (i.e., ε(I gt ))

[0113] When the number of iterations reaches the preset number, it indicates that the image exposure correction model has achieved good feature extraction and image restoration capabilities. The preset number of iterations for the first stage can be 48,000 or 50,000, and the preset number of iterations for the second stage can be 52,000, depending on the specific scenario.

[0114] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied in a distributed scenario and performed by multiple devices working together. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the method.

[0115] It should be noted that the above description is limited to some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that described in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0116] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides an exposure correction device in a video Internet of Things environment.

[0117] refer to Figure 7 , an exposure correction device in a video Internet of Things environment, comprising:

[0118] The feature extraction module 100 is used to input the image to be adjusted into a pre-trained feature extraction model, so that the feature extraction model outputs first feature information corresponding to the image to be adjusted.

[0119] The feature fusion module 200 is configured to input the first feature information and the preset second feature information into a pre-trained feature fusion model, so that the feature fusion model outputs a fusion feature. The second feature information is feature information output by the feature extraction model corresponding to the preset normal exposure image.

[0120] The image adjustment module 300 is used to input the fusion features into a pre-trained diffusion network model, so that the diffusion network model adjusts the exposure of the image to be adjusted to a normal value and outputs the adjusted image.

[0121] In some embodiments, the first feature information includes original image features and downsampled image features.

[0122] The feature extraction module 100 specifically includes:

[0123] The downsampling module is used to downsample the image to be adjusted at least once to obtain a downsampled image.

[0124] The first image extraction module is used to extract features from the image to be adjusted to obtain features of the original image.

[0125] The second image extraction module is used to extract features from the downsampled image to obtain downsampled image features.

[0126] In some embodiments, the first image extraction module specifically includes:

[0127] The feature calculation module is used to calculate the first amplitude information and the first phase information of the image to be adjusted.

[0128] The mask adjustment module is used to set the first mask and the second mask, wherein the first mask and the second mask are determined by the following formula:

[0129] X′=iFFT(A(x)*M+P(X)*(1-M))

[0130] Where X′ represents the mask Fourier prior of the first mask and the second mask. A(x) represents the first amplitude information. P(x) represents the first phase information. M represents the first mask. (1-M) represents the second mask.

[0131] The feature adjustment module is configured to calculate the second amplitude information based on the first mask and the first amplitude information, and calculate the second phase information based on the second mask and the first phase information.

[0132] The inverse Fourier module is used to perform inverse Fourier calculation on the second amplitude information and the second phase information to obtain the original image features.

[0133] For the convenience of description, the above devices are described as being divided into various modules according to their functions. Of course, when implementing this application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0134] The device of the above embodiment is used to implement a corresponding exposure correction method in a video Internet of Things environment in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0135] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, it implements an exposure correction method in a video Internet of Things environment as described in any of the above embodiments.

[0136] Figure 910 is a schematic diagram showing a more specific hardware structure of an electronic device provided in this embodiment. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are communicatively connected to each other within the device via the bus 1050.

[0137] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0138] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0139] The input / output interface 1030 is used to connect input / output modules to implement information input and output. The input / output modules can be configured as components within the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0140] The communication interface 1040 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0141] The bus 1050 comprises a path for transmitting information between the various components of the device (eg, the processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 ).

[0142] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may only include the components necessary to implement the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0143] The electronic device of the above embodiment is used to implement a corresponding exposure correction method in a video Internet of Things environment in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0144] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute an exposure correction method in a video Internet of Things environment as described in any of the above embodiments.

[0145] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0146] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute an exposure correction method in a video Internet of Things environment as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0147] As a more specific implementation method, Figure 8 As shown, the exposure correction method in a video IoT environment provided by this application can also be performed by the following steps:

[0148] The abnormal exposure image is input into the Fourier feature extraction network, and a preset convolution operation and frequency domain processing based on mask prior are performed on the abnormal exposure image, so as to map the abnormal exposure image to the Fourier coding space and obtain a feature image of fixed scale.

[0149] Existing methods are based on the naive Fourier prior. First, the Fourier transform of the image domain, also known as the 2D-Fourier transform, is calculated using the following formula:

[0150]

[0151] The image x(h,w) with a resolution of (H,W) is converted to the frequency domain. The resulting frequency domain image X(u,v) still has a resolution of (H,W).

[0152] In addition, the obtained frequency domain image X can be split into the real part R and the imaginary part I. Through the real part R and the imaginary part I, the amplitude A(X) and phase P(X) of the image can be calculated:

[0153]

[0154] In the field of exposure correction, existing work has demonstrated that the image amplitude A(X) primarily contains luminosity information, while the phase R(X) primarily contains texture information. This work demonstrates this by swapping the amplitudes of paired underexposed and overexposed images, which alters the image's luminosity. Visualizing the phases of these paired images reveals that the phases contain a significant amount of texture information. However, this proof only considers the role of amplitude in this swapping, without exploring the phase.

[0155] Inspired by the conclusions of the 2D Fourier transform (the low frequencies of the frequency-domain image X(u,v) represent large-scale information, such as luminosity, while the high frequencies contain detailed information, such as texture), similar conclusions can be drawn from studying amplitude and phase. Using a method similar to existing techniques, but exchanging only the low-frequency components of the amplitude, the resulting image is superior (as demonstrated using the PSNR metric. A higher PSNR indicates greater similarity between two images. The PSNR between the image after exchanging the low-frequency components of the amplitude and the target image is superior to exchanging the entire amplitude). The same proof is provided for phase.

[0156] Based on the preference of amplitude and phase, combined with the fact that the existing prior only exchanges amplitude but not phase, it is easy to derive a new prior: complementary masks can be used to guide the amplitude and phase to take low-frequency and high-frequency content respectively, which is called the masked Fourier prior.

[0157] The use of masked Fourier prior has the following advantages; first, it has good scalability. It is easy to find that the naive Fourier prior is actually a special form of masked Fourier transform. When only the amplitude is exchanged, the amplitude and phase masks are still complementary. For the convenience of expression, all frequency domain information is defined here as central low-frequency (that is, the center of the frequency domain image is low-frequency information, and the frequency becomes higher when the frequency band expands outward). The mask ratio value α is defined to control the ratio of the mask in the frequency domain (starting from frequency 0 to the highest frequency f_max of the frequency domain image, the information of frequency [0, f_max*α) is controlled by the mask). The corresponding mask is defined as M. Masks M and 1-M are applied to the amplitude A(X) and phase P(X) respectively. The masked Fourier prior can be written as:

[0158] X′=iFFT(A(x)*M+P(X)*(1-M))

[0159] Second, the masked Fourier prior can better decouple amplitude (luminosity) and phase (texture) information. Using different mask ratio values ​​α can improve the completeness of the masked Fourier transform decoupling, and choosing an appropriate α value can achieve a more complete image decoupling. Experiments based on 1500 underexposed and overexposed images show that α = 0.04 is a good mask ratio value, achieving a higher degree of decoupling than the naive Fourier prior (α = 0).

[0160] Using a masked Fourier prior can assist in image encoding. However, using fixed empirical values ​​cannot cover all cases. Existing work has demonstrated that convolution is a low-frequency filter. After convolution, low-frequency information in an image is gradually removed, leaving mostly high-frequency components. Based on these considerations, a low-to-high-frequency filtering method can be designed. Furthermore, the mask ratio value should be image-dependent (learnable).

[0161] The resulting Fourier feature extraction network first estimates n mask ratio values ​​α of the image (corresponding to n Fourier feature processing modules in the network):

[0162] [d1,d2,...,d n ]=Softplus(MLP(C3(C3(I))))

[0163] Where C3 represents the convolution operation with a convolution kernel of 3, MLP is a multi-layer perceptron, and Softplus is the perceptual function, which is defined as:

[0164] Softplus(x)=log(1+e x )

[0165] The obtained sequence d is then regularized to obtain

[0166]

[0167] Δ i By performing the accumulation operation, we can obtain the monotonically increasing sequence of α

[0168]

[0169] Each Fourier feature processing module will obtain the mask ratio value α i and the corresponding input features as input, and after decomposing the features into amplitude and frequency domains, they are processed in three ways.

[0170] After several Fourier feature processing modules and downsampling, fixed-size features are obtained.

[0171] The number of Fourier feature processing modules is set to 3. The input image resolution is 3x512x512, and the downsampling ratio is 2 each time. After three downsamplings, the resulting feature resolution is 3x64x64.

[0172] The Fourier cross attention fusion network combines the fixed-scale feature image with the feature image obtained after the Fourier feature extraction network of another normal exposure image, and uses the Fourier cross attention fusion network to perform feature fusion to obtain fused feature information;

[0173] Fixed-size features often lose texture details due to exposure. Furthermore, neural networks tend to learn to directly restore the network's luminosity, but are less capable of processing detail and color. In a Fourier cross-attention fusion network, a properly exposed image is introduced as a reference image. Using the cross-attention mechanism, amplitude and phase are modified in the frequency domain, guiding the model to learn texture and correctly restore luminosity and color.

[0174] After processing the normally exposed image, the fixed-size features of the normally exposed image are obtained. Then, the abnormal / normally exposed images are decomposed into amplitude and phase components by Fourier transform, and two groups of cross-attention mechanisms are used to recover the amplitude and phase components. Specifically, the cross-attention mechanism is:

[0175]

[0176] Where Q is the amplitude or phase of the non-normally exposed image; K and V are the amplitude or phase of the normally exposed image. Q, K, and V are all 3x64x64. Finally, an inverse Fourier transform is performed on the corrected amplitude and phase to obtain the fused feature information.

[0177] Based on the diffusion model network improved by masked Fourier prior, the fused feature information is input as conditional information together with the noise sampled from the normal space into the Fourier diffusion model network. The corrected features are obtained through the back propagation process of the diffusion model improved by masked Fourier prior.

[0178] The fused feature information above initially fixes the luminosity and texture information, but there may still be some luminosity and texture errors. In the diffusion model, the forward process describes the process of gradually adding noise to the image until the model learns a latent space distribution:

[0179]

[0180] Among them, β is a predefined sequence used to control time sampling.

[0181] In order to further refine the low-frequency brightness and reconstruct the high-frequency structural information, we observed that in the diffusion model denoising process, the recovery of high-frequency and low-frequency information exhibits a time-step-dependent characteristic: low-frequency information is preferentially recovered at earlier time steps, while high-frequency information is preferentially recovered at later time steps. Therefore, we use the change of time steps to guide the learning of frequency-specific information. We use the following formula to fix the condition Modified to be time-related Specifically, at each time step t, by generating a mask ratio value α=t / T based on the time step, and then deriving the mask M and 1-M applied to the fixed condition to modify the conditions related to the time step.

[0182] The reverse process of the diffusion model is modified as follows:

[0183]

[0184] As an optional implementation, the number of time steps T in the network is 2000, the sampling strategy for β is uniform sampling, and backpropagation uses a jump sampling strategy with a time step of 40 per jump. The noise estimator used in the network is Unet, and the conditional introduction method is concat.

[0185] The corrected features are input into the decoder of the Fourier feature extraction network to obtain the exposure-corrected image;

[0186] In one embodiment, the decoder is similar to the Fourier feature extraction network structure, except that downsampling is changed to upsampling. The final image resolution is 3x512x512.

[0187] It should be noted that the embodiments of the present application can be further described in the following manner:

[0188] A method for exposure correction in a video Internet of Things environment, comprising:

[0189] The image to be adjusted is input into a pre-trained feature extraction model, so that the feature extraction model outputs first feature information corresponding to the image to be adjusted.

[0190] The first feature information and the preset second feature information are input into a pre-trained feature fusion model so that the feature fusion model outputs a fusion feature, wherein the second feature information is feature information output by the feature extraction model and corresponds to the preset normal exposure image.

[0191] The fused features are input into a pre-trained diffusion network model, so that the diffusion network model adjusts the exposure of the image to be adjusted to a normal value and outputs the adjusted image.

[0192] Optionally, the first feature information includes original image features and downsampled image features.

[0193] Inputting the image to be adjusted into a pre-trained feature extraction model so that the feature extraction model outputs first feature information corresponding to the image to be adjusted, specifically comprising:

[0194] The image to be adjusted is downsampled at least once to obtain a downsampled image.

[0195] Perform feature extraction on the image to be adjusted to obtain the original image features.

[0196] Perform feature extraction on the downsampled image to obtain downsampled image features.

[0197] Optionally, feature extraction is performed on the image to be adjusted to obtain features of the original image, specifically including:

[0198] Calculate first amplitude information and first phase information of the image to be adjusted.

[0199] Set the first mask and the second mask, where the first mask and the second mask are determined by the following formula:

[0200] X′=iFFT(A(x)*M+P(X)*(1-M))

[0201] Where X′ represents the mask Fourier prior of the first mask and the second mask. A(x) represents the first amplitude information. P(x) represents the first phase information. M represents the first mask. (1-M) represents the second mask.

[0202] The second amplitude information is calculated according to the first mask and the first amplitude information. The second phase information is calculated according to the second mask and the first phase information.

[0203] Perform inverse Fourier transformation on the second amplitude information and the second phase information to obtain original image features.

[0204] Optionally, the first characteristic information includes a first amplitude characteristic and a first phase characteristic, and the second characteristic information includes a second amplitude characteristic and a second phase characteristic.

[0205] Inputting the first feature information and the preset second feature information into a pre-trained feature fusion model so that the feature fusion model outputs a fusion feature, specifically including:

[0206] The first amplitude feature and the second amplitude feature are fused based on a preset fusion algorithm to obtain a fused amplitude feature.

[0207] The first phase feature and the second phase feature are fused based on a fusion algorithm to obtain a fused phase feature.

[0208] Perform inverse Fourier transform on the fused amplitude feature and the fused phase feature to obtain the fused feature.

[0209] Optionally, the diffusion network model includes a forward propagation network and a backpropagation network.

[0210] The fused features are input into the pre-trained diffusion network model so that the diffusion network model adjusts the exposure of the image to be adjusted to a normal value and outputs the adjusted image, specifically including:

[0211] The fused feature information is input into the forward propagation network, so that the forward propagation network adds noise to the image to be adjusted according to the fused feature information to obtain a noisy image.

[0212] The noisy image is input into the back propagation network, and the back propagation network reduces the noise image according to the preset noise reduction model to obtain an adjusted image.

[0213] This application also provides an exposure correction device in a video IoT environment, comprising:

[0214] The feature extraction module is used to input the image to be adjusted into a pre-trained feature extraction model so that the feature extraction model outputs first feature information corresponding to the image to be adjusted.

[0215] The feature fusion module is configured to input the first feature information and the preset second feature information into a pre-trained feature fusion model, so that the feature fusion model outputs a fusion feature. The second feature information is feature information output by the feature extraction model corresponding to the preset normal exposure image.

[0216] The image adjustment module is used to input the fusion features into a pre-trained diffusion network model so that the diffusion network model adjusts the exposure of the image to be adjusted to a normal value and outputs the adjusted image.

[0217] Optionally, the first feature information includes original image features and downsampled image features.

[0218] The feature extraction module specifically includes:

[0219] The downsampling module is used to downsample the image to be adjusted at least once to obtain a downsampled image.

[0220] The first image extraction module is used to extract features from the image to be adjusted to obtain features of the original image.

[0221] The second image extraction module is used to extract features from the downsampled image to obtain downsampled image features.

[0222] Optionally, the first image extraction module specifically includes:

[0223] The feature calculation module is used to calculate the first amplitude information and the first phase information of the image to be adjusted.

[0224] The mask adjustment module is used to set the first mask and the second mask, wherein the first mask and the second mask are determined by the following formula:

[0225] X′=iFFT(A(x)*M+P(X)*(1-M))

[0226] Where X′ represents the mask Fourier prior of the first mask and the second mask. A(x) represents the first amplitude information. P(x) represents the first phase information. M represents the first mask. (1-M) represents the second mask.

[0227] The feature adjustment module is configured to calculate the second amplitude information based on the first mask and the first amplitude information, and calculate the second phase information based on the second mask and the first phase information.

[0228] The inverse Fourier module is used to perform inverse Fourier calculation on the second amplitude information and the second phase information to obtain the original image features.

[0229] The present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any one of the above methods when executing the program.

[0230] The present application also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute any of the above methods.

[0231] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. Within the scope of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0232] In addition, for simplicity of description and discussion, and in order not to make the embodiment of the application difficult to understand, the known power supply / ground connection with integrated circuit (IC) chip and other components may or may not be shown in the accompanying drawings provided. In addition, the device can be shown in the form of a block diagram to avoid making the embodiment of the application difficult to understand, and this also takes into account the following fact, that is, the details of the embodiment of these block diagram devices are highly dependent on the platform to be implemented in the embodiment of the application (that is, these details should be fully within the scope of understanding of those skilled in the art). When specific details (for example, circuit) are set forth to describe exemplary embodiments of the application, it will be apparent to those skilled in the art that the embodiment of the application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0233] Although the present invention has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may utilize the embodiments discussed.

[0234] The embodiments of the present application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of this application.

Claims

1. A method for exposure correction in a video IoT environment, comprising: Inputting the image to be adjusted into a pre-trained feature extraction model so that the feature extraction model outputs first feature information corresponding to the image to be adjusted; Inputting the first feature information and preset second feature information into a pre-trained feature fusion model, so that the feature fusion model outputs a fusion feature; wherein the second feature information is feature information output by the feature extraction model and corresponds to a preset normal exposure image; The fused features are input into a pre-trained diffusion network model, so that the diffusion network model adjusts the exposure of the image to be adjusted to a normal value and outputs an adjusted image.

2. The exposure correction method in a video IoT environment according to claim 1, wherein: The first feature information includes original image features and downsampled image features; Inputting the image to be adjusted into a pre-trained feature extraction model so that the feature extraction model outputs first feature information corresponding to the image to be adjusted specifically includes: Downsampling the image to be adjusted at least once to obtain a downsampled image; Performing feature extraction on the image to be adjusted to obtain the original image features; Perform feature extraction on the downsampled image to obtain the downsampled image features.

3. The exposure correction method in a video IoT environment according to claim 2, wherein: Extracting features of the image to be adjusted to obtain the features of the original image specifically includes: Calculating first amplitude information and first phase information of the image to be adjusted; Set a first mask and a second mask, wherein the first mask and the second mask are determined by the following formula: X′=iFFT(A(x)*M+P(X)*(1-M)) Wherein, X′ represents the mask Fourier prior of the first mask and the second mask; A(x) represents the first amplitude information; P(X) represents the first phase information; M represents the first mask; (1-M) represents the second mask; Calculating second amplitude information according to the first mask and the first amplitude information; calculating second phase information according to the second mask and the first phase information; Perform inverse Fourier calculation on the second amplitude information and the second phase information to obtain the original image feature.

4. The exposure correction method in a video IoT environment according to claim 1, wherein: The first characteristic information includes a first amplitude characteristic and a first phase characteristic, and the second characteristic information includes a second amplitude characteristic and a second phase characteristic; Inputting the first feature information and the preset second feature information into a pre-trained feature fusion model so that the feature fusion model outputs a fusion feature specifically includes: fusing the first amplitude feature and the second amplitude feature based on a preset fusion algorithm to obtain a fused amplitude feature; fusing the first phase feature and the second phase feature based on the fusion algorithm to obtain a fused phase feature; An inverse Fourier transform is performed on the fused amplitude feature and the fused phase feature to obtain the fused feature.

5. The exposure correction method in a video IoT environment according to claim 1, wherein: The diffusion network model includes a forward propagation network and a back propagation network; Inputting the fused features into a pre-trained diffusion network model so that the diffusion network model adjusts the exposure of the image to be adjusted to a normal value and outputs the adjusted image specifically includes: Inputting the fused feature information into the forward propagation network, so that the forward propagation network adds noise to the image to be adjusted according to the fused feature information to obtain a noisy image; The noisy image is input into a back propagation network, and the back propagation network performs denoising on the noisy image according to a preset denoising model to obtain the adjusted image.

6. An exposure correction device for a video IoT environment, comprising: a feature extraction module, configured to input the image to be adjusted into a pre-trained feature extraction model, so that the feature extraction model outputs first feature information corresponding to the image to be adjusted; a feature fusion module, configured to input the first feature information and preset second feature information into a pre-trained feature fusion model, so that the feature fusion model outputs a fusion feature; wherein the second feature information is feature information output by the feature extraction model and corresponds to a preset normal exposure image; The image adjustment module is used to input the fusion feature into a pre-trained diffusion network model so that the diffusion network model adjusts the exposure of the image to be adjusted to a normal value and outputs the adjusted image.

7. The exposure correction device in a video IoT environment according to claim 6, wherein: The first feature information includes original image features and downsampled image features; The feature extraction module specifically includes: A downsampling module, configured to downsample the image to be adjusted at least once to obtain a downsampled image; A first image extraction module is used to extract features of the image to be adjusted to obtain features of the original image; The second image extraction module is used to perform feature extraction on the downsampled image to obtain the features of the downsampled image.

8. The exposure correction device in a video IoT environment according to claim 7, wherein: The first image extraction module specifically includes: a feature calculation module, configured to calculate first amplitude information and first phase information of the image to be adjusted; The mask adjustment module is used to set a first mask and a second mask, wherein the first mask and the second mask are determined by the following formula: X′=iFFT(A(x)*M+P(X)*(1-M)) Wherein, X′ represents the mask Fourier prior of the first mask and the second mask; A(x) represents the first amplitude information; P(X) represents the first phase information; M represents the first mask; (1-M) represents the second mask; a feature adjustment module, configured to calculate second amplitude information based on the first mask and the first amplitude information; and calculate second phase information based on the second mask and the first phase information; An inverse Fourier module is used to perform inverse Fourier calculation on the second amplitude information and the second phase information to obtain the original image features.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 5 when executing the program. 10 . A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are configured to cause a computer to execute the method according to claim 1 .

Images