Arbitrary frame low-light video enhancement method based on autoregression space-time interaction

Abstract

The invention relates to an arbitrary frame low-light video enhancement method based on autoregression space-time interaction, and belongs to the field of computer vision processing of images and videos. The method comprises the following steps: preprocessing input video data to obtain a training data set; designing a low-illumination video enhancement network which comprises a spatial feature encoder, an autoregression-based space-time interaction enhancement module and a spatial feature decoder; designing a loss function for training the designed network, and training by using the training data set to obtain a trained low-illumination video enhancement model; and inputting a to-be-measured low-illumination video into the obtained model to generate a normal-illumination video. According to the method, an autoregressive space-time interaction modeling method is utilized, and on the basis of keeping the original space processing capability of an image model based on an encoder and decoder architecture, the processing capability of the model on time sequence information is enhanced, so that the model can better enhance the brightness of a video and improve the time consistency under a low-illumination condition, and the video quality is improved. And the video enhancement effect is further improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of computer vision processing of images and videos, and particularly relates to an arbitrary frame low-light video enhancement method based on autoregressive spatio-temporal interaction. BACKGROUND

[0002] With the continuous progress of technology and the popularity of video acquisition devices, the application demand of low-light video data in the fields of security monitoring, autonomous driving, medical endoscopy, etc. is growing rapidly. In complex lighting environments such as night, tunnel or weak indoor light, the traditional video acquisition system is limited by the physical characteristics of the sensor, often resulting in insufficient brightness, significant noise, dynamic range compression, etc., leading to phenomena such as temporal flicker, motion blur and color distortion in the video. Unlike single-frame image enhancement, low-light video enhancement needs to simultaneously solve the dual challenges of spatial quality improvement and temporal consistency maintenance, which puts higher requirements on algorithm design. Existing methods are mainly divided into two categories: multi-frame fusion based on traditional optimization and end-to-end enhancement based on deep learning, but the former is sensitive to motion and prone to artifacts, and the latter often ignores the inter-frame correlation, resulting in jitter in the enhancement results.

[0003] The application value of low-light video enhancement technology is increasingly prominent in multiple key fields. In the field of intelligent security, the enhancement effect of night monitoring video in the city is directly related to the accuracy of abnormal behavior detection and suspect identification. Although existing algorithms based on single-frame processing can improve the quality of single images, they ignore the temporal characteristics of the video, resulting in identity jump when tracking targets. In the medical imaging field, real-time enhancement of endoscopic surgery video needs to balance detail restoration and motion coherence. Traditional methods often introduce non-physiological flicker when enhancing tissue texture, affecting the surgeon's intraoperative decision-making. More notably, when driving in tunnels or at night, the low-light video captured by the vehicle-mounted camera of the autonomous driving system may only rely on frame-by-frame enhancement, which can lead to deviations in the trajectory prediction of moving objects by the perception module. These scenarios collectively reveal the key bottleneck of current technology: independent processing of video frames can destroy the spatio-temporal continuity, while simple temporal smoothing can lose dynamic details.

[0004] In recent years, deep learning has revolutionized the paradigm of low-light video enhancement technology. From the early 3D convolution-based coarse-grained spatio-temporal processing to the current Transformer-based fine-grained interaction method. The mainstream technology route faces three dilemmas: local temporal modeling based on sliding windows is difficult to capture long-range dependencies, cross-frame alignment guided by optical flow has high computational complexity, and pure temporal convolutional networks are limited by the dynamic adaptability of fixed receptive fields. SUMMARY

[0005] The purpose of the present application is to provide an arbitrary frame low-light video enhancement method based on autoregressive spatio-temporal interaction, which can realize the coherent enhancement of video sequences of any length by constructing a dynamic memory query mechanism and a spatio-temporal feature fusion strategy, effectively solving the time consistency fault problem existing in the traditional method in long video processing; through the autoregressive memory query enhancement block and the memory bank mechanism, the current frame can adaptively retrieve and fuse the historical key frame features, ensuring the temporal coherence of the enhancement result, and at the same time, the spatio-temporal interaction attention block can dynamically integrate the most relevant memory features into the current spatial features, such an autoregressive spatio-temporal interaction strategy not only retains the spatial details of single-frame images when enhancing low-illumination videos, but also significantly improves the temporal stability of video sequences. The present application innovatively combines the autoregressive modeling capability of Transformer with the spatial feature extraction advantage of the encoder-decoder architecture, realizes the elegant expansion of the image model to the video model based on the encoder-decoder architecture under the premise of minimizing the model modification, and provides an efficient and reliable solution for low-illumination video enhancement in practical applications.

[0006] To achieve the above purpose, the technical scheme of the present application is: an arbitrary frame low-light video enhancement method based on autoregressive spatio-temporal interaction, comprising the following steps:

[0007] Step A, pre-processing the input video data, including data pairing, data sampling, data cropping and data enhancement processing, to obtain a training data set;

[0008] Step B, designing a low-illumination video enhancement network, including a spatial feature encoder, an autoregressive-based spatio-temporal interaction enhancement module and a spatial feature decoder;

[0009] Step C, designing a loss function for training the network designed in step B, and training the network in step B using the training data set in step A to obtain a trained low-illumination video enhancement model;

[0010] Step D, inputting the low-illumination video to be tested into the model obtained in step C to generate a normal-illumination video.

[0011] Further, step A is implemented as follows:

[0012] Step A1, pairing the normal-illumination video and the corresponding low-illumination video in the data set, wherein the normal-illumination video is used as the label video; then all video frames are uniformly scaled to WxH resolution, where W is the width and H is the height;

[0013] Step A2, randomly select a continuous video segment with a length of L from each low-illumination video, and at the same time, obtain a continuous video segment with a length of L corresponding to the normal-illumination video in the same way to ensure that they correspond to each other in the time dimension, to obtain paired low-illumination video segments and normal-illumination video segments with a length of L; repeat the operation of each paired low-illumination video and normal-illumination video several times; repeat the operation of all paired low-illumination videos and normal-illumination videos to obtain video segments with a size of LxHxWx3, wherein L is the number of video frames in each video segment in the training data set;

[0014] Step A3, randomly crop the paired low-illumination video segments and normal-illumination video segments with a length of L into paired video segments with a size of LxPxPx3 in a consistent manner, wherein P is the height and width of the cropped video frame;

[0015] Step A4, for each paired video, randomly apply one of the following four data enhancement methods: keep the original video frame, vertical flip, horizontal flip, and vertical flip followed by horizontal flip; finally, obtain the paired low-illumination video and normal-illumination video training data set.

[0016] Further, step B is implemented as follows:

[0017] Step B1, design a spatial feature encoder, use the encoder of the UNet backbone network as the spatial feature encoder, and perform feature extraction on the input low-illumination video frame to obtain the extracted feature wherein P is the height and width of the low-illumination video frame, h, w, and d are the height, width, and channel number of the low-illumination video frame after feature extraction by the spatial feature encoder, and subscript i represents the serial number of the video frame;

[0018] Step B2, design an autoregressive-based spatiotemporal interaction enhancement module, which consists of an input embedding module, an autoregressive memory query enhancement block, an output embedding module, and a memory bank, for the feature I i extracted in step B1, through the autoregressive-based spatiotemporal interaction enhancement module, to obtain the spatiotemporal interaction enhanced feature Then, the spatiotemporal interaction enhanced feature is connected in residual to the feature I i extracted in step B1 to obtain the comprehensive spatiotemporal interaction enhanced feature The entire process is represented by the formula:

[0019]

[0020] wherein ARSTEM represents the autoregressive-based spatiotemporal interaction enhancement module;

[0021] Step B3, design a spatial feature decoder, adopt the decoder of UNet backbone network as the spatial feature decoder, and the integrated spatio-temporal interaction enhanced feature obtained in step B2 is input into the spatial feature decoder as the input of the spatial feature decoder, and the output is the enhanced video frame

[0022] Further, step B2 is specifically implemented as follows:

[0023] Step B21, design an input embedding module, which sequentially comprises a depth-wise convolution unit, a point-wise convolution unit and a normalization reshaping unit; wherein the depth-wise convolution unit is composed of a convolution layer with a convolution kernel size of 10x10, a step of 7, a padding of 3 and a group number of input feature channel number d, the point-wise convolution unit is composed of a convolution layer with a convolution kernel size of 1x1, a step of 1, a padding of 0 and a group number of 1, and the normalization reshaping unit comprises three learnable parameters a, ω and β and a reshaping unit, and the reshaping unit reshapes the feature with a dimension of 3 into a feature with a dimension of 2; the input embedding module is used for feature embedding processing of , and specifically, I i is input into the depth-wise convolution unit and the point-wise convolution unit in sequence to obtain intermediate feature is input into the normalization reshaping unit for processing, wherein the reshaping unit flattens the feature to obtain embedded feature wherein r represents the reduction rate of the feature after the depth-wise convolution unit processing, represents the channel number of the embedded feature; the whole process is represented by the formula:

[0024]

[0025] wherein, DWConv represents the depth-wise convolution unit, PWConv represents the point-wise convolution unit, a, ω and β are learnable parameters, represents element-wise addition, represents element-wise multiplication, and Reshape(·) is the operation of the reshaping unit, which reshapes the feature with a size of to a feature with a size of ;

[0026] Step B22, design a memory bank, which is used to store memory features, and the initial memory feature corresponding to the i-th frame in the memory bank is which is a zero feature, and m represents the dimension value of the first dimension of the memory feature; the low-light video enhancement network in step B contains historical memory features Z i-1 ,…,Z0in the memory bank before processing the i-th frame, wherein ​k e {0, 1, …, i-1} ;

[0027] Step B23, design an autoregressive memory query enhancement block, which is composed of a learnable query position encoding, an absolute position encoding, a memory query attention block, a spatiotemporal interaction attention block, and a multilayer perception memory update block; wherein the embedding feature X i obtained in step B21 is processed by n stacked autoregressive memory query enhancement blocks to output a query enhancement feature and a current memory feature , and the current memory feature Z i is appended to the memory bank, and the whole process is represented by the formula:

[0028]

[0029] wherein ARQB l represents the lth autoregressive memory query enhancement block, l e {0, 1, …, n, n is the number of stacked autoregressive memory query enhancement blocks, and the symbol represents the stacking operation of the autoregressive memory query enhancement block, and Mean(·) represents the average value calculation operation along the first dimension of the feature;

[0030] Step B24, design an output embedding module, which sequentially includes a reshaping upsampling unit, a depth-wise convolution unit, and a point-wise convolution unit; wherein the reshaping upsampling unit includes a reshaping unit and a bilinear upsampling operation, the reshaping unit reshapes the feature with a dimension of 2 to a feature with a dimension of 3, and the bilinear upsampling operation up-samples the feature with a width and height of to a feature with a width and height of h x w, the depth-wise convolution unit is composed of a convolution layer with a kernel size of 3 x 3, a stride of 1, a padding of 1, and a group number of the input feature channel number , and the point-wise convolution unit is composed of a convolution layer with a kernel size of 1 x 1, a stride of 1, a padding of 0, and a group number of 1; the output embedding module is used to reconstruct the query enhancement feature obtained in step B23, specifically: using the reshaping upsampling unit to reshape and up-sample to obtain an intermediate restoration feature , and then passing through the depth-wise convolution unit and the point-wise convolution unit in sequence to obtain an output feature , and the whole process is represented by the formula:

[0031]

[0032] wherein the operation of the reshaping unit here reshapes the feature with a size of to a feature with a size of characteristics, Upsample(·) represents a bilinear up-sampling operation, and the role is to up-sample the size of characteristics to characteristics.

[0033] Further, step B23 is specifically implemented as follows:

[0034] Step B231, for the characteristics obtained by inputting the embedding module in step B21 obtain its position encoding from the absolute position encoding for the historical memory feature Z k , k e {0, 1,..., i-1}, obtain its corresponding position encoding from the absolute position encoding k e {0, 1,..., i-1}, specifically using the sinusoidal position encoding method to generate; the whole process is represented as:

[0035]

[0036] wherein the PE function represents a sinusoidal position encoding generation function of the pos frame, pos represents a time position where the historical frame or the current frame is located, τ is a temperature coefficient, the value is 10000.0, [·; ·] represents a splicing operation, and repeat(·, N) represents a replication splicing operation, that is, the size of characteristics is replicated and spliced into the size of characteristics;

[0037] The embedding feature is input into the spatio-temporal interaction attention block as a value feature, the embedding feature X i is added to the position encoding to obtain the embedding feature with position information is input into the spatio-temporal interaction attention block as a key feature; meanwhile, the historical memory features Z i-1 ,..., Z0 are input into the memory query attention block as value features, and the position encoding thereof is obtained from the absolute position encoding in the same way The historical memory features are added to the corresponding position encodings to obtain as a key feature input into the memory query attention block; the initial memory feature obtained in step B22 is taken as the stage initial memory feature Z init of the first autoregressive memory query enhancement block, and is input into the memory query attention block as a query feature.

[0038] Step B232, design the memory query attention block, which is composed of a multi-head memory cross-attention layer, a normalization layer and a residual connection; the initial current memory feature Zinit as the query feature, the history memory feature with position information as the key feature, the history memory feature Z i-1 ,…,Z0as the value feature input into the multi-head memory cross attention layer, and the obtained feature is input into the normalization layer and combined with Z init after residual connection, the memory enhanced feature is output The whole process is represented by the formula as follows:

[0039] M i = Z init + Norm(MMCA(Z init ,[Z i-1 +;…;Z0+],[Z i-1 ;…;Z0])

[0040] Wherein, MMCA represents the multi-head memory cross attention layer, and Norm(·) represents the normalization operation;

[0041] Step B233, a space-time interaction attention block is designed, which is composed of a multi-head space-time cross attention layer, a normalization layer and a residual connection; the enhanced memory feature M i is added to the learnable query position encoding to obtain M i + Q i as the query feature, the embedded feature with position information as the key feature, the embedded feature X i is input into the multi-head space-time cross attention layer, and the obtained feature is input into the normalization layer and combined with M i after residual connection, the space-time interaction feature is output The whole process is represented by the formula as follows:

[0042]

[0043] Wherein, MSTA represents the multi-head space-time cross attention layer;

[0044] Step B234, a multi-layer perception memory update block is designed, which is composed of a multi-layer perception feedforward layer and a normalization layer; the space-time interaction feature T i is processed through the multi-layer perception feedforward layer and the normalization layer, and the obtained feature is combined with the space-time interaction feature T i after residual connection, the current memory feature output by the first autoregressive memory query enhancement block is obtained The whole process is represented by the formula as follows:

[0045]

[0046] Wherein, FFN represents the multi-layer perception feedforward layer;

[0047] Step B235, the current memory feature output by the u-1th autoregressive memory query enhancement block The stage initial memory feature Z as input of the u th autoregressive memory query enhancement block init , repeating steps B232 to B234, gradually enhancing the spatiotemporal consistency feature of the video frame through the processing of the stack of n autoregressive memory query enhancement blocks, wherein u e {1, 2, …, n}; the current memory feature output by the n th autoregressive memory query enhancement block The current memory feature Z as the i th frame i , and appending the current memory feature Z i to the memory bank.

[0048] Further, step B232 is specifically implemented as follows:

[0049] Step B2321, designing a multi-head memory cross-attention layer, for the input stage initial memory feature Z init , the historical memory feature with position information and the historical memory feature Z i-1 ,…, Z0, first concatenate the historical memory feature with position information into a feature of size concatenate the historical memory feature into a feature [Z i-1 ;…; Z0] of size , wherein M = m x (i-1); then each is divided into h heads along the channel dimension, and then each head of the stage initial memory feature Z init is mapped to a “query” matrix each head of the concatenated historical memory feature with position information [Z i-1 + P i-1 ;…; Z0+ p0] is mapped to a “key” matrix each head of the concatenated historical memory feature [Z i-1 ;…; Z0] is mapped to a “value” matrix wherein j e {1, 2, …, h};

[0050] Step B2322, dividing Q mem,j by the temperature coefficient τ d , transposing K mem,j to obtain performing matrix multiplication on Q mem,j and to obtain a historical memory attention Att mem,j , calculating softmax along the channel dimension of Att mem,j ; then performing matrix multiplication on Att mem,j and Vmem,j Perform matrix multiplication to get a matrix of size The head-by-head memory interaction enhancement feature M query,j , all the head-by-head memory interaction enhancement features are spliced ​​along the channel dimension to obtain a size of Memory interaction enhancement feature M query , the whole process is expressed as follows:

[0051]

[0052] M query =[M query,1 ;M query,2 ;…;M query,h ]

[0053] in, represents matrix multiplication, softmax(·) represents the softmax operation calculated along the channel dimension, represents the temperature coefficient;

[0054] Step B2323: query the memory interaction enhancement feature M obtained by the multi-head memory cross attention layer query in step B2322 query , use linear mapping to map it to the output matrix Send it to the normalization layer, and the obtained features are combined with Z init Residual connection to obtain memory enhancement feature M i .

[0055] Furthermore, the specific implementation steps of step B233 are as follows:

[0056] Step B2331, design a multi-head spatiotemporal cross attention layer to enhance the memory feature M with position information of the input i +Q i , embedded features with position information and embedding feature X i , first split each into h heads along the channel dimension, and then use linear mapping to transform the enhanced memory feature M with position information i +Q i Each head is mapped to a "query" matrix Embedded feature X with position information i +P i Each head is mapped to a "key" matrix Embedded feature X i Each head is mapped to a "value" matrix where j∈{1,2,…,h};

[0057] Step B2332, Q st,j Divide by the temperature coefficient τ d, for K st,j Perform the transpose operation to obtain Q st,j and Perform matrix multiplication to obtain the spatiotemporal cross attention Att of size m×N st,j , for Att st,j Calculate softmax along the channel dimension; then st,j and V st,j Perform matrix multiplication to get a matrix of size The head-by-head spatiotemporal cross-enhanced feature T st,j , the spatiotemporal cross-enhancement features are spliced ​​along the channel dimension to obtain a size of The spatiotemporal cross-enhancement feature T st , the process is expressed by the formula:

[0058]

[0059] T st =[T st,1 ;T st,2 ;…;T st,h ]

[0060] in, represents matrix multiplication, softmax(·) represents the softmax operation calculated along the channel dimension, represents the temperature coefficient;

[0061] Step B2333: The spatiotemporal cross-enhanced feature T obtained by the interaction of the multi-head spatiotemporal cross-attention layer in step B2332 is st , use linear mapping to map it to the output matrix Send it to the normalization layer, and the obtained features are combined with M i Residual connection to obtain spatiotemporal interaction features T i .

[0062] Furthermore, the specific implementation steps of step C are as follows:

[0063] Step C1: Construct L1 loss and VGG perceptual loss The total loss function Its expression is:

[0064]

[0065] Among them, λ1 and λ2 are equilibrium parameters, Represents the enhanced video frame of the input i-th video frame, V gt,i represents the video frame with normal illumination corresponding to the i-th video frame, ||·|| 2Indicates mean square error calculation, VGG 3,8,15 (·) represents the 3rd, 8th, and 15th layer features extracted by the ImageNet pre-trained VGG-16 network;

[0066] Step C2: randomly divide the training dataset obtained in step A into several batches, each batch containing several pairs of video clips;

[0067] Step C3: Use a phased optimization strategy: Calculate the loss described in step C1 for the single-frame enhancement result Update the parameters of the spatial feature encoder and spatial feature decoder through backpropagation; calculate the loss of all frames for the complete video segment The parameters of the spatiotemporal interaction enhancement module based on autoregression described in step B are updated by back propagation; the Adam optimization method is used to update the network parameters;

[0068] Step C4: Repeat steps C2 to C3 until the model converges, and finally obtain a trained low-light video enhancement model.

[0069] Furthermore, the specific implementation steps of step D are as follows:

[0070] Step D1, splitting the low-light video of size F×H×W×3 into F low-light video frames of size H×W×3, where H and W are the height and width of the low-light frame, respectively, and F represents the number of frames in the video;

[0071] Step D2: Input the low-light video frame by frame into the low-light video enhancement model trained in step C to obtain F frames of enhanced video frames of size H×W×3;

[0072] Step D3: merge F frames of enhanced video with a size of H×W×3 into an enhanced video with a size of F×H×W×3.

[0073] The present invention also provides a computer-readable storage medium on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, any of the method steps described above can be implemented.

[0074] Compared with the prior art, the present application has the following beneficial effects: first, the present application uses the encoder of the UNet backbone network as a spatial feature encoder, and the decoder of the UNet backbone network as a spatial feature decoder, which improves the expandability of the encoder-decoder architecture model. Secondly, the features extracted by the spatial feature encoder are embedded in the input embedding module, which can effectively reduce the complexity of subsequent calculations while extracting embedded features with spatial local interaction characteristics, and unlike the traditional Transformer, which realizes block coding based on linear layers and non-overlapping embedding layers, the present application uses an overlapping embedding method, which can simultaneously retain position and adjacent feature information. Thirdly, the present application designs a self-recurrent memory query enhancement block, including a learnable query position encoding, an absolute position encoding, a memory query attention block, a space-time interaction attention block and a multi-layer perception memory update block. These modules use memory query and space-time interaction to enhance the temporal consistency of the image model based on the encoder-decoder architecture processing video, realize adaptive retrieval of historical key frame features through the memory query attention block, and complete feature fusion by combining the space-time interaction attention block and the multi-layer perception memory update block, effectively ensuring the temporal consistency of the video. Finally, the present application uses the output embedding module to reconstruct the query enhancement features, which can adjust the time on the basis of preserving the original feature space information and enhance the temporal consistency of the features. Unlike other advanced low-light video enhancement methods that build model frameworks from scratch, the present application not only retains the advantages of image model processing spatial information based on the encoder-decoder architecture, but also effectively improves the temporal consistency of the image model based on the encoder-decoder architecture processing video using self-recurrent query interaction operations, realizes high-quality and efficient arbitrary length video enhancement in a limited computing resource environment, and greatly reduces the deployment cost. BRIEF DESCRIPTION OF DRAWINGS

[0075] Figure 1 FOR THE INVENTION Figure 1 is the implementation flowchart of the method in the preferred embodiment of the present application.

[0076] Figure 2 is the structure diagram of the low-light video enhancement network in the preferred embodiment of the present application.

[0077] Figure 3 is the structure diagram of the input and output embedding modules in the preferred embodiment of the present application.

[0078] Figure 4 is the structure diagram of the space-time interaction enhancement module based on self-recurrence in the preferred embodiment of the present application. DETAILED DESCRIPTION

[0079] The technical solutions of the present application will be specifically described below with reference to the drawings.

[0080] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present application belongs.

[0081] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form, and it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0082] The present invention provides a method for enhancing low-light video of any frame based on autoregressive spatiotemporal interaction. Figures 1-4 As shown, the following steps are included:

[0083] Step A: preprocess the input video data, including data pairing, data sampling, data trimming and data enhancement, to obtain a training data set;

[0084] Step B: Design a low-light video enhancement network, including a spatial feature encoder, an autoregressive spatiotemporal interaction enhancement module, and a spatial feature decoder;

[0085] Step C: Design a loss function for training the network designed in step B, and use the training data set described in step A to train the network in step B to obtain a trained low-light video enhancement model;

[0086] Step D: Input the low-light video to be tested into the model obtained in step C to generate a normal-light video.

[0087] Furthermore, the specific implementation steps of step A are as follows:

[0088] Step A1: Pair the normal illumination video with the corresponding low illumination video in the dataset, with the normal illumination video as the label video; then scale all video frames to a W×H resolution, where W is width and H is height;

[0089] Step A2, randomly select a continuous video segment with a length of L from each low-illumination video, and at the same time, obtain a continuous video segment with a length of L corresponding to the normal-illumination video in the same way to ensure that they correspond to each other in the time dimension, to obtain paired low-illumination video segments and normal-illumination video segments with a length of L; repeat the operation of each paired low-illumination video and normal-illumination video several times; repeat the operation of all paired low-illumination videos and normal-illumination videos to obtain video segments with a size of LxHxWx3, wherein L is the number of video frames in each video segment in the training data set;

[0090] Step A3, randomly crop the paired low-illumination video segments and normal-illumination video segments with a length of L into paired video segments with a size of LxPxPx3 in a consistent manner, wherein P is the height and width of the cropped video frame;

[0091] Step A4, for each paired video, randomly apply one of the following four data enhancement methods: keep the original video frame, vertical flip, horizontal flip, and vertical flip followed by horizontal flip; finally, obtain the paired low-illumination video and normal-illumination video training data set.

[0092] Further, step B is implemented as follows:

[0093] Step B1, design a spatial feature encoder, use the encoder of the UNet backbone network as the spatial feature encoder, and perform feature extraction on the input low-illumination video frame to obtain the extracted feature wherein P is the height and width of the low-illumination video frame, h, w and d are the height, width and channel number of the low-illumination video frame after feature extraction by the spatial feature encoder, and subscript i represents the serial number of the video frame;

[0094] Step B2, design an autoregressive-based spatio-temporal interaction enhancement module, which consists of an input embedding module, an autoregressive memory query enhancement block, an output embedding module and a memory bank, for the feature I i extracted in step B1, through the autoregressive-based spatio-temporal interaction enhancement module, to obtain the spatio-temporal interaction enhanced feature Then, the spatio-temporal interaction enhanced feature is connected in residual to the feature I i extracted in step B1 to obtain the comprehensive spatio-temporal interaction enhanced feature The whole process is represented by the formula as follows:

[0095]

[0096] wherein ARSTEM represents the autoregressive-based spatio-temporal interaction enhancement module;

[0097] Step B3, design a spatial feature decoder, adopt the decoder of UNet backbone network as the spatial feature decoder, and the integrated spatio-temporal interaction enhanced feature obtained in step B2 is input into the spatial feature decoder as the input of the spatial feature decoder, and the output is the enhanced video frame

[0098] Further, step B2 is specifically implemented as follows:

[0099] Step B21, design an input embedding module, which sequentially includes a depth-wise convolution unit, a point-wise convolution unit and a normalization reshaping unit; wherein the depth-wise convolution unit is composed of a convolution layer with a convolution kernel size of 10x10, a step of 7, a padding of 3 and a group number of input feature channel number d, the point-wise convolution unit is composed of a convolution layer with a convolution kernel size of 1x1, a step of 1, a padding of 0 and a group number of 1, and the normalization reshaping unit includes three learnable parameters α, ω, β and a reshaping unit, and the reshaping unit reshapes the feature with a dimension of 3 into a feature with a dimension of 2; the input embedding module is used for feature embedding processing of , specifically: I i After sequentially passing through the depth-wise convolution unit and the point-wise convolution unit, the intermediate feature is obtained The intermediate feature is input into the normalization reshaping unit for processing, wherein the reshaping unit flattens the feature to obtain the embedded feature r represents the reduction rate of the feature after the depth-wise convolution unit processing, represents the channel number of the embedded feature; the whole process is represented by the formula:

[0100]

[0101] wherein, DWConv represents the depth-wise convolution unit, PWConv represents the point-wise convolution unit, α, ω, β are learnable parameters, ⊕ represents element-wise addition, ⊙ represents element-wise multiplication, and Reshape(·) is the operation of the reshaping unit, which reshapes the feature with a size of to a feature with a size of ;

[0102] Step B22, design a memory bank, which is used to store memory features, and the initial memory feature corresponding to the i-th frame in the memory bank is a zero feature, and m represents the dimension value of the first dimension of the memory feature; the low-light video enhancement network described in step B contains historical memory features Z i-1 ,…,Z0in the memory bank before processing the i-th frame, wherein k e {0, 1, …, i-1} ;

[0103] Step B23, design an autoregressive memory query enhancement block, which is composed of a learnable query position encoding, an absolute position encoding, a memory query attention block, a spatiotemporal interaction attention block, and a multilayer perception memory update block; wherein the embedding feature X i obtained in step B21 is processed by n stacked autoregressive memory query enhancement blocks to output a query enhancement feature and a current memory feature and the current memory feature Z i is appended to the memory bank, and the whole process is represented by the formula:

[0104]

[0105] wherein ARQB l represents the lth autoregressive memory query enhancement block, l e {0, 1, …, n}, n is the number of stacked autoregressive memory query enhancement blocks, and the symbol represents the stacking operation of the autoregressive memory query enhancement block, and Mean(·) represents the average value calculation operation along the first dimension of the feature;

[0106] Step B24, design an output embedding module, which sequentially includes a reshaping upsampling unit, a depth-wise convolution unit, and a point-wise convolution unit; wherein the reshaping upsampling unit includes a reshaping unit and a bilinear upsampling operation, the reshaping unit reshapes the feature with a dimension of 2 to a feature with a dimension of 3, and the bilinear upsampling operation up-samples the feature with a width and height of to a feature with a width and height of h x w, the depth-wise convolution unit is composed of a convolution layer with a kernel size of 3 x 3, a stride of 1, a padding of 1, and a group number of the input feature channel number , and the point-wise convolution unit is composed of a convolution layer with a kernel size of 1 x 1, a stride of 1, a padding of 0, and a group number of 1; the output embedding module is used to reconstruct the query enhancement feature obtained in step B23, specifically: using the reshaping upsampling unit to reshape and up-sample to obtain an intermediate restoration feature then passing through the depth-wise convolution unit and the point-wise convolution unit in sequence to obtain an output feature The whole process is represented by the formula:

[0107]

[0108] wherein the operation of the reshaping unit here reshapes the feature with a size of to a feature with a size of characteristics, Upsample(·) represents a bilinear up-sampling operation, and the role is to up-sample the size of characteristics to the size of characteristics.

[0109] Further, step B23 is specifically implemented as follows:

[0110] Step B231, for the characteristics obtained by inputting the embedding module in step B21 obtain its position encoding from the absolute position encoding for the historical memory characteristics Z k , k e {0, 1, …, i-1}, obtain its corresponding position encoding from the absolute position encoding k e {0, 1, …, i-1}, specifically using the sinusoidal position encoding method to generate; the whole process is represented as:

[0111]

[0112] wherein, PE function represents the sinusoidal position encoding generation function of the pos frame, pos represents the time position of the historical frame or the current frame, τ is the temperature coefficient, the value is 10000.0, [·; ·] represents the splicing operation, repeat(·, N) represents the replication splicing operation, that is, the size of characteristics are replicated and spliced into characteristics with a size of ;

[0113] The embedding feature is input into the spatio-temporal interaction attention block as the value feature, the embedding feature X i and the position encoding are added to obtain the embedding feature with position information as the key feature input into the spatio-temporal interaction attention block; at the same time, the historical memory characteristics Z i-1 ,…, Z0 are input into the memory query attention block as the value feature, and the position encoding is obtained from the absolute position encoding in the same way The historical memory characteristics and the corresponding position encoding are added to obtain as the key feature input into the memory query attention block; the initial memory feature obtained in step B22 is taken as the stage initial memory feature Z init of the first autoregressive memory query enhancement block, and is input into the memory query attention block as the query feature;

[0114] Step B232, design the memory query attention block, which is composed of a multi-head memory cross-attention layer, a normalization layer and a residual connection; the initial current memory feature Zinit as the query feature, the history memory feature with position information as the key feature, the history memory feature Z i-1 ,…,Z0as the value feature input into the multi-head memory cross attention layer, and the obtained feature is input into the normalization layer and combined with Z init to perform residual connection and output the memory enhanced feature The entire process is represented by the formula as follows:

[0115] M i =Z init +Norm(MMCA(Z init ,[Z i-1 +;…;Z0+],[Z i-1 ;…;Z0))

[0116] wherein MMCA represents the multi-head memory cross attention layer, and Norm(·) represents the normalization operation;

[0117] Step B233, designing the space-time interaction attention block, which is composed of a multi-head space-time cross attention layer, a normalization layer and residual connection; the enhanced memory feature M i is added to the learnable query position encoding to obtain M i +Q i as the query feature, the embedded feature with position information as the key feature, the embedded feature X i is input into the multi-head space-time cross attention layer, and the obtained feature is input into the normalization layer and combined with M i to perform residual connection and output the space-time interaction feature The entire process is represented by the formula as follows:

[0118]

[0119] wherein MSTA represents the multi-head space-time cross attention layer;

[0120] Step B234, designing the multi-layer perception memory update block, which is composed of a multi-layer perception feedforward layer and a normalization layer; the space-time interaction feature T i is processed through the multi-layer perception feedforward layer and the normalization layer, and the obtained feature is combined with the space-time interaction feature T i to perform residual connection and obtain the current memory feature output by the first autoregressive memory query enhancement block The entire process is represented by the formula as follows:

[0121]

[0122] wherein FFN represents the multi-layer perception feedforward layer;

[0123] Step B235, the current memory feature output by the u-1th autoregressive memory query enhancement block The stage initial memory feature Z as input of the u th autoregressive memory query enhancement block init , repeat steps B232 to B234, gradually enhance the spatiotemporal consistency feature of the video frame through the processing of the stack of n autoregressive memory query enhancement blocks, wherein u e {1, 2, …, n}; the current memory feature output by the n th autoregressive memory query enhancement block The current memory feature Z as the i th frame i , and append the current memory feature Z i to the memory bank.

[0124] Further, step B232 is specifically implemented as follows:

[0125] Step B2321, design a multi-head memory cross-attention layer, for the input stage initial memory feature Z init , the historical memory feature with position information and the historical memory feature Z i-1 , …, Z0, first concatenate the historical memory feature with position information into a feature with size Concatenate the historical memory feature into a feature with size [Z i-1 , …, Z0], wherein M = m x (i-1); then respectively split into h heads along the channel dimension, and then use linear mapping to map each head of the stage initial memory feature Z init to a “query” matrix Map each head of the concatenated historical memory feature with position information [Z i-1 + P i-1 , …, Z0+ P0] to a “key” matrix Map each head of the concatenated historical memory feature [Z i-1 , …, Z0] to a “value” matrix , wherein j e {1, 2, …, h};

[0126] Step B2322, divide Q mem,j by the temperature coefficient τ d , transpose K mem,j to obtain Perform matrix multiplication on Q mem,j and to obtain a historical memory attention Att mem,j , calculate softmax along the channel dimension of Att mem,k ; then multiply Att mem,k and V​mem,k Matrix multiplication is performed to obtain the head-by-head memory interaction enhanced feature M query,k of size query The entire process is represented by the formula:

[0127]

[0128] M query = [M query,1 ; M query,2 ; …; M query,h ]

[0129] wherein, denotes matrix multiplication, softmax(·) denotes a softmax operation calculated along the channel dimension, denotes a temperature coefficient;

[0130] Step B2323, the memory interaction enhanced feature M query obtained by querying the multi-head memory cross-attention layer in step B2322 is mapped to an output matrix Z by linear mapping, and the obtained feature is connected with Z init by residual connection to obtain the memory enhanced feature M i .

[0131] Further, step B233 is specifically implemented as follows:

[0132] Step B2331, a multi-head spatio-temporal cross-attention layer is designed, and for the input enhanced memory feature M i + Q i with position information, the embedded feature X with position information, and the embedded feature X i , each is first divided into h heads along the channel dimension, and then linear mapping is performed to map each head of the enhanced memory feature M i + Q i with position information to a “query” matrix each head of the embedded feature X i + P i with position information to a “key” matrix and each head of the embedded feature X i to a “value” matrix wherein j ∈ {1, 2, …, h};

[0133] Step B2332, Q st,j is divided by the temperature coefficient τ d ​​, for K st,j Perform the transpose operation to obtain Q st,j and Perform matrix multiplication to obtain the spatiotemporal cross attention Att of size m×N st,j , for Att st,j Calculate softmax along the channel dimension; then st,j and V st,j Perform matrix multiplication to get a matrix of size The head-by-head spatiotemporal cross-enhanced feature T st,j , the spatiotemporal cross-enhancement features are spliced ​​along the channel dimension to obtain a size of The spatiotemporal cross-enhancement feature T st , the process is expressed by the formula:

[0134]

[0135] T st =[T st,1 ;T st,2 ;…;T st,h ]

[0136] in, represents matrix multiplication, softmax(·) represents the softmax operation calculated along the channel dimension, represents the temperature coefficient;

[0137] Step B2333: The spatiotemporal cross-enhanced feature T obtained by the interaction of the multi-head spatiotemporal cross-attention layer in step B2332 is st , use linear mapping to map it to the output matrix Send it to the normalization layer, and the obtained features are combined with M i Residual connection to obtain spatiotemporal interaction features T i .

[0138] Furthermore, the specific implementation steps of step C are as follows:

[0139] Step C1: Construct L1 loss and VGG perceptual loss The total loss function Its expression is:

[0140]

[0141] Among them, λ1 and λ2 are equilibrium parameters, Represents the enhanced video frame of the input i-th video frame, V gt,i represents the video frame with normal illumination corresponding to the i-th video frame, ||·|| 2MSE, mean square error 3,8,15 (·) represents the 3rd, 8th, 15th layer features extracted by the VGG-16 network pre-trained with ImageNet;

[0142] Step C2, randomly divide the training data set obtained in step A into several batches, each batch containing several pairs of video clips;

[0143] Step C3, adopt a staged optimization strategy: for single-frame enhancement results, calculate the loss described in step C1 Update the parameters of the spatial feature encoder and the spatial feature decoder by back propagation; for the complete video clip, calculate the average value of the loss of all frames, update the parameters of the autoregressive-based spatio-temporal interaction enhancement module described in step B by back propagation; wherein, the network parameters are updated by using the Adam optimization method;

[0144] Step C4, repeat steps C2 to C3 until the model converges, and finally obtain the trained low-light video enhancement model.

[0145] Further, step D is specifically implemented as follows:

[0146] Step D1, split the low-light video with a size of FxHxWx3 into F frames of low-light video with a size of HxWx3, wherein H and W are the height and width of the low-light frame respectively, and F represents the number of frames in the video.

[0147] Step D2, input the low-light video frame by frame into the low-light video enhancement model trained in step C to obtain F frames of enhanced video frames with a size of HxWx3.

[0148] Step D3, combine the F frames of enhanced video frames with a size of HxWx3 into an enhanced video with a size of FxHxWx3.

[0149] ​The present application aims to further solve the key problems of video length limitation and time sequence consistency fault caused by offline video loading mode in the existing low-illumination video enhancement method. To this end, an arbitrary frame low-illumination video enhancement method based on autoregressive space-time interaction is designed. First, a spatial encoder is designed to extract features from the input low-illumination video frame. Then, an autoregressive space-time interaction enhancement module is designed, including an input embedding module, an autoregressive memory query enhancement block, an output embedding module and a memory bank. The input embedding module is designed to embed the input features from the spatial encoder, the memory bank is designed to store memory features, the autoregressive memory query enhancement block is designed, including the design of learnable query position encoding and absolute position encoding to assist the retrieval of the most relevant memory in the memory bank, the memory query attention block is designed to retrieve the most relevant memory from the historical memory, the space-time interaction attention block is designed to integrate the most relevant memory into the spatial features through space-time interaction, and the output embedding module is designed to reconstruct the query enhancement features into space-time interaction enhancement features. Finally, a spatial feature decoder is designed to decode the integrated space-time interaction enhancement features into normal-illumination video frames. This arbitrary frame low-illumination video enhancement method based on autoregressive space-time interaction combines the global modeling advantages of the autoregressive space-time interaction enhancement module and the spatial feature extraction capabilities of the image model based on the encoder-decoder architecture: the autoregressive space-time interaction enhancement module stores historical frame features through the memory bank and realizes long-time space-time feature interaction through the multi-head memory cross-attention mechanism, which can effectively model the dynamic dependency between video frames and enhance the network's perception of time dimension continuity and spatial dimension consistency. At the same time, it retains the spatial feature extraction capability of the image model based on the encoder-decoder architecture, which can effectively process spatial features and effectively reduce the time cost of building a model from scratch. Therefore, the method of adding an autoregressive space-time interaction component to the image model based on the encoder-decoder architecture can better improve the frame-by-frame video inference performance of the image model based on the encoder-decoder architecture with only minor modifications to the image model based on the encoder-decoder architecture.

[0150] The present application also provides a computer-readable storage medium having stored thereon computer program instructions capable of being executed by a processor, which can implement the method steps of any of the above when the processor executes the computer program instructions.

[0151] The above is the preferred embodiment of the present application, any changes made in accordance with the technical solutions of the present application, as long as the resulting functional effects do not exceed the scope of the technical solutions of the present application, are within the scope of protection of the present application.

Claims

1. A method for enhancing low-light video of any frame based on autoregressive spatiotemporal interaction, characterized in that: The steps include: Step A: preprocess the input video data, including data pairing, data sampling, data trimming and data enhancement, to obtain a training data set; Step B: Design a low-light video enhancement network, including a spatial feature encoder, an autoregressive spatiotemporal interaction enhancement module, and a spatial feature decoder; Step C: Design a loss function for training the network designed in step B, and use the training data set described in step A to train the network in step B to obtain a trained low-light video enhancement model; Step D: Input the low-light video to be tested into the model obtained in step C to generate a normal-light video.

2. The method for enhancing low-light video of any frame based on autoregressive spatiotemporal interaction according to claim 1, characterized in that: The specific implementation steps of step A are as follows: Step A1: Pair the normal illumination video with the corresponding low illumination video in the dataset, with the normal illumination video as the label video; then scale all video frames to a W×H resolution, where W is width and H is height; Step A2: Randomly select a continuous video segment of length L from each low-light video. Simultaneously, use the same method to obtain a corresponding continuous video segment of length L for the normal-light video corresponding to the low-light video, ensuring that each frame corresponds to each other in the temporal dimension, thereby obtaining paired low-light video segments and normal-light video segments of length L. Repeat the operation several times for each pair of low-light video and normal-light video. Repeat the operation for all pairs of low-light video and normal-light video, and the size of the resulting video segments is L×H×W×3, where L is the number of video frames in each video segment in the training dataset. Step A3: randomly cropping the paired low-illumination video clips and normal-illumination video clips of length L into paired video clips of size L×P×P×3 in a consistent manner, where P is the height and width of the cropped video frame; Step A4: For each paired video, randomly apply one of the following four data augmentation methods: keep the original video frame, vertical flip, horizontal flip, and vertical flip followed by horizontal flip; finally, obtain the paired low-light video and normal-light video training datasets.

3. The method for enhancing low-light video of any frame based on autoregressive spatiotemporal interaction according to claim 1, characterized in that: The specific implementation steps of step B are as follows: Step B1: Design a spatial feature encoder and use the UNet backbone network encoder as the spatial feature encoder to input low-light video frames. Perform feature extraction to obtain the extracted features Where P is the height and width of the low-light video frame, h, w, and d are the height, width, and number of channels of the low-light video frame after feature extraction by the spatial feature encoder, and the subscript i represents the sequence number of the video frame; Step B2: Design an autoregressive spatiotemporal interaction enhancement module, which consists of an input embedding module, an autoregressive memory query enhancement block, an output embedding module, and a memory bank. i , after the spatiotemporal interaction enhancement module based on autoregression, the spatiotemporal interaction enhancement feature is obtained Then the spatiotemporal interaction enhancement features and feature I extracted in step B1 i Perform residual connection to obtain comprehensive spatiotemporal interaction enhancement features The whole process is expressed as follows: Among them, ARSTEM represents the spatiotemporal interaction enhancement module based on autoregression; Step B3, design a spatial feature decoder, use the decoder of the UNet backbone network as the spatial feature decoder, and the comprehensive spatiotemporal interaction enhancement feature obtained in step B2 As the input of the spatial feature decoder, the output is the enhanced video frame 4. The method for enhancing low-light video of any frame based on autoregressive spatiotemporal interaction according to claim 3, characterized in that: The specific implementation steps of step B2 are as follows: Step B21, design an input embedding module, which includes a depth-wise convolution unit, a point-wise convolution unit and a normalized reshaping unit in sequence; wherein the depth-wise convolution unit is composed of a convolution layer with a convolution kernel size of 10×10, a step size of 7, a padding of 3, and the number of groups being the number of input feature channels d, and the point-wise convolution unit is composed of a convolution layer with a convolution kernel size of 1×1, a step size of 1, a padding of 0, and the number of groups being 1. The normalized reshaping unit contains three learnable parameters ɑ, ω, β and a reshaping unit, and the reshaping unit reshapes the feature of dimension 3 into the feature of dimension 2; the input embedding module is used to Perform feature embedding processing, specifically: I i After passing through the depth-wise convolution unit and the point-wise convolution unit in sequence, the intermediate features are obtained The intermediate features Input normalization reshaping unit processing, where the reshaping unit transforms the features Flatten to get embedded features Where N = r represents the ratio of feature reduction after processing by the depth-wise convolution unit, Represents the number of channels of embedded features; The whole process is expressed as follows: Among them, DWConv represents the depth-wise convolution unit, PWConv represents the point-wise convolution unit, α, ω, and β are learnable parameters. Represents element-by-element addition, ⊙ represents element-by-element multiplication, and Reshape(·) is an operation to reshape the unit. The features of are reshaped into a size of characteristics; Step B22: Design a memory library to store memory features. The initial memory features corresponding to the i-th frame in the memory library are is an all-zero feature, m represents the dimension value of the first dimension of the memory feature; before the low-light video enhancement network described in step B processes the i-th frame, the historical memory feature contained in the memory bank is Z i-1 ,…,Z0, where Step B23, design an autoregressive memory query enhancement block, which consists of a learnable query position code, an absolute position code, a memory query attention block, a spatiotemporal interaction attention block, and a multi-layer perceptual memory update block; wherein, for the embedded feature X obtained in step B21 i After being processed by n stacked autoregressive memory query enhancement blocks, the query enhancement feature is output and current memory characteristics And the current memory feature Z i Append to the memory bank, the whole process is expressed as follows: Among them, ARQB l represents the lth autoregressive memory query enhancement block, l∈{0,1,…,n, n is the number of stacked autoregressive memory query enhancement blocks, symbol represents the stacking operation of the autoregressive memory query enhancement block, and Mean(·) represents the average value calculation operation along the first dimension of the feature; Step B24, design an output embedding module, which includes a reshaping upsampling unit, a depth-wise convolution unit, and a point-wise convolution unit in sequence; wherein the reshaping upsampling unit includes a reshaping unit and a bilinear upsampling operation, the reshaping unit reshapes the feature of dimension 2 into the feature of dimension 3, and the bilinear upsampling operation reshapes the width and height into the feature of dimension 3. The features are upsampled to features with a width and height of h×w. The depth-wise convolution unit consists of a convolution kernel size of 3×3, a step size of 1, a padding of 1, and the number of groups is the number of input feature channels. The point-by-point convolution unit is composed of a convolution layer with a convolution kernel size of 1×1, a step size of 1, a padding of 0, and a group number of 1; the output embedding module is used to enhance the query feature obtained in step B23 Reconstruction is performed by using the reshape upsampling unit to Reshape and upsample to obtain intermediate restored features Then After passing through the depth-wise convolution unit and the point-wise convolution unit in sequence, the output feature is obtained The whole process is expressed as follows: Here, the reshape operation will be of size The features of are reshaped into a size of The feature of , Upsample(·) represents the bilinear upsampling operation, which is used to convert the size of The features are upsampled to a size of h×w× characteristics.

5. The method for enhancing low-light video of any frame based on autoregressive spatiotemporal interaction according to claim 4, characterized in that: The specific implementation steps of step B23 are as follows: Step B231: For the features obtained by inputting the embedding module in step B21 Get the position code from the absolute position code For historical memory feature Z k ,k∈{0,1,…,i-1}, obtain its corresponding position code from the absolute position code Specifically, the sinusoidal position encoding method is used for generation; the whole process is expressed as: Among them, PE function represents the sinusoidal position encoding function of the posth frame, pos represents the time position of the historical frame or the current frame, τ is the temperature coefficient, and the value is 10000.0, [·;·] represents the splicing operation, and repeat(·,N) represents the copy splicing operation, that is, the size is The features of are copied and spliced ​​into a size of characteristics; Embed features As the value feature input spatiotemporal interaction attention block, the embedding feature X i and positional encoding Add together to get the embedded features with position information Input the spatiotemporal interaction attention block as the key feature; at the same time, obtain the historical memory feature Z from the memory bank i-1 ,…,Z0, is input into the memory query attention block as the value feature, and its position encoding is obtained from the absolute position encoding in the same way Add the historical memory feature to the corresponding position code to get Input the memory query attention block as the key feature; the initial memory feature obtained in step B22 As the first autoregressive memory query enhancement block, the initial memory feature Z init , and is input into the memory query attention block as query features; Step B232, design the memory query attention block, which consists of a multi-head memory cross attention layer, a normalization layer and a residual connection; the initial current memory feature Z init As query features, historical memory features with location information As key feature, historical memory feature Z i-1 ,…,Z0 is input as the value feature into the multi-head memory cross attention layer, and the obtained feature is input into the normalization layer and then compared with Z init Perform residual connection and output memory enhancement features The whole process is expressed as follows: M i =Z init +Norm(MMCA(Z init ,[WITH i-1 +;…;Z0+],[Z i-1 ;…;Z0])) Among them, MMCA represents the multi-head memory cross attention layer, and Norm(·) represents the normalization operation; Step B233, design the spatiotemporal interactive attention block, which consists of a multi-head spatiotemporal cross attention layer, a normalization layer and a residual connection; enhance the memory feature M i and learnable query position encoding The sum of M i +Q i As query features, embedded features with location information As key features, embedding features X i As the value feature input multi-head spatiotemporal cross attention layer, the obtained feature input normalization layer, and M i Residual connection, output spatiotemporal interaction features The whole process is expressed as follows: Among them, MSTA represents the multi-head spatiotemporal cross attention layer; Step B234, design a multi-layer perception memory update block, which consists of a multi-layer perception machine feedforward layer and a normalization layer; transform the spatiotemporal interaction feature T i After being processed by the multi-layer perceptron feedforward layer and normalization layer, the features obtained are then combined with the spatiotemporal interaction features T i Perform residual connection to obtain the current memory feature output by the first autoregressive memory query enhancement block The whole process is expressed as follows: Among them, FFN represents the feedforward layer of the multi-layer perceptron; Step B235: The current memory feature output by the u-1th autoregressive memory query enhancement block As input to the u-th autoregressive memory query enhancement block, the initial memory feature Z init Repeat steps B232 to B234, and gradually enhance the spatiotemporal consistency features of the video frame by stacking n autoregressive memory query enhancement blocks, where u∈{1,2,…,n}; the current memory feature output by the nth autoregressive memory query enhancement block is As the current memory feature Z of the i-th frame i , and store the current memory feature Z i Append to the memory.

6. The method for enhancing low-light video of any frame based on autoregressive spatiotemporal interaction according to claim 5, characterized in that: The specific implementation steps of step B232 are as follows: Step B2321, design a multi-head memory cross attention layer, for the initial memory feature Z of the input stage init , historical memory features with location information and historical memory characteristics Z i-1 ,…,Z0, first concatenate the historical memory features with location information into a size of Features Concatenate historical memory features into a Characteristics of [Z i-1 ;…;Z0], where M = m×(i-1); then each is split into h heads along the channel dimension, and then a linear mapping is used to convert the initial memory feature Z init Each head is mapped to a "query" matrix The spliced ​​historical memory features with location information [Z i-1 +P i-1 ; ...; Each head of Z0+P0] is mapped to a "key" matrix The spliced ​​historical memory features [Z i-1 ; ...; Each head of Z0] is mapped to a "value" matrix where j∈{1,2,…,h}; Step B2322, Q mem,j Divide by the temperature coefficient τ d , for K mem,j Perform the transpose operation to obtain Q mem,j and Perform matrix multiplication to obtain the history memory attention Att of size m×M mem,j , for Att mem,j Calculate softmax along the channel dimension; then mem,j and V mem,j Perform matrix multiplication to get a matrix of size The head-by-head memory interaction enhancement feature M query,j , all the head-by-head memory interaction enhancement features are spliced ​​along the channel dimension to obtain a size of Memory interaction enhancement feature M query , the whole process is expressed as follows: M query =[M query,1 ;M query,2 ;…;M query,h ] in, represents matrix multiplication, softmax(·) represents the softmax operation calculated along the channel dimension, represents the temperature coefficient; Step B2323: query the memory interaction enhancement feature M obtained by the multi-head memory cross attention layer query in step B2322 query , use linear mapping to map it to the output matrix Send it to the normalization layer, and the obtained features are combined with Z init Residual connection to obtain memory enhancement feature M i .

7. The method for enhancing low-light video of any frame based on autoregressive spatiotemporal interaction according to claim 5, characterized in that: The specific implementation steps of step B233 are as follows: Step B2331, design a multi-head spatiotemporal cross attention layer to enhance the memory feature M with position information of the input i +Q i , embedded features with position information and embedding feature X i , first split each into h heads along the channel dimension, and then use linear mapping to transform the enhanced memory feature M with position information i +Q i Each head is mapped to a "query" matrix Embedded feature X with position information i +P i Each head is mapped to a "key" matrix Embedded feature X i Each head is mapped to a "value" matrix where j∈{1,2,…,h}; Step B2332, Q st,j Divide by the temperature coefficient τ d , for K st,j Perform the transpose operation to obtain Q st,j and Perform matrix multiplication to obtain the spatiotemporal cross attention Att of size m×N st,j , for Att st,j Calculate softmax along the channel dimension; then st,j and V st,j Perform matrix multiplication to get a matrix of size The head-by-head spatiotemporal cross-enhanced feature T st,j , the spatiotemporal cross-enhancement features are spliced ​​along the channel dimension to obtain a size of The spatiotemporal cross-enhancement feature T st , the process is expressed by the formula: T st =[T st,1 ;T st,2 ;…;T st,h ] in, represents matrix multiplication, softmax(·) represents the softmax operation calculated along the channel dimension, represents the temperature coefficient; Step B2333: The spatiotemporal cross-enhanced feature T obtained by the interaction of the multi-head spatiotemporal cross-attention layer in step B2332 is st , use linear mapping to map it to the output matrix Send it to the normalization layer, and the obtained features are combined with M i Residual connection to obtain spatiotemporal interaction features T i .

8. The method for enhancing low-light video of any frame based on autoregressive spatiotemporal interaction according to claim 1, characterized in that: The specific implementation steps of step C are as follows: Step C1: Construct L1 loss and VGG perceptual loss The total loss function Its expression is: Among them, λ1 and λ2 are equilibrium parameters, Represents the enhanced video frame of the input i-th video frame, V gt,i represents the video frame with normal illumination corresponding to the i-th video frame, ||·|| 2 Indicates mean square error calculation, VGG 3,8,15 (·) represents the 3rd, 8th, and 15th layer features extracted by the ImageNet pre-trained VGG-16 network; Step C2: randomly divide the training dataset obtained in step A into several batches, each batch containing several pairs of video clips; Step C3: Use a phased optimization strategy: Calculate the loss described in step C1 for the single-frame enhancement result Update the parameters of the spatial feature encoder and spatial feature decoder through backpropagation; calculate the loss of all frames for the complete video segment The parameters of the spatiotemporal interaction enhancement module based on autoregression described in step B are updated by back propagation; the Adam optimization method is used to update the network parameters; Step C4: Repeat steps C2 to C3 until the model converges, and finally obtain a trained low-light video enhancement model.

9. The method for enhancing low-light video of any frame based on autoregressive spatiotemporal interaction according to claim 1, characterized in that: The specific implementation steps of step D are as follows: Step D1, splitting the low-light video of size F×H×W×3 into F low-light video frames of size H×W×3, where H and W are the height and width of the low-light frame, respectively, and F represents the number of frames in the video; Step D2: Input the low-light video frame by frame into the low-light video enhancement model trained in step C to obtain F frames of enhanced video frames of size H×W×3; Step D3: merge F frames of enhanced video with a size of H×W×3 into an enhanced video with a size of F×H×W×3.

10. A computer-readable storage medium storing computer program instructions that can be executed by a processor, wherein when the processor executes the computer program instructions, the method steps according to any one of claims 1 to 9 can be implemented.

Images