An implicit video representation method based on decoupled spatial and temporal information

Abstract

The present invention relates to the field of computer vision and discloses a method for implicit video representation based on decoupled spatial and temporal information, comprising the following steps: S1, training a specific video clip, optimizing network parameters by reducing the network loss function until the network converges, and obtaining network weights for efficient implicit video representation based on decoupled spatial and temporal information; S2, using the network weights obtained in the training phase to reconstruct the entire video sequence frame by frame under the input of a given frame index; S3, comparing the reconstructed video sequence with the original video to evaluate the implicit representation capability. The present invention decouples the spatial and temporal information in the video using a specific network structure, performs more efficient network parameter allocation, enables the system to more accurately and effectively represent the video to be represented within a given parameter range, and achieves a faster network convergence speed.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a video implicit representation method based on decoupled spatial and temporal information. Background Art

[0002] Modeling and representing different signals is currently a major research topic in computer vision, with significant real-world applications and significance, such as reconstruction based on implicit representations of three-dimensional objects and super-resolution restoration based on implicit representations of images. Driven by deep learning, research on using neural networks to represent specific signals has made significant progress. This technology primarily targets implicit video representation tasks, where a video is represented through network fitting. During testing, the entire video signal is reconstructed from the network and used for specific downstream tasks such as video compression. The primary challenge in this task is how to accurately and efficiently fit the image of each frame in a video using fewer network parameters.

[0003] Currently, methods for implicitly representing videos using neural networks can be roughly divided into two categories: pixel-by-pixel representation and frame-by-frame representation. The pixel-by-pixel representation approach is an extension of implicit image representation. The network input is the 3D coordinates of any pixel in the video—that is, the pixel's position in image space and the frame index—and the output is the 3D RGB color of that pixel. This approach implicitly represents the entire video using the weights of the neural network. This approach is very simple, and the network structure is correspondingly concise: a multilayer perceptron serves as the neural network. However, in practice, this approach is slow due to the pixel-by-pixel optimization and inference process, and is difficult to scale to long video sequences (hundreds of frames). Recently, a frame-by-frame implicit video representation approach has been proposed. This approach takes as input the index of a single frame and outputs the entire 2D image of that frame. Specifically, this approach employs a multilayer perceptron coupled to a convolutional neural network. The multilayer perceptron maps the input one-dimensional frame index into a high-dimensional feature vector, and the convolutional network is used to recover the two-dimensional image information from the one-dimensional feature vector. Due to its frame-by-frame optimization and inference paradigm, this representation approach significantly outperforms previous pixel-by-pixel approaches in terms of speed. Although the frame-by-frame implicit representation of videos can achieve relatively fast optimization and inference, and can be used for downstream tasks such as video compression, its network structure still has a large number of redundant parameters and structures, resulting in suboptimal performance. This implicit representation paradigm also has room for further improvement. Summary of the Invention

[0004] In response to the problem of redundant parameter structure in frame-by-frame representation, the present invention proposes an implicit video representation method based on decoupling spatial and temporal information to save redundant parameter structure and further improve representation performance. Intuitively, a large number of unnecessary parameters in the original frame-by-frame representation method come from the part that transforms high-dimensional features into two-dimensional feature matrices. In order to obtain sufficient representation performance, this step requires feature vectors with higher dimensions, so the last layer of the multi-layer perceptron will take up a lot of space. The deeper reason is that the spatiotemporal information of the entire video is fitted together in a single structure, so a large number of parameters are required. Decoupling spatial and temporal information and fitting them with different structures of the network can save a large number of unnecessary parameters while having comparable performance. On this basis, the present invention further analyzes the original method structure and performs further optimization to continue to reduce the redundant parameter structure.

[0005] To achieve the above object, the present invention provides a video implicit representation method based on decoupled spatial and temporal information, comprising the following steps:

[0006] S1. Train a specific video clip and optimize the network parameters by reducing the network loss function until the network converges, obtaining the network weights for efficient implicit video representation based on decoupled spatial and temporal information;

[0007] S2, using the network weights obtained in the training phase, reconstruct the entire video sequence frame by frame under the input of a given frame index;

[0008] S3. Compare the reconstructed video sequence with the original video to evaluate the ability of implicit representation.

[0009] Preferably, the step S1 specifically includes the following steps:

[0010] S11, executing the training video clip to construct a network training data pair and determining the network input frame index value corresponding to any frame image;

[0011] S12, transforming the input frame index value into a high-dimensional time series feature through a sine-cosine transform unit, and then mapping the transformed high-dimensional time series feature into a one-dimensional time series feature through a multi-layer perceptron network;

[0012] S13, using two-dimensional spatial pixel coordinates to initialize the training video clip to obtain a coordinate map, and then executing the first self-attention unit after the sine-cosine transform unit to perform information interaction on the spatial scale to obtain a spatial feature map;

[0013] S14, multiplying the temporal feature and the spatial feature map position by position in space to obtain a first spatiotemporal feature map;

[0014] S15, executing a second self-attention unit on the first spatiotemporal feature map to perform a second spatial position interaction to expand the equivalent fused temporal information of each spatial position to obtain a second spatiotemporal feature map;

[0015] S16, sending the high-dimensional time series features to another independent shallow perceptron network to execute the time series distribution offset feature encoding unit to obtain an offset feature vector used in the convolution module;

[0016] S17, sending the offset feature vector and the second spatiotemporal feature map into a convolutional network module to obtain a video frame image;

[0017] S18. Perform network training in an end-to-end manner to obtain the loss function value of gradient backpropagation optimization;

[0018] S19. Use the Adam optimizer to iteratively update the network parameters to reduce the target loss function until it converges to the local optimum to obtain the trained network weights of the implicit video representation method based on decoupled spatial and temporal information.

[0019] Preferably, the training video clip is a long video clip, and the video resolution is fixed at 1280×720.

[0020] Preferably, the number of dimensions of the one-dimensional temporal feature and the spatial feature map is 256.

[0021] Preferably, the first self-attention unit is single-head self-attention, and the second self-attention unit is multi-head attention.

[0022] Preferably, the convolutional network module is five convolutional network modules connected in series, and each convolutional network module consists of an instance regularization module, a 3×3 convolution, a pixel-shuffle upsampling module, and a 3×3 convolution.

[0023] Preferably, the instance regularization module is a two-layer fully connected layer network, and the instance regularization module offsets the distribution of the feature map by obtaining the mean and variance through the input offset feature vector.

[0024] Preferably, the pixel-shuffle upsampling module deforms the input feature map of size (C·r·r)×h×w into C×(h·r)×(w·r).

[0025] Preferably, the step S18 is specifically as follows: using a norm distance loss function L1, an image similarity loss function L SSIM Perform joint training, with the weights of the two loss functions being 0.7 and 0.3 respectively, to obtain the final total loss function L total ; The mathematical expression of each loss is:

[0026] L1(J t ,I t )=|J t -I t |

[0027] L SSIM (J t ,I t )=1-SSIM(J t ,I t )

[0028] L total =0.7×L1(J t ,I t )+0.3×L2(J t ,I t )

[0029] Among them, J t and I t They represent the network's image output for the frame and the true image value of the frame, respectively. The function SSIM(a,b) calculates the structural similarity of the two input images. All calculations are performed pixel by pixel in space, and then the average is taken over all pixel positions and all frames of the video to obtain the loss function value of the gradient backpropagation optimization.

[0030] Preferably, the step S2 specifically includes the following steps:

[0031] S21, inputting a frame index value of a specific frame from 0 to 1 for the video representation network after training and optimization;

[0032] S22, executing a sine-cosine transform unit to obtain a high-dimensional feature corresponding to the frame index value;

[0033] S23, sending the high-dimensional features to two different multi-layer perceptron networks respectively, executing the time series feature encoding unit and the time series distribution offset feature encoding unit to obtain the required time series features and the distribution offset features used for calculation in the subsequent convolution module;

[0034] S24, executing the first self-attention unit on the spatial pixel coordinates to obtain a spatial feature map that is equal to the input index value of any frame of the video;

[0035] S25, perform position-by-position feature multiplication on the temporal feature and the spatial feature map, and obtain the spatiotemporal feature map through the second self-attention unit;

[0036] S26. Perform a convolution upsampling unit on the obtained spatiotemporal feature map, perform feature regularization and convolution upsampling on the distribution offset feature in each module to obtain a feature map;

[0037] S27. Pass the feature map through a 1×1 convolution layer to reduce the number of channels to the required three-channel color output, and use the sigmoid function to normalize the output to between 0 and 1, and finally multiply it by 255 to obtain the output three-channel color.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] The present invention provides an efficient implicit video representation method based on decoupled spatial and temporal information. By achieving decoupling and independently generating temporal features and spatial features of the video and then performing information interaction in the spatial and feature dimensions, the method greatly saves the number of parameters required for the network, avoids the large number of redundant parameters required to directly generate spatiotemporal feature maps through a multi-layer perceptron in the original structure, and retains most of the model's performance. By introducing temporal features in the convolution module in the form of distributed offset, the network's representation capability is further improved. At the same time, the optimized convolution upsampling structure further saves the occupied parameter space while ensuring that the performance remains unchanged. This method can accurately represent video content in many different virtual and real videos with different contents, and the representation results and optimization convergence speed are significantly better than previous baseline methods, showing good results. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Schematic diagram of a network framework of a video implicit representation method based on decoupled spatial and temporal information according to the present invention;

[0041] Figure 2 Comparison of the process of generating spatiotemporal feature maps by an efficient implicit video representation method based on decoupled spatial and temporal information and a baseline method;

[0042] Figure 3 The figure is a comparison of the video representation results of the method of the present invention and the baseline method. DETAILED DESCRIPTION

[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0044] In response to the problems of multiple parameters and structural redundancy caused by the simultaneous mixed representation of spatial and temporal information in existing neural network-based implicit video representation methods, the present invention proposes an efficient implicit video representation method based on decoupling spatial and temporal information. The method aims to remove a large number of redundant parts and parameters in the original network structure by decoupling spatial and temporal information using different structures, while retaining most of the representation performance. On this basis, the saved parameters are used to increase the number of channels in the entire network to further improve the representation performance and convergence speed. Ultimately, the entire frame-by-frame implicit video representation network has fewer parameters than previous methods, but better representation performance and optimized convergence speed.

[0045] The present invention proposes a method for implicitly representing a video based on decoupled spatial and temporal information, comprising the following steps:

[0046] S1. Train a specific video clip and optimize the network parameters by reducing the network loss function until the network converges, obtaining the network weights for efficient implicit video representation based on decoupled spatial and temporal information;

[0047] S2, using the network weights obtained in the training phase, reconstruct the entire video sequence frame by frame under the input of a given frame index;

[0048] S3. Compare the reconstructed video sequence with the original video to evaluate the ability of implicit representation.

[0049] The present invention utilizes a server to perform the above steps. The server includes multiple functional modules, such as a sine-cosine transform unit, a temporal feature encoding unit, a temporal distribution offset feature encoding unit, a self-attention unit, a spatiotemporal feature self-attention calculation unit, and a convolutional upsampling unit. Furthermore, the server also includes a training video segment construction unit.

[0050] Each functional unit is as follows:

[0051] (a) A training video segment construction unit; used to construct data pairs required for training from long video segments, calculate the input normalization value based on the frame index, and scale the video frame image size to the required size;

[0052] (b) Sine-cosine transform unit: used to transform the input frame index and two-dimensional spatial coordinates to obtain high-dimensional feature input, which can make the network converge better;

[0053] (c) a temporal feature encoding unit for encoding the required high-dimensional temporal features from the high-dimensional input after frame index transformation;

[0054] (d) Temporal distribution offset feature encoding unit; another independent branch, which obtains the fine-tuned distribution offset features used in the convolutional upsampling unit from the high-dimensional input encoding after frame index transformation;.

[0055] (e) Self-attention unit; used to encode spatial features from the transformed coordinate map, perform self-attention calculation and feature interaction on the spatial scale to obtain the spatial feature S;

[0056] (f) Spatiotemporal feature self-attention calculation unit; used to perform information interaction on the spatial dimension and feature dimension of the multiplied spatiotemporal feature map to obtain the required spatiotemporal feature map f for the current frame t ;

[0057] (g) Convolutional upsampling unit; used to perform feature interaction and upsampling transformation on the spatiotemporal feature map, while introducing the influence of temporal distribution offset features in the form of instance regularization, and finally restoring the image of the original size.

[0058] The following is a detailed description of each step.

[0059] Step S1: train a specific video clip, optimize the network parameters by reducing the network loss function until the network converges, and obtain the network weights of efficient implicit video representation based on decoupled spatial and temporal information. Figure 1 The figure shows a network framework diagram of a video implicit representation method based on decoupled spatial and temporal information of the present invention, wherein the decoupled temporal feature takes the current frame index as input, and obtains a one-dimensional temporal feature through sine-cosine transform and multi-layer perceptron network; the spatial feature comes from a fixed spatial coordinate map, which is obtained after the same transformation and spatial self-attention feature interaction; the decoupled temporal and spatial features are multiplied and then passed through a self-attention module to obtain a mixed spatiotemporal feature map; five convolution modules are connected in series to upsample the feature map; the distribution offset of the convolution module is obtained by inputting the frame index through another independent branch; in the convolution module, the temporal information is first used for distribution offset, that is, the influence of temporal information is introduced again, and then the cascaded convolution and pixel-shuffle modules are used for feature interaction and upsampling to expand the feature map size; finally, the network outputs the corresponding image of the frame at the required scale, and the specific steps are as follows:

[0060] S11, executing the training video clip to construct a network training data pair and determining the network input frame index value corresponding to any frame image;

[0061] Specifically, the server is used to execute the training video segment construction unit. The training set of a single network of the implicit video representation method is the entire video. At the same time, the frame-by-frame generation paradigm requires a fixed image size for each frame of the generated video. A training video of length T is used, and the video resolution is fixed to 1280×720. In order to reflect the superiority of the frame-by-frame method over the pixel-by-pixel method, a long video segment is selected, where T≥100. The server is used to execute the training video segment construction unit to construct the training data pair, determine the network input frame index value corresponding to any frame image, and for any frame I t , t represents the position of the frame in the entire video sequence, and the input of the network is the normalized frame index value This constructs the network training data pair

[0062] S12, transforming the input frame index value into a high-dimensional time series feature through a sine-cosine transform unit, and then mapping the transformed high-dimensional time series feature into a one-dimensional time series feature through a multi-layer perceptron network;

[0063] Specifically, the server executes the temporal feature encoding unit. For the input frame index, a fixed sine-cosine transform unit γ is first executed to transform the input one-dimensional index into a high-dimensional feature. In implicit representation tasks, this step can significantly improve the learning ability of the multi-layer perceptron network. Then, a very small multi-layer perceptron network F maps the transformed high-dimensional features into a one-dimensional temporal feature with a dimension of 256. The features obtained at this time can be regarded as the temporal features of the video to be represented at a specific moment.

[0064] S13, using two-dimensional spatial pixel coordinates to initialize the training video clip to obtain a coordinate map, and then executing the first self-attention unit after the sine-cosine transform unit to perform information interaction on the spatial scale to obtain a spatial feature map;

[0065] Specifically, for the spatial information of the video, two-dimensional spatial pixel coordinates are used for initialization to obtain a 16×9×2 coordinate map S0, which is one-eightieth of the size of the image that needs to be restored. After executing the same sine-cosine transform unit γ, the server executes the self-attention unit Φ to perform information interaction on the spatial scale to obtain the spatial feature map S of the video. In the self-attention module, the feature dimension is increased to 256, which is consistent with the temporal feature.

[0066] S14, multiplying the temporal feature and the spatial feature map position by position in space to obtain a first spatiotemporal feature map;

[0067] Specifically, the outputs of steps S12 and S13 are the required decoupled temporal and spatial information representations. Since the feature dimensions of the two are consistent, the server is used to perform spatial multiplication position by position to obtain the required spatiotemporal feature map.

[0068] S15, executing a second self-attention unit on the first spatiotemporal feature map to perform a second spatial position interaction to expand the equivalent fused temporal information of each spatial position to obtain a second spatiotemporal feature map;

[0069] Specifically: Use the server to execute the spatiotemporal feature self-attention unit F θ , perform the second spatial position interaction, expand the time series information equivalently fused for each spatial position in step S14, and obtain the final required spatiotemporal feature map f t ,like Figure 2 As shown, this is a comparison diagram of the process of generating spatiotemporal feature maps by an efficient implicit video representation method based on decoupled spatial and temporal information of the present invention and a baseline method, wherein the upper part is the process of the baseline method, and the lower part is the method of generating by decoupling the temporal and spatial information of the present invention. From the comparison, it can be seen that the main bottleneck of the baseline method comes from the fact that in order to directly use a multi-layer perceptron to generate feature maps of the required dimensions, the number of neurons in the last layer of the multi-layer perceptron needs to be 112×16×9=16128. This layer will take up a large amount of space and parameters. Since the performance of the network is mainly guaranteed by the number of channels, simply reducing the network parameters and then using 1×1 convolution to increase the number of channels will lead to a sharp drop in performance. In comparison, the decoupled representation of the present invention avoids directly generating a large number of feature representations. At the same time, experimental results show that this improvement retains most of the original performance while reducing a large number of parameters.

[0070] S16, sending the high-dimensional time series features to another independent shallow perceptron network to execute the time series distribution offset feature encoding unit to obtain the offset feature vector used in the convolution module;

[0071] Specifically, the server sends the high-dimensional time series features through the sine-cosine mapping γ into another independent shallow perceptron network, executes the time series distribution offset feature encoding unit to obtain the offset feature vector l used in the convolution module t .

[0072] S17, sending the offset feature vector and the second spatiotemporal feature map into a convolutional network module to obtain a video frame image;

[0073] Specifically, the server executes the convolution upsampling unit and feeds the spatiotemporal feature map into five convolutional network modules in series. Each network module consists of a serial instance regularization module, a 3×3 convolution, a pixel-shuffle upsampling module, and a 3×3 convolution. In the instance regularization module, the offset feature vector l t The required mean and variance are obtained to shift the distribution of the feature map. The designed 3×3 convolution-pixel-shuffle upsampling module-3×3 convolution structure introduces an intermediate feature scale compared to the 3×3 convolution-pixel-shuffle upsampling module of the original baseline method, which can reduce the required number of parameters by a quarter while maintaining comparable performance. The upsampling scales of the five convolution modules are 5, 2, 2, 2, 2, respectively, and the required 1280×720 video frame image I is finally obtained. t .

[0074] S18. Perform network training in an end-to-end manner to obtain the loss function value of gradient backpropagation optimization;

[0075] Specifically: use the server to train the network in an end-to-end manner; use the one-norm distance loss function L1 and the image similarity loss function L SSIM Perform joint training, with the weights of the two loss functions being 0.7 and 0.3 respectively, to obtain the final total loss function L total ; The mathematical expression of each loss is:

[0076] L1(J t ,I t )=|J t -I t |

[0077] L SSIM (J t ,I t )=1-SSIM(J t ,I t )

[0078] L total =0.7×L1(J t ,I t )+0.3×L2(J t ,I t )

[0079] Among them, J t and I tThe SSIM(a,b) function calculates the structural similarity between two input images. The loss function is calculated by averaging across all pixel locations and all video frames to obtain the gradient backpropagation loss.

[0080] S19. Use the Adam optimizer to iteratively update the network parameters to reduce the target loss function until it converges to the local optimum to obtain the trained network weights of the implicit video representation method based on decoupled spatial and temporal information;

[0081] Specifically: optimize the objective function to obtain the local optimal network parameters; convert the total loss function L in step S18 total As the objective function, the Adam optimizer is used to iteratively update the network parameters to reduce the target loss function until it converges to the local optimum. At this point, the training is completed and the trained network weights of the efficient video implicit representation method based on decoupled spatial and temporal information are obtained.

[0082] Step S2: Using the network weights obtained in the training phase, the entire video sequence is reconstructed frame by frame under the input of a given frame index. This specifically includes the following steps:

[0083] S21. For the video representation network after training optimization, input the frame index value of a specific frame from 0 to 1

[0084] S22, executing a sine-cosine transform unit to obtain a high-dimensional feature corresponding to the frame index value;

[0085] S23, sending the high-dimensional features to two different multi-layer perceptron networks respectively, executing the time series feature encoding unit and the time series distribution offset feature encoding unit to obtain the required time series features and the distribution offset features used for calculation in the subsequent convolution module;

[0086] S24, the spatial pixel coordinates are executed by the self-attention unit to obtain a spatial feature map S that is equal to the index value input of any frame of the video;

[0087] S25, perform position-by-position feature multiplication, and obtain the required spatiotemporal feature map f through the second self-attention unit. t ;

[0088] S26, performing a convolution upsampling unit on the spatiotemporal feature map in step S25, performing feature regularization within each module and the distribution offset feature calculation in step S23, and performing convolution upsampling;

[0089] S27, the final output feature map scale is the required image scale, and then a 1×1 convolution layer is used to reduce the number of channels to the required three-channel color output, and the sigmoid function is used to normalize the output to between 0 and 1, and finally multiplied by 255 to obtain the output three-channel color. Figure 3 The figure shows the comparison of the results of the method of the present invention and the baseline method in video representation. It can be seen that the method of the present invention can better restore the characteristics of the video in some detail areas.

[0090] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the invention. It should be understood that many modifications may be made to the illustrative embodiments, and that other arrangements may be devised, without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in ways other than those described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be employed in conjunction with other described embodiments.

Claims

1. A video implicit representation method based on decoupled spatial and temporal information, characterized in that: The following steps are involved: S1. Train a specific video clip and optimize the network parameters by reducing the network loss function until the network converges. Obtaining network weights for efficient implicit video representation based on decoupled spatial and temporal information; S2, using the network weights obtained in the training phase, reconstruct the entire video sequence frame by frame under the input of a given frame index; S3. Compare the reconstructed video sequence with the original video to evaluate the implicit representation capability. The step S1 specifically includes the following steps: S11, executing the training video clip to construct a network training data pair and determining the network input frame index value corresponding to any frame image; S12, transforming the input frame index value into a high-dimensional time series feature through a sine-cosine transform unit, and then mapping the transformed high-dimensional time series feature into a one-dimensional time series feature through a multi-layer perceptron network; S13, using two-dimensional spatial pixel coordinates to initialize the training video clip to obtain a coordinate map, and then executing the first self-attention unit after the sine-cosine transform unit to perform information interaction on the spatial scale to obtain a spatial feature map; S14, multiplying the temporal feature and the spatial feature map position by position in space to obtain a first spatiotemporal feature map; S15, executing a second self-attention unit on the first spatiotemporal feature map to perform a second spatial position interaction to expand the equivalent fused temporal information of each spatial position to obtain a second spatiotemporal feature map; S16, sending the high-dimensional time series features to another independent shallow perceptron network to execute the time series distribution offset feature encoding unit to obtain an offset feature vector used in the convolution module; S17, sending the offset feature vector and the second spatiotemporal feature map into a convolutional network module to obtain a video frame image; S18. Perform network training in an end-to-end manner to obtain the loss function value of gradient backpropagation optimization; S19. Use the Adam optimizer to iteratively update the network parameters to reduce the target loss function until it converges to the local optimum to obtain the trained network weights of the implicit video representation method based on decoupled spatial and temporal information; The step S2 specifically includes the following steps: S21, inputting a frame index value of a specific frame from 0 to 1 for the video representation network after training and optimization; S22, executing a sine-cosine transform unit to obtain a high-dimensional feature corresponding to the frame index value; S23, sending the high-dimensional features to two different multi-layer perceptron networks respectively, executing the time series feature encoding unit and the time series distribution offset feature encoding unit to obtain the required time series features and the distribution offset features used for calculation in the subsequent convolution module; S24, executing the first self-attention unit on the spatial pixel coordinates to obtain a spatial feature map that is equal to the input index value of any frame of the video; S25, perform position-by-position feature multiplication on the temporal feature and the spatial feature map, and obtain the spatiotemporal feature map through the second self-attention unit; S26. Perform a convolution upsampling unit on the obtained spatiotemporal feature map, perform feature regularization and convolution upsampling on the distribution offset feature in each module to obtain a feature map; S27. Pass the feature map through a 1×1 convolution layer to reduce the number of channels to the required three-channel color output, and use the sigmoid function to normalize the output to between 0 and 1, and finally multiply it by 255 to obtain the output three-channel color.

2. The method for implicitly representing a video based on decoupled spatial and temporal information according to claim 1, characterized in that: The training video clip is a long video clip, and the video resolution is fixed at 1280×720.

3. The method for implicitly representing a video based on decoupled spatial and temporal information according to claim 1, characterized in that: The number of dimensions of the one-dimensional temporal feature and the spatial feature map is 256.

4. The method for implicitly representing a video based on decoupled spatial and temporal information according to claim 1, characterized in that: The first self-attention unit is single-head self-attention, and the second self-attention unit is multi-head attention.

5. The method for implicitly representing a video based on decoupled spatial and temporal information according to claim 1, characterized in that: The convolutional network module is five convolutional network modules connected in series, and each convolutional network module consists of an instance normalization module, a 3×3 convolution, a pixel-shuffle upsampling module, and a 3×3 convolution.

6. The method for implicitly representing a video based on decoupled spatial and temporal information according to claim 5, characterized in that: The instance regularization module is a two-layer fully connected layer network, and the instance regularization module offsets the distribution of the feature map by obtaining the mean and variance through the input offset feature vector.

7. The method for implicitly representing a video based on decoupled spatial and temporal information according to claim 5, characterized in that: The pixel-shuffle upsampling module deforms the input feature map of size (C·r·r)×h×w into C×(h·r)×(w·r).

8. The method for implicitly representing a video based on decoupled spatial and temporal information according to claim 1, characterized in that: The step S18 is specifically as follows: using a norm distance loss function L1, an image similarity loss function L SSIM Perform joint training, with the weights of the two loss functions being 0.7 and 0.3 respectively, to obtain the final total loss function L total ; The mathematical expression of each loss is: L1(I t ,I t )=|I t -I t | L SSIM (I t ,I t )=1-SSIM(I t ,I t ) L total =0.7×L1(I t ,I t )+0.3×L2(I t ,I t ) Among them, I t and I t They represent the network's image output for the frame and the true image value of the frame, respectively. The function SSIM(a,b) calculates the structural similarity of the two input images. All calculations are performed pixel by pixel in space, and then the average is taken over all pixel positions and all frames of the video to obtain the loss function value of the gradient backpropagation optimization.

Images