Deep Learning-Based Video Image Spatial Transformation Method

Abstract

This invention discloses a deep learning-based video image spatial conversion method, comprising the following steps: inputting video information into a feature extraction network to obtain multi-scale feature information of time and content; inputting the multi-scale feature information into a multi-scale leaf layer classification network, using the leaf layer classification network to classify the extracted multi-scale feature representations to ensure accurate capture of local and global information of the image at different scales; calculating the correlation volume between input frames through a full pixel association network, and obtaining optical flow estimation through iterative updates; optimizing the video conversion effect in the time and spatial dimensions through contrastive learning; ensuring the overall consistency of the generated video sequence through cycle consistency optimization; generating a target image using a generator, and evaluating the quality of the generated image through a discriminator; and converting the generated high-quality image into a video output.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a video image spatial conversion method based on deep learning. Background Technology

[0002] Night vision enhancement and infrared imaging technologies are crucial components in modern military, security, autonomous driving, and scientific research. These technologies aim to enhance visual perception in low-light or complete darkness environments and convert the infrared spectrum, invisible to the human eye, into visible light images for observation and analysis. Video image spatial transformation technology plays a key role in this application, enabling night vision enhancement and infrared imaging by converting infrared or other spectral images into visible light images. Before the advent of deep learning algorithms, much research on image and video transformation was based on different color channel description methods, aiming to find linear or nonlinear mapping relationships, such as converting grayscale images to color images or adjusting image contrast. However, the distribution of different state spaces does not have a one-to-one correspondence, making transformation impossible using traditional mathematical methods. Furthermore, reinforcement learning algorithms require not only discrete state spaces but also continuous state spaces in their datasets. Therefore, image transformation techniques need to be incorporated, and even more challenging video transformation techniques need to be considered, such as converting images from infrared state space to visible light state space. While not a particularly serious problem, when all the transformation results are combined into a continuous video, even subtle inconsistencies can lead to inaccurate data.

[0003] Despite significant progress in video-image spatial transformation, several challenges and issues remain. During video transformation, the continuity and consistency between frames are often not adequately guaranteed, leading to flickering, distortion, or unnatural transitions of objects in the video after transformation. 1. Especially in high-resolution images and complex scenes, existing techniques are prone to detail loss or image blurring, affecting the quality and visual effect of the transformed image. Existing techniques often require large amounts of labeled data and complex model training processes, placing high demands on computational resources and datasets, limiting the widespread application of these technologies. Generative Adversarial Network (GAN)-based techniques are prone to instability during training, such as mode collapse and training oscillations, resulting in inconsistent quality of generated images. Summary of the Invention

[0004] In view of this, the objective of this invention is to propose a novel video image spatial transformation method using the principles and methods of deep learning, comprising the following steps:

[0005] Step 1: Input the video information into the feature extraction network to obtain multi-scale feature information of time and content;

[0006] Step 2: Input the multi-scale feature information into the multi-scale leaf layer classification network, and use the leaf layer classification network to classify the extracted multi-scale feature representation to ensure that the local and global information of the image can be accurately captured at different scales.

[0007] Step 3: Calculate the correlation volume between input frames using a full pixel association network, and obtain the optical flow estimate through iterative updates;

[0008] Step 4: Optimize video conversion effects in time and space dimensions through comparative learning; ensure overall consistency of the generated video sequence through cycle consistency optimization.

[0009] Step 5: Use the generator to generate the target image and use the discriminator to evaluate the quality of the generated image;

[0010] Step 6: Convert the generated image into a video output.

[0011] Specifically, the input to the feature extraction network is a single-frame video image I with a size of H×W×3, where H and W represent the height and width of the image, respectively, and 3 represents the RGB color channels. The output is a set of multi-scale feature maps. The calculation steps of the feature extraction network include the following:

[0012] Step 101: Perform convolution operation on the input image through the initial convolutional layer to extract basic spatial features and obtain the initial feature map;

[0013] Step 102: Perform multiple convolutions on the initial feature map to generate a series of intermediate feature maps;

[0014] Step 103: Extract features at different scales and generate multi-scale feature maps by using convolution kernels or pooling operations of different sizes;

[0015] Step 104: The feature maps extracted at different scales are fused to generate the final multi-scale feature map.

[0016] Specifically, the classification of the extracted multi-scale feature information using a leaf layer classification network includes the following steps:

[0017] Step 201: Perform convolution operation on the input multi-scale feature map to capture local patterns in the feature map and generate a preliminary leaf layer representation.

[0018] Step 202: Pool the convolutional feature map to reduce its dimensionality while preserving the most important spatial information;

[0019] Step 203: Flatten the pooled feature map and further process it through a fully connected layer to extract the global pattern;

[0020] Step 204: Input the flattened feature vector into the classification layer, and output the class probability of the leaf layer through the Softmax activation function.

[0021] Specifically, the process of calculating the correlation volume between input frames through a full-pixel correlation network and obtaining the optical flow estimate through iterative updates includes the following steps:

[0022] Step 301: Extract the multi-scale feature maps of the input image frames I1 and I2 as the basis for association calculation;

[0023] Step 302: Calculate the correlation of all pixel pairs between input image frame pairs to generate a 4D correlation volume;

[0024] Step 303: Process the 4D related volume at different scales and continuously update the optical flow estimate through a recurrent unit (GRU).

[0025] Step 304: Based on the updated 4D related volume, the updated optical flow estimate is processed by convolution operation to generate the final optical flow field, i.e., the motion vector of each pixel.

[0026] Specifically, in step 302, for each pixel in each feature map, the inner product with all pixels in another feature map is calculated to generate a 4D related volume C, the calculation formula of which is:

[0027]

[0028] Among them, C ijkl Let f1(i,j,h) represent the correlation between the (i,j)th pixel in I1 and the (k,l)th pixel in I2, where h is the index of the feature dimension, f1 and f2 are the multi-scale feature maps extracted from I1 and I2 respectively, f1(i,j,h) represents the feature value of the (i,j)th pixel in I1 at the h-th feature dimension, and f2(k,l,h) represents the feature value of the (k,l)th pixel in I2 at the h-th feature dimension;

[0029] In step 303, relevant volumes at different scales are matched and associated, and the optical flow estimate is updated iteratively. The relevant volume C is pooled to generate multi-scale relevant volumes at different resolutions. A recurrent neural network (GRU) is used to iteratively update the relevant volumes. The optical flow estimate is updated each time, and the calculation formula is: f k+1 =f k +Δf k , where f k Let Δf represent the optical flow estimate in the k-th iteration. k It is the updated optical flow increment, f k+1 This represents the optical flow estimate in the (k+1)th iteration.

[0030] Furthermore, the generation of the 4D related volume includes the following steps:

[0031] Extract the feature vector for each pixel location from the multi-scale feature maps of two consecutive image frames I1 and I2: Where f1(i,j) represents the feature vector at position (i,j) in image frame I1, f2(k,l) represents the feature vector at position (k,l) in image frame I2, and D is the feature dimension. Represent real numbers;

[0032] The similarity between two feature vectors is calculated using the inner product as a measure of similarity; for each pair of pixel positions (i,j) and (k,l), the inner product of their feature vectors is calculated. Among them, C ijkl It is an element in the 4D correlated volume, representing the similarity between pixel pairs (i,j) and (k,l);

[0033] The similarity calculation results of all pixel pairs are organized into a 4D tensor, namely the 4D correlation volume; by traversing all pixel positions in I1 and I2, the 4D correlation volume C of the entire image frame is calculated.

[0034] Pooling operations are used to generate 4D correlated volumes at different resolutions to capture motion information at different scales; pooling operations are performed on the 4D correlated volume C to generate 4D correlated volumes C at different resolutions. s , Among them, C s It is the pooled 4D related volume, which contains pixel association information at lower resolution.

[0035] Furthermore, step 304, generating the final optical flow field, includes the following steps:

[0036] Obtain 4D related volume C at different resolutions s ;

[0037] Initialize the optical flow estimate f0, typically as a zero vector or based on the optical flow field of the previous frame;

[0038] The optical flow estimate is iteratively updated at different scales using a recurrent neural network (GRU) to gradually approximate the true optical flow field; at each scale s, the current optical flow estimate f is used. k The corresponding pixel position, from the 4D related volume C s Retrieve relevant information from the data and update the optical flow estimate using GRU units;

[0039] After updating the optical flow estimate at the current scale, it is upsampled to a higher resolution so that iterative updates can continue at higher scales; after completing the optical flow update at each scale s, the optical flow estimate is upsampled to a higher resolution using bilinear interpolation or convolutional upsampling. It is the optical flow estimate after sampling at scale s;

[0040] After iterative updates at all scales are completed, the final optical flow estimation result is output; after the final optical flow update at the highest resolution, the final optical flow field f is output. final .

[0041] Preferably, the optical flow estimation includes the following steps:

[0042] Based on the updated optical flow estimate f final The optical flow field is further refined using matching information from the relevant volumes to ensure high-precision motion estimation; for each pixel location (i,j), the updated optical flow estimate f is used. final (i,j) locate the corresponding position in feature map f2 and extract the corresponding matching information from the relevant volume C; Among them, f refined (i,j) is the refined optical flow vector. The Refine operation refines the optical flow based on information in the relevant volume.

[0043] The refined optical flow field is reverse-mapped and checked for consistency to ensure bidirectional consistency of the optical flow; for each pixel position (i,j), according to the optical flow vector f refined (i,j) calculate its corresponding position in the previous frame (i ′ ,j ′ Check the consistency between forward and reverse optical flow to ensure that the relative error is within a certain range;

[0044] Smoothing and regularization operations are applied to eliminate noise and discontinuities in the optical flow field and ensure its smoothness; Gaussian filtering or global optimization methods are used to smooth the refined optical flow field.

[0045] The optical flow field after refinement, consistency check, and smoothing is the final output optical flow field.

[0046] Specifically, the optimization of video conversion effects in the temporal and spatial dimensions through comparative learning includes the following steps:

[0047] Calculate the temporal similarity loss between consecutive frames to ensure smoothness and consistency of motion between frames; for two adjacent frames I t and I t+1 Based on the optical flow estimation f(t), I t+1 eigenvector mapping back to It The feature space is used to calculate the similarity loss. Where F1(t) and F2(t+1) represent the video frame feature maps at times t and t+1, respectively. It is the temporal similarity loss, where T is the total duration of the video;

[0048] Based on temporal similarity loss, the generator is optimized through backpropagation to adjust the parameters of the video conversion model; the Adam optimization algorithm is used to minimize the temporal similarity loss and adjust the generator parameters.

[0049] Calculate the spatial similarity loss within the same frame to ensure the consistency of spatial structure and details within the frame;

[0050] In each frame I t Within the range, select multiple pixel pairs (i1,j1) and (i2,j2) and calculate the similarity between the feature vectors of these pixel pairs; Where F1(t,i1,j1) represents the feature vector of position (i1,j1) at time t. Spatial similarity loss;

[0051] Based on spatial similarity loss, the generator is optimized through backpropagation, and the parameters of the video conversion model are adjusted; the spatial similarity loss is minimized using an optimization algorithm, and the generator parameters are adjusted accordingly.

[0052] By combining temporal and spatial similarity losses, a final optimization objective is formed; and by combining temporal and spatial similarity losses, a total loss function is constructed. Where, λ time and λ space These are the weights for temporal similarity and spatial similarity losses, respectively.

[0053] Minimize the total loss function and optimize all parameters of the video conversion model.

[0054] Specifically, ensuring the overall consistency of the generated video sequence through cycle consistency optimization includes the following steps:

[0055] Use generator G to convert the original image frame I t Convert to image frames in the target state

[0056] Using the inverse generator G -1 The generated image frame By converting back to the original state, the reconstructed image frame is obtained.

[0057] Calculate the original image frame I t and reconstructed image frames The pixel-level differences between them are used as the cycle consistency loss;

[0058] Minimize cycle consistency loss through backpropagation Among them, I t (i,j) and Representing image frame I respectively t and Pixel value optimization generator G and inverse generator G at position (i,j) -1 The parameters.

[0059] Specifically, step 5, which involves generating the target image using a generator, includes the following steps:

[0060] The generator first extracts multi-scale features from the input image, and then combines the classification results of the leaf layer classification network to perform structural constraints and fusion on these features to ensure that the generator preserves the spatial geometric features and hierarchical structure of the image when generating the image.

[0061] After incorporating spatial structure information, the generator further references the optical flow field information provided by the full pixel association network. Guided by optical flow, the target image generated by the generator maintains motion consistency in the time series and performs motion compensation and adjustment on dynamically changing parts.

[0062] Finally, the generator utilizes information fused from spatial structure and temporal dynamics to generate the target image through a series of convolution and deconvolution operations.

[0063] Specifically, step 5, which involves evaluating the quality of the generated image using a discriminator, includes the following steps:

[0064] Spatial structure check: The discriminator compares the features of the generated image with the output of the leaf layer classification network to check whether the image is consistent with the original image in terms of spatial structure.

[0065] In the temporal dynamic inspection, the discriminator uses optical flow information from the full pixel association network to evaluate the motion consistency between the generated image and adjacent frames;

[0066] The discriminator comprehensively evaluates the results of spatial structure inspection and temporal dynamics inspection to generate a final discrimination result and outputs an image quality score. If the generated image performs well in both aspects, the discriminator will give a high quality score, otherwise it will give a low quality score.

[0067] The beneficial effects of this invention are as follows: Each pixel in a video frame is considered as a point on a complex projective plane. By constructing leaf layer models of different dimensions, more detailed classification and feature extraction of moving objects in the video are performed. The correlation between each pair of pixels is calculated. In each video frame, a multi-scale 4D correlation volume is constructed to perform correlation calculations on all pixels, and a recurrent neural network is used to iteratively update the data to optimize the mapping between state spaces. The mutual information between pixels is enhanced in the spatial dimension, ensuring that the converted frames maintain consistency in detail. Simultaneously, the optimization in the temporal dimension further enhances the smooth transition between frames through a convolutional neural network, avoiding visual distortion caused by temporal inconsistencies. This application maintains excellent spatiotemporal consistency, meeting the practical application requirements of video conversion. Attached Figure Description

[0068] Figure 1 A flowchart illustrating the overall process of video spatial conversion. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0070] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0071] This embodiment proposes a video image spatial transformation method based on deep learning, including the following steps:

[0072] Step 1: Input the video information into the feature extraction network to obtain multi-scale feature representations of time and content information;

[0073] Step 2: Input the multi-scale feature information into the multi-scale leaf layer classification network, and use the leaf layer classification network to classify the extracted multi-scale feature representation to ensure that the local and global information of the image can be accurately captured at different scales.

[0074] Step 3: Calculate the correlation volume between input frames using a full pixel association network, and obtain the optical flow estimate through iterative updates;

[0075] Step 4: Optimize video conversion effects in time and space dimensions through comparative learning; ensure overall consistency of the generated video sequence through cycle consistency optimization.

[0076] Step 5: Use the generator to generate the target image and use the discriminator to evaluate the quality of the generated image;

[0077] Step 6: Convert the generated image into a video output.

[0078] Feature extraction networks typically consist of multiple convolutional layers, batch normalization layers, and activation function layers (such as ReLU). These layers extract and process information from the input image layer by layer, transforming the original image into a high-dimensional feature representation. The network structure can be divided into the following main parts: Initial convolutional layers: used to capture basic image features, such as edges and corners; Intermediate convolutional layers: further extract higher-level features, such as texture and shape; Multi-scale feature layers: extract features of the image at different scales through convolutional kernels of different sizes and pooling operations; Feature fusion layers: fuse features from different scales to generate the final high-dimensional feature vector.

[0079] Specifically, the input to the feature extraction network is a single-frame video image I with a size of H×W×3, where H and W represent the height and width of the image, respectively, and 3 represents the RGB color channels. The output is a set of multi-scale feature maps. The calculation steps of the feature extraction network include the following:

[0080] Step 101: Perform convolution operation on the input image through the initial convolutional layer to extract basic spatial features and obtain the initial feature map;

[0081] Step 102: Perform multiple convolutions on the initial feature map to generate a series of intermediate feature maps;

[0082] Step 103: Extract features at different scales and generate multi-scale feature maps by using convolution kernels or pooling operations of different sizes;

[0083] Step 104: The feature maps extracted at different scales are fused to generate the final multi-scale feature map.

[0084] The primary function of the leaf layer classification network is to further process the multi-scale features generated by the feature extraction network, identifying and classifying different structural features in the image. These features are treated as leaf layers (i.e., blades on the complex projective plane), and the network classifies them by identifying properties of these leaf layers (such as whether they are convex). The classification results will guide subsequent video image conversion processes, resulting in converted images that are more accurate in structure and detail.

[0085] Specifically, the classification of the extracted multi-scale feature information using a leaf layer classification network includes the following steps:

[0086] Step 201: Perform convolution operation on the input multi-scale feature map to capture local patterns in the feature map and generate a preliminary leaf layer representation.

[0087] Step 202: Pool the convolutional feature map to reduce its dimensionality while preserving the most important spatial information;

[0088] Step 203: Flatten the pooled feature map and further process it through a fully connected layer to extract the global pattern;

[0089] Step 204: Input the flattened feature vector into the classification layer, and output the class probability of the leaf layer through the Softmax activation function.

[0090] The classification results are achieved by performing leaf-level classification on each pixel in the input feature map, outputting either a class probability distribution or a class label. These classification results will be used to guide subsequent video image conversion, ensuring that the structure and details of the original image are accurately preserved during the conversion process.

[0091] The core function of the full pixel association network is to perform global association calculations on all pixels in the input image frame pair, thereby accurately estimating optical flow, that is, the motion vector of each pixel between frames.

[0092] Specifically, the process of calculating the correlation volume between input frames through a full-pixel correlation network and obtaining the optical flow estimate through iterative updates includes the following steps:

[0093] Step 301: Extract the multi-scale feature maps of the input image frames I1 and I2 as the basis for association calculation;

[0094] Step 302: Calculate the correlation of all pixel pairs between input image frame pairs to generate a 4D correlation volume;

[0095] Step 303: Process the 4D related volume at different scales and continuously update the optical flow estimate through a recurrent unit (GRU).

[0096] Step 304: Based on the updated 4D related volume, the updated optical flow estimate is processed by convolution operation to generate the final optical flow field, i.e., the motion vector of each pixel.

[0097] Specifically, in step 302, for each pixel in each feature map, the inner product with all pixels in another feature map is calculated to generate a 4D related volume C, the calculation formula of which is:

[0098]

[0099] Among them, C ijklLet f1(i,j,h) represent the correlation between the (i,j)th pixel in I1 and the (k,l)th pixel in I2, where h is the index of the feature dimension, f1 and f2 are the multi-scale feature maps extracted from I1 and I2 respectively, f1(i,j,h) represents the feature value of the (i,j)th pixel in I1 at the h-th feature dimension, and f2(k,l,h) represents the feature value of the (k,l)th pixel in I2 at the h-th feature dimension;

[0100] In step 303, relevant volumes at different scales are matched and associated, and the optical flow estimate is updated iteratively. The relevant volume C is pooled to generate multi-scale relevant volumes at different resolutions. A recurrent neural network (GRU) is used to iteratively update the relevant volumes. The optical flow estimate is updated each time, and the calculation formula is: f k+1 =f k +Δf k , where f k Let Δf represent the optical flow estimate in the k-th iteration. k It is the updated optical flow increment, f k+1 This represents the optical flow estimate in the (k+1)th iteration.

[0101] Generating the 4D correlated volume is one of the core steps, used to capture the interrelationships between all pixel pairs between two consecutive image frames. This process generates a 4D tensor containing global correlation information by calculating the similarity between the feature vectors of each pixel.

[0102] Furthermore, the generation of the 4D related volume includes the following steps:

[0103] Extract the feature vector for each pixel location from the multi-scale feature maps of two consecutive image frames I1 and I2: Where f1(i,j) represents the feature vector at position (i,j) in image frame I1, f2(k,l) represents the feature vector at position (k,l) in image frame I2, and D is the feature dimension. Represent real numbers;

[0104] The similarity between two feature vectors is calculated using the inner product as a measure of similarity; for each pair of pixel positions (i,j) and (k,l), the inner product of their feature vectors is calculated. Among them, C ijkl It is an element in the 4D correlated volume, representing the similarity between pixel pairs (i,j) and (k,l);

[0105] The similarity calculation results of all pixel pairs are organized into a 4D tensor, namely the 4D correlation volume; by traversing all pixel positions in I1 and I2, the 4D correlation volume C of the entire image frame is calculated.

[0106] Pooling operations are used to generate 4D correlated volumes at different resolutions to capture motion information at different scales; pooling operations are performed on the 4D correlated volume C to generate 4D correlated volumes C at different resolutions. s , Among them, C s It is the pooled 4D related volume, which contains pixel association information at lower resolution.

[0107] Furthermore, step 304, generating the final optical flow field, includes the following steps:

[0108] Obtain 4D related volume C at different resolutions s ;

[0109] Initialize the optical flow estimate f0, typically as a zero vector or based on the optical flow field of the previous frame;

[0110] The optical flow estimate is iteratively updated at different scales using a recurrent neural network (GRU) to gradually approximate the true optical flow field; at each scale s, the current optical flow estimate f is used. k The corresponding pixel position, from the 4D related volume C s Retrieve relevant information from the data and update the optical flow estimate using GRU units;

[0111] After updating the optical flow estimate at the current scale, it is upsampled to a higher resolution so that iterative updates can continue at higher scales; after completing the optical flow update at each scale s, the optical flow estimate is upsampled to a higher resolution using bilinear interpolation or convolutional upsampling. It is the optical flow estimate after sampling at scale s;

[0112] After iterative updates at all scales are completed, the final optical flow estimation result is output; after the final optical flow update at the highest resolution, the final optical flow field f is output. final .

[0113] In this novel method for video image spatial transformation, the multi-scale correlation and update step is one of the core processes, used to progressively optimize optical flow estimation. Optical flow estimation is a crucial step in generating the motion vector for each pixel. Based on the correlation volume and optical flow estimation results after multi-scale correlation and update, this step ultimately outputs an accurate optical flow field to guide the video image transformation.

[0114] Preferably, the optical flow estimation includes the following steps:

[0115] Based on the updated optical flow estimate f final The optical flow field is further refined using matching information from the relevant volumes to ensure high-precision motion estimation; for each pixel location (i,j), the updated optical flow estimate f is used. final(i,j) locate the corresponding position in feature map f2 and extract the corresponding matching information from the relevant volume C; Among them, f refined (i,j) is the refined optical flow vector. The Refine operation refines the optical flow based on information in the relevant volume.

[0116] The refined optical flow field is reverse-mapped and checked for consistency to ensure bidirectional consistency of the optical flow; for each pixel position (i,j), according to the optical flow vector f refined (i,j) calculate its corresponding position in the previous frame (i ′ ,j ′ Check the consistency between forward and reverse optical flow to ensure that the relative error is within a certain range;

[0117] Smoothing and regularization operations are applied to eliminate noise and discontinuities in the optical flow field and ensure its smoothness; Gaussian filtering or global optimization methods are used to smooth the refined optical flow field.

[0118] The optical flow field after refinement, consistency check, and smoothing is the final output optical flow field.

[0119] Specifically, the optimization of video conversion effects in the temporal and spatial dimensions through comparative learning includes the following steps:

[0120] Calculate the temporal similarity loss between consecutive frames to ensure smoothness and consistency of motion between frames; for two adjacent frames I t and I t+1 Based on the optical flow estimation f(t), I t+1 eigenvector mapping back to I t The feature space is used to calculate the similarity loss. Where F1(t) and F2(t+1) represent the video frame feature maps at times t and t+1, respectively. It is the temporal similarity loss, where T is the total duration of the video;

[0121] Based on temporal similarity loss, the generator is optimized through backpropagation to adjust the parameters of the video conversion model; the Adam optimization algorithm is used to minimize the temporal similarity loss and adjust the generator parameters.

[0122] Calculate the spatial similarity loss within the same frame to ensure the consistency of spatial structure and details within the frame;

[0123] In each frame I t Within the range, select multiple pixel pairs (i1,j1) and (i2,j2) and calculate the similarity between the feature vectors of these pixel pairs; Where F1(t,i1,j1) represents the feature vector of position (i1,j1) at time t. Spatial similarity loss;

[0124] Based on spatial similarity loss, the generator is optimized through backpropagation, and the parameters of the video conversion model are adjusted; the spatial similarity loss is minimized using an optimization algorithm, and the generator parameters are adjusted accordingly.

[0125] By combining temporal and spatial similarity losses, a final optimization objective is formed; and by combining temporal and spatial similarity losses, a total loss function is constructed. Where, λ time and λ space These are the weights for temporal similarity and spatial similarity losses, respectively.

[0126] Minimize the total loss function and optimize all parameters of the video conversion model.

[0127] Specifically, ensuring the overall consistency of the generated video sequence through cycle consistency optimization includes the following steps:

[0128] Use generator G to convert the original image frame I t Convert to image frames in the target state

[0129] Using the inverse generator G -1 The generated image frame By converting back to the original state, the reconstructed image frame is obtained.

[0130] Calculate the original image frame I t and reconstructed image frames The pixel-level differences between them are used as the cycle consistency loss;

[0131] Minimize cycle consistency loss through backpropagation Among them, I t (i,j) and Representing image frame I respectively t and Pixel value optimization generator G and inverse generator G at position (i,j) -1 The parameters.

[0132] Specifically, step 5, which involves generating the target image using a generator, includes the following steps:

[0133] The generator first extracts multi-scale features from the input image, and then combines the classification results of the leaf layer classification network to perform structural constraints and fusion on these features to ensure that the generator preserves the spatial geometric features and hierarchical structure of the image when generating the image.

[0134] After incorporating spatial structure information, the generator further references the optical flow field information provided by the full pixel association network. Guided by optical flow, the target image generated by the generator maintains motion consistency in the time series and performs motion compensation and adjustment on dynamically changing parts.

[0135] Finally, the generator utilizes information fused from spatial structure and temporal dynamics to generate the target image through a series of convolution and deconvolution operations.

[0136] Specifically, step 5, which involves evaluating the quality of the generated image using a discriminator, includes the following steps:

[0137] Spatial structure check: The discriminator compares the features of the generated image with the output of the leaf layer classification network to check whether the image is consistent with the original image in terms of spatial structure.

[0138] In the temporal dynamic inspection, the discriminator uses optical flow information from the full pixel association network to evaluate the motion consistency between the generated image and adjacent frames;

[0139] The discriminator comprehensively evaluates the results of spatial structure inspection and temporal dynamics inspection to generate a final discrimination result and outputs an image quality score. If the generated image performs well in both aspects, the discriminator will give a high quality score, otherwise it will give a low quality score.

[0140] Leaf layer classification networks and full pixel association networks play different but complementary roles in video image spatial transformation. The relationship between the two can be summarized as follows.

[0141] Division of labor between structure and motion: Leaf layer classification networks primarily focus on the spatial structure of images, i.e., identifying and classifying static geometric features in the image. These feature classifications guide local transformations of the image, ensuring the integrity of the structure and the accuracy of details are maintained during the transformation process. Full pixel association networks, on the other hand, focus on dynamic changes over time, i.e., capturing and calculating the motion of each pixel in the image between frames. Optical flow information is crucial for maintaining the coherence and spatiotemporal consistency between video frames.

[0142] Interaction and Collaborative Work: Leaf Layer Classification Guides Optical Flow Estimation: The structural information provided by the leaf layer classification network can be used to guide the optical flow estimation of the full pixel association network. For example, leaf layer classification results can help determine which regions require higher accuracy in optical flow estimation (such as boundaries and convex regions), thereby optimizing optical flow calculations and avoiding motion artifacts in structurally complex regions. Optical Flow Feedback Improves Leaf Layer Classification: On the other hand, the optical flow field calculated by the full pixel association network can be fed back to the leaf layer classification network to help update and adjust the leaf layer classification results. Especially when dealing with dynamic scenes, optical flow information can help identify moving boundaries or dynamically changing regions in the image, thereby updating the leaf layer classification of these regions and improving the overall classification accuracy.

[0143] The overall transformation effect is improved: the leaf layer classification network ensures geometric fidelity of static structures, while the full pixel association network ensures spatiotemporal consistency in dynamic scenes. The synergistic effect of the two makes the entire video image spatial transformation process more accurate, and the generated images achieve high-quality performance in both static and dynamic aspects.

[0144] The generator produces high-quality target images by combining spatial structure information provided by a leaf layer classification network and temporal dynamic information provided by a full pixel association network. The leaf layer classification network ensures that the generated image retains the structural and geometric features of the original image, while the full pixel association network ensures that the generated image is temporally consistent with preceding and following frames. Through this integration, the generator is able to produce high-quality images with rich detail and natural dynamics during the transformation process.

[0145] The discriminator comprehensively evaluates the generated images by utilizing spatial structure information from the leaf layer classification network and temporal dynamic information from the full pixel association network. Through a spatial structure consistency check, the discriminator ensures that the geometric structure of the generated image is consistent with the original image; through a temporal dynamic consistency check, the discriminator guarantees the motion coherence of the generated image between frames. Finally, the discriminator judges the quality of the generated image based on this comprehensive information, outputting its "realism" probability, providing feedback for the training of the generative adversarial network, thereby improving the quality of the generated images.

[0146] As used herein, the term "preferred" is meant as an example, illustration, or illustration. Any aspect or design described herein as "preferred" need not be construed as being more advantageous than other aspects or designs. Rather, the use of the term "preferred" is intended to present the concept in a specific manner. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusionary "or." That is, unless otherwise specified or clear from the context, "X uses A or B" naturally includes either of the permutations. That is, if X uses A; X uses B; or X uses both A and B, then "X uses A or B" is satisfied in any of the foregoing examples.

[0147] Furthermore, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art based on a reading and understanding of this specification and the accompanying drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the aforementioned components (e.g., elements, etc.), the terminology used to describe such components is intended to correspond to any component (unless otherwise indicated) that performs the specified function of said component (e.g., is functionally equivalent to it), even if structurally not equivalent to the disclosed structure performing the functions in the exemplary implementations of this disclosure shown herein. Moreover, although specific features of this disclosure have been disclosed with respect to only one of several implementations, such features may be combined with one or more features of other implementations that may be desirable and advantageous for a given or particular application. Furthermore, with regard to the use of the terms “comprising,” “having,” “containing,” or variations thereof in the Detailed Description or claims, such terms are intended to be included in a manner similar to the term “including.”

[0148] The functional units in this invention embodiment can be integrated into a processing module, or each unit can exist physically separately, or multiple units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. The aforementioned devices or systems can execute the storage methods in the corresponding method embodiments.

[0149] In summary, the above embodiments are one implementation of the present invention, but the implementation of the present invention is not limited to the embodiments described above. Any changes, modifications, substitutions, combinations, or simplifications made that deviate from the spirit and principle of the present invention should be considered equivalent substitutions and are included within the protection scope of the present invention.

Claims

1. A video image spatial transformation method based on deep learning, characterized in that, Includes the following steps: Step 1: Input the video information into the feature extraction network to obtain multi-scale feature information of time and content; Step 2: Input the multi-scale feature information into the multi-scale leaf layer classification network, and use the leaf layer classification network to classify the extracted multi-scale feature information to ensure that the local and global information of the image can be accurately captured at different scales. Step 3: Calculate the correlation volume between input frames using a full pixel association network, and obtain the optical flow estimate through iterative updates; Step 4: Optimize video conversion effects in time and space dimensions through comparative learning; ensure overall consistency of the generated video sequence through cycle consistency optimization. Step 5: Use the generator to generate the target image and use the discriminator to evaluate the quality of the generated image; Step 6: Convert the generated high-quality image into video output; Step 5, which describes generating the target image using a generator, includes the following steps: The generator first extracts multi-scale features from the input image, and then combines the classification results of the leaf layer classification network to perform structural constraints and fusion on these features to ensure that the generator preserves the spatial geometric features and hierarchical structure of the image when generating the image. After incorporating spatial structure information, the generator further references the optical flow field information provided by the full pixel association network. Guided by optical flow, the target image generated by the generator maintains motion consistency in the time series and performs motion compensation and adjustment on dynamically changing parts. Finally, the generator utilizes information fused from spatial structure and temporal dynamics to generate the target image through a series of convolution and deconvolution operations; Step 5, which describes evaluating the quality of the generated image using a discriminator, includes the following steps: Spatial structure check: The discriminator compares the features of the generated image with the output of the leaf layer classification network to check whether the image is consistent with the original image in terms of spatial structure. In the temporal dynamic inspection, the discriminator uses optical flow information from the full pixel association network to evaluate the motion consistency between the generated image and adjacent frames; The discriminator comprehensively evaluates the results of spatial structure inspection and temporal dynamics inspection to generate a final discrimination result and outputs an image quality score. If the generated image performs well in both aspects, the discriminator will give a high quality score, otherwise it will give a low quality score.

2. The video image spatial transformation method based on deep learning according to claim 1, characterized in that, The input to the feature extraction network is a single frame of video image. Size is ,in and These represent the image height and width, respectively, and 3 represents the RGB color channels. The output is a set of multi-scale feature maps. The computational steps of the feature extraction network include... the following: Step 101: Perform convolution operation on the input image through the initial convolutional layer to extract basic spatial features and obtain the initial feature map; Step 102: Perform multiple convolutions on the initial feature map to generate a series of intermediate feature maps; Step 103: Extract features at different scales and generate multi-scale feature maps by using convolution kernels or pooling operations of different sizes; Step 104: The feature maps extracted at different scales are fused to generate the final multi-scale feature map.

3. The video image spatial transformation method based on deep learning according to claim 1, characterized in that, The method of classifying the extracted multi-scale feature information using a leaf layer classification network includes the following steps: Step 201: Perform convolution operation on the input multi-scale feature map to capture local patterns in the feature map and generate a preliminary leaf layer representation. Step 202: Pool the convolutional feature map to reduce its dimensionality while preserving the most important spatial information; Step 203: Flatten the pooled feature map and further process it through a fully connected layer to extract the global pattern; Step 204: Input the flattened feature vector into the classification layer, and output the class probability of the leaf layer through the Softmax activation function.

4. The video image spatial transformation method based on deep learning according to claim 1, characterized in that, The method of calculating the correlation volume between input frames through a full-pixel correlation network and obtaining optical flow estimation through iterative updates includes the following steps: Step 301, process the input image frame and Multi-scale feature maps are extracted and used as the basis for association calculation; Step 302: Calculate the correlation of all pixel pairs between input image frame pairs to generate a 4D correlation volume; Step 303: Process the 4D related volume at different scales, and continuously update the optical flow estimate through a recurrent unit (GRU). Step 304: Based on the updated 4D related volume, the updated optical flow estimate is processed by convolution operation to generate the final optical flow field, i.e., the motion vector of each pixel.

5. The video image spatial transformation method based on deep learning according to claim 4, characterized in that, In step 302, for each pixel in each feature map, the inner product with all pixels in another feature map of the image frame pair is calculated to generate a 4D related volume. The calculation formula is as follows: in, Indicates the first 1 pixel in Middle and the first 1 pixel in The correlation in It is an index of the feature dimension. and They are from and Extracted multi-scale feature maps. express The Middle The pixel number Feature values ​​of the feature dimension express The Middle The pixel number eigenvalues; In step 303, relevant volumes at different scales are matched and correlated, and the optical flow estimate is updated iteratively to link the relevant volumes. Pooling is performed to generate multi-scale correlation volumes at different resolutions. A recurrent neural network (GRU) is used to iteratively update the correlation volumes, with optical flow estimation updated each time. The calculation formula is as follows: ， in, Indicates the first Next iteration optical flow estimation It is the updated optical flow increment. Indicates the first Next iteration of optical flow estimation.

6. The video image spatial transformation method based on deep learning according to claim 5, characterized in that, The generation of the 4D related volume includes the following steps: Extract two consecutive image frames and Feature vector at each pixel location in the multi-scale feature map: ， in, Represents an image frame Middle position eigenvectors, Represents an image frame Middle position eigenvectors, It is the feature dimension. Represent real numbers; Calculate the similarity between two feature vectors, using the inner product as a measure of similarity; for each pair of pixel positions and Calculate the inner product of their eigenvectors. ， in, It is an element in the 4D related volume, representing a pixel pair. and Similarity between them; The similarity calculation results of all pixel pairs are organized into a 4D tensor, i.e., a 4D correlation volume; by traversing... and The 4D correlated volume of the entire image frame is calculated from all pixel locations. ; Pooling operations are used to generate 4D correlated volumes at different resolutions to capture motion information at different scales; 4D correlated volumes Perform pooling operations to generate 4D correlated volumes at different resolutions. , ， in, It is the pooled 4D related volume, which contains pixel correlation information at lower resolution. yes One of the elements.

7. The video image spatial transformation method based on deep learning according to claim 5 or 6, characterized in that, Step 304, generating the final optical flow field, includes the following steps: Obtain 4D related volumes at different resolutions ; Initial optical flow estimation Initialize to a zero vector or an optical flow field based on the previous frame; The optical flow estimate is iteratively updated at different scales using a recurrent neural network (GRU) to gradually approximate the true optical flow field; at each scale... Above, using the current optical flow estimation The corresponding pixel position, from the 4D related volume Retrieve relevant information from the data and update the optical flow estimate using GRU units; After updating the optical flow estimate at the current scale, it is upsampled to a higher resolution so that iterative updates can continue at higher scales; at each scale After completing the optical flow update, bilinear interpolation or convolutional upsampling is used to upsample the optical flow estimate to a higher resolution. In scale Optical flow estimation after upsampling; After iterative updates at all scales are completed, the final optical flow estimation result is output; after the final optical flow update at the highest resolution, the final optical flow field is output. .

8. The video image spatial transformation method based on deep learning according to claim 7, characterized in that, The optical flow estimation includes the following steps: Based on the updated optical flow estimation The optical flow field is further refined using matching information from the relevant volume to ensure high-precision motion estimation; for each pixel location... Using the updated optical flow estimation In feature map Locate the corresponding position in the middle and from the relevant volume Extract the corresponding matching information; ， in, It is the refined optical flow vector. The Refine operation refines the optical flow based on information in the relevant volume. The refined optical flow field is then reverse-mapped and subjected to a consistency check to ensure bidirectional consistency of the optical flow; for each pixel location... According to the optical flow vector Calculate its corresponding position in the previous frame. Check the consistency between forward and reverse optical flow to ensure that the relative error is within a certain range; Smoothing and regularization operations are applied to eliminate noise and discontinuities in the optical flow field and ensure its smoothness; Gaussian filtering or global optimization methods are used to smooth the refined optical flow field. The optical flow field after refinement, consistency check, and smoothing is the final output optical flow field.

9. The video image spatial transformation method based on deep learning according to claim 1, characterized in that, The aforementioned method of optimizing video conversion effects in both temporal and spatial dimensions through comparative learning includes the following steps: Calculate the temporal similarity loss between consecutive frames to ensure smoothness and consistency of motion between frames; for two adjacent frames... and According to optical flow estimation Will Feature vector mapping back The feature space is used to calculate the similarity loss. ， in, and They represent time. and Video frame feature map at any given time It is a loss of temporal similarity. It is the total duration of the video; Based on temporal similarity loss, the generator is optimized through backpropagation to adjust the parameters of the video conversion model; the Adam optimization algorithm is used to minimize the temporal similarity loss and adjust the generator parameters. Calculate the spatial similarity loss within the same frame to ensure the consistency of spatial structure and details within the frame; In each frame Inside, select multiple pixel pairs and Calculate the similarity between the feature vectors of these pixel pairs; ， in, Indicates time Time, location eigenvectors, For spatial similarity loss; Based on spatial similarity loss, the generator is optimized through backpropagation, and the parameters of the video conversion model are adjusted; the spatial similarity loss is minimized using an optimization algorithm, and the generator parameters are adjusted accordingly. By combining temporal and spatial similarity losses, a final optimization objective is formed; and by combining temporal and spatial similarity losses, a total loss function is constructed. , ， in, and These are the weights for temporal similarity and spatial similarity losses, respectively. Minimize the total loss function and optimize all parameters of the video conversion model; The method of ensuring the overall consistency of the generated video sequence through cycle consistency optimization includes the following steps: Using generators Original image frame Convert to image frames in the target state ; Using a reverse generator The generated image frame By converting back to the original state, the reconstructed image frame is obtained. ; Calculate the original image frame and reconstructed image frames The pixel-level differences between them are used as the cycle consistency loss; Minimize cycle consistency loss through backpropagation , ， in, and Representing image frames respectively and Middle position Pixel value optimization generator and reverse generator The parameters.

Images