Video image space conversion method based on deep learning

Abstract

The invention discloses a video image space conversion method based on deep learning, and the method comprises the following steps: inputting video information into a feature extraction network, and obtaining multi-scale feature information of time and content information; inputting the multi-scale feature information into a multi-scale leaf layer classification network, and classifying the extracted multi-scale feature representation by using the leaf layer classification network to ensure that local and global information of the image can be accurately captured on different scales; the correlation volume between the input frames is calculated through a full-pixel correlation network, and optical flow estimation is obtained through iterative updating; through comparative learning, the video conversion effect is optimized in time and space dimensions; the overall consistency of the generated video sequence is ensured through cyclic consistency optimization; generating a target image by using a generator, and evaluating the quality of the generated image through a discriminator; and the generated high-quality image is converted into a video to be output.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular to a video image space conversion method based on deep learning. Background Art

[0002] Night vision enhancement and infrared imaging technologies are important components of modern military, security, autonomous driving, and scientific research. These technologies are designed to enhance visual perception in low-light or completely dark environments, and to convert infrared spectra that are invisible to the human eye into visible light images for observation and analysis. Video image space conversion technology plays a key role in this application, achieving night vision enhancement and infrared imaging by converting infrared images or other spectral images into visible light images. Before the emergence of deep learning algorithms, many image and video conversion-related research works were based on different color channel descriptions, which aimed to find some linear or nonlinear mapping relationships, such as converting grayscale images to color images and adjusting image contrast. However, the distribution of different state spaces does not have a one-to-one correspondence, so it is impossible to convert them by traditional mathematical methods. At the same time, the data set of the reinforcement learning algorithm requires not only discrete state spaces, but also continuous state spaces. Therefore, it is necessary to add image conversion technology, and it is also necessary to consider video conversion technology that is more difficult than images, such as converting images from infrared state space to visible light state space. It is not a particularly serious problem, but when all the conversion results are combined into a continuous video, even very subtle incoherent differences can cause data unreality.

[0003] Although existing technologies have made significant progress in video image space conversion, there are still some challenges and problems. During the video conversion process, the continuity and consistency between frames are often not fully guaranteed, resulting in flickering, distortion or unnatural transitions of objects in the video after conversion. 1. Especially in high-resolution images and complex scenes, existing technologies are prone to loss of details or image blur, affecting the quality and visual effects of the converted images. Existing technologies often require a large amount of labeled data and a complex model training process, which places high demands on computing resources and data sets, limiting the promotion and application of the technology. Technologies based on generative adversarial networks (GANs) are prone to instability during training, such as mode collapse and training oscillations, resulting in inconsistent quality of generated images. Summary of the invention

[0004] In view of this, the object of the present invention is to use the principles and methods of deep learning to propose a new video image space conversion method, comprising the following steps:

[0005] Step 1: Input the video information into the feature extraction network to obtain multi-scale feature information of time and content information;

[0006] Step 2, inputting the multi-scale feature information into a multi-scale leaf-level classification network, and using the leaf-level classification network to classify the extracted multi-scale feature representations to ensure that the local and global information of the image can be accurately captured at different scales;

[0007] Step 3: Calculate the correlation volume between input frames through the full pixel association network, and obtain the optical flow estimation through iterative update;

[0008] Step 4: Optimize the video conversion effect in time and space dimensions through contrastive learning; ensure the overall consistency of the generated video sequence through cycle consistency optimization;

[0009] Step 5: Use the generator to generate the target image, and use the discriminator to evaluate the quality of the generated image;

[0010] Step 6, the generated image is converted into video output.

[0011] Specifically, the input of the feature extraction network is a single-frame video image I, the size of which is H×W×3, where H and W represent the height and width of the image, respectively, and 3 represents the RGB color channel. The output is a set of multi-scale feature maps. The calculation steps of the feature extraction network include the following:

[0012] Step 101, performing a convolution operation on the input image through an initial convolution layer to extract basic spatial features and obtain an initial feature map;

[0013] Step 102, performing multiple convolutions on the initial feature map to generate a series of intermediate feature maps;

[0014] Step 103, extracting features at different scales, and generating multi-scale feature maps through convolution kernels or pooling operations of different sizes;

[0015] Step 104, the feature maps extracted at different scales are fused to generate a final multi-scale feature map.

[0016] Specifically, the method of using a leaf-level classification network to classify the extracted multi-scale feature information includes the following steps:

[0017] Step 201, performing a convolution operation on the input multi-scale feature map to capture local patterns in the feature map and generate a preliminary leaf layer representation;

[0018] Step 202, pooling the convolved feature map to reduce the dimension of the feature map while retaining the most important spatial information;

[0019] Step 203, flattening the pooled feature map and further processing it through a fully connected layer to extract the global pattern;

[0020] Step 204, input the flattened feature vector to the classification layer, and output the category probability of the leaf layer through the Softmax activation function.

[0021] Specifically, the method of calculating the correlation volume between input frames through a full pixel association network and obtaining the optical flow estimation through iterative updating includes the following steps:

[0022] Step 301: input image frame I 1 and I 2 Extract multi-scale feature maps as the basis for association calculation;

[0023] Step 302, calculating the correlation of all pixel pairs between pairs of input image frames to generate a 4D correlation volume;

[0024] Step 303, processing the 4D correlation volume at different scales, and continuously iteratively updating the optical flow estimation through the recurrent unit GRU;

[0025] Step 304 , based on the updated 4D correlation volume, the updated optical flow estimate is processed using a convolution operation to generate a final optical flow field, ie, a motion vector for each pixel.

[0026] Specifically, in step 302, for each pixel in each feature map, the inner product with all pixels in another feature map is calculated to generate a 4D correlation volume C, which is calculated as follows:

[0027]

[0028] Among them, C ijkl Indicates the (i,j)th pixel in I 1 and the (k,l)th pixel in I 2 The correlation in , h is the index of the feature dimension, f 1 and f 2 From I 1 and I 2 The extracted multi-scale feature map, f 1 (i,j,h) means I 1 The feature value of the hth feature dimension of the (i, j)th pixel in the 2 (k,l,h) represents I 2 The feature value of the hth feature dimension of the (k, l)th pixel in ;

[0029] In step 303, the relevant volumes of different scales are matched and associated, and the optical flow estimation is cyclically updated to pool the relevant volume C to generate multi-scale relevant volumes of different resolutions. The recurrent neural network GRU is used to iteratively update the relevant volumes, and the optical flow estimation is updated each time. The calculation formula is: k+1 =fk +Δf k , where f k represents the optical flow estimate of the kth iteration, Δf k is the updated optical flow increment, f k+1 represents the optical flow estimate at the k+1th iteration.

[0030] Furthermore, the generation of the 4D correlation volume comprises the following steps:

[0031] Extract two consecutive image frames I 1 and I 2 The feature vector of each pixel position in the multi-scale feature map is: Among them, f 1 (i,j) represents image frame I 1 The feature vector at position (i, j) in 2 (k,l) represents the image frame I 2 The feature vector of position (k, l) in the middle, D is the feature dimension, represents a real number;

[0032] Calculate the similarity between two feature vectors, using the inner product as a measure of similarity; for each pair of pixel positions (i, j) and (k, l), calculate the inner product of their feature vectors, Among them, C ijkl is an element in the 4D correlation volume, indicating the similarity between the pixel pairs (i, j) and (k, l);

[0033] The similarity calculation results of all pixel pairs are organized into a 4D tensor, i.e., a 4D correlation volume; by traversing I 1 and I 2 All pixel positions in the image are calculated to obtain the 4D correlation volume C of the entire image frame;

[0034] Generate 4D related volumes of different resolutions through pooling operations to capture motion information at different scales; perform pooling operations on the 4D related volumes C to generate 4D related volumes C of different resolutions s , Among them, C s It is the pooled 4D correlation volume, which contains pixel correlation information at a lower resolution.

[0035] Furthermore, step 304 of generating the final optical flow field includes the following steps:

[0036] Obtain 4D correlation volumes C at different resolutions s ;

[0037] Initialize the optical flow estimate f 0, usually initialized to a zero vector or an optical flow field based on the previous frame;

[0038] Use the recurrent neural network GRU to iteratively update the optical flow estimate at different scales to gradually approach the real optical flow field; at each scale s, use the current optical flow estimate f k The corresponding pixel position, from the 4D correlation volume C s Retrieve relevant information from the , and update the optical flow estimate through the GRU unit;

[0039] After updating the optical flow estimate at the current scale, it is upsampled to a higher resolution so that it can continue to be iteratively updated at a higher scale; after completing the optical flow update at each scale s, the optical flow estimate is upsampled to a higher resolution using bilinear interpolation or convolution upsampling. is the optical flow estimate after sampling at scale s;

[0040] After the iterative update of all scales is completed, the final optical flow estimation result is output; after the last optical flow update is completed at the highest resolution, the final optical flow field f is output final .

[0041] Preferably, the optical flow estimation comprises the following steps:

[0042] Based on the updated optical flow estimate f final And the matching information in the relevant volume is used to further refine the optical flow field to ensure high-precision motion estimation; for each pixel position (i, j), the updated optical flow estimate f is used final (i,j) in the feature map f 2 Locate the corresponding position in the image and extract the corresponding matching information from the relevant volume C; Among them, f refined (i, j) is the refined optical flow vector. The Refine operation refines the optical flow based on the information in the relevant volume.

[0043] The refined optical flow field is reverse mapped and checked for consistency to ensure the bidirectional consistency of the optical flow; for each pixel position (i, j), according to the optical flow vector f refined (i,j) calculates its corresponding position in the previous frame (i ′ ,j ′ ); Check the consistency of the forward optical flow and the reverse optical flow to ensure that the relative error is within a certain range;

[0044] Apply smoothing and regularization operations to eliminate noise and discontinuity in the optical flow field and ensure the smoothness of the optical flow field; use Gaussian filtering or global optimization methods to smooth the refined optical flow field, usually;

[0045] The optical flow field after refinement, consistency check and smoothing is the final output optical flow field.

[0046] Specifically, the optimization of the video conversion effect in time and space dimensions through contrastive learning includes the following steps:

[0047] Calculate the temporal similarity loss between consecutive frames to ensure the smoothness and consistency of inter-frame motion; for two adjacent frames I t and I t+1 , according to the optical flow estimation f(t), I t+1 The feature vector of I is mapped back to t The feature space of , and calculate the similarity loss, Among them, F 1 (t) and F 2 (t+1) represents the video frame feature graph at time t and t+1 respectively, is the temporal similarity loss, T is the total duration of the video;

[0048] Based on the temporal similarity loss, the generator is optimized through back propagation and the parameters of the video conversion model are adjusted. The Adam optimization algorithm is used to minimize the temporal similarity loss and adjust the generator parameters.

[0049] Calculate the spatial similarity loss within the same frame to ensure the consistency of spatial structure and details within the frame;

[0050] In each frame I t In the 1 ,j 1 ) and (i 2 ,j 2 ), calculate the similarity between the feature vectors of these pixel pairs; Among them, F 1 (t,i 1 ,j 1 ) means that at time t, position (i 1 ,j 1 ), Spatial similarity loss;

[0051] Based on the spatial similarity loss, the generator is optimized through back propagation and the parameters of the video conversion model are adjusted; the optimization algorithm is used to minimize the spatial similarity loss and adjust the generator parameters;

[0052] Combine temporal similarity and spatial similarity losses to form the final optimization goal; combine temporal and spatial similarity losses to construct the total loss function Among them, λ time and λ space are the weights of temporal similarity and spatial similarity losses, respectively;

[0053] Minimize the total loss function and optimize all parameters of the video conversion model.

[0054] Specifically, the overall consistency of the generated video sequence is ensured by cycle consistency optimization, comprising the following steps:

[0055] Use the generator G to transform the original image frame I t Convert to the image frame in the target state

[0056] Using the reverse generator G -1 The generated image frame Convert back to the original state to get the reconstructed image frame

[0057] Calculate the original image frame I t and the reconstructed image frame The pixel-level difference between them is used as the cycle consistency loss;

[0058] Minimizing the cycle consistency loss via backpropagation Among them, I t (i,j) and Represents image frame I t and The pixel value optimization generator G and the reverse generator G at position (i, j) in -1 Parameters.

[0059] Specifically, the use of the generator to generate the target image described in step 5 includes the following steps:

[0060] The generator first extracts multi-scale features from the input image, and then combines the classification results of the leaf-level classification network to structurally constrain and fuse these features to ensure that the generator retains the spatial geometric features and hierarchical structure of the image when generating the image;

[0061] After integrating the spatial structure information, the generator further refers to the optical flow field information provided by the full pixel association network. Guided by the optical flow, the target image generated by the generator maintains motion consistency in the time series and performs motion compensation and adjustment on the dynamically changing parts.

[0062] Finally, the generator uses the information of spatial structure and temporal dynamic fusion to generate the target image through a series of convolution and deconvolution operations.

[0063] Specifically, the step 5 of evaluating the quality of the generated image by the discriminator includes the following steps:

[0064] Spatial structure check: the discriminator compares the features of the generated image with the output of the leaf-level classification network to check whether the image is consistent with the original image in terms of spatial structure;

[0065] Temporal dynamics check, the discriminator uses the optical flow information of the full pixel association network to evaluate the motion consistency between the generated image and adjacent frames;

[0066] The discriminator comprehensively evaluates the results of spatial structure inspection and temporal dynamic inspection, generates a final discrimination result, and outputs the quality score of the image. If the generated image performs well in both aspects, the discriminator will give a higher quality score, otherwise it will give a lower quality score.

[0067] The beneficial effects of the present invention are as follows: each pixel in the video frame is regarded as a point on the complex projection plane, and by constructing leaf layer models of different dimensions, the moving objects in the video are classified and features extracted in a more detailed manner; the correlation between each pair of pixels is calculated, and in each frame of the video, a multi-scale 4D correlation volume is constructed to perform correlation calculations on all pixels, and the mapping between state spaces is optimized by continuously iterating and updating through a recurrent neural network; the mutual information between pixels is enhanced in the spatial dimension to ensure that the converted frames remain consistent in details. At the same time, the optimization of the time dimension further enhances the smooth transition between frames through a convolutional neural network, avoiding visual distortion caused by time inconsistency. The present application maintains very good spatiotemporal consistency and can meet the actual application requirements of video conversion. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 Overall flow chart of the video space conversion method. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0071] This embodiment proposes a video image space conversion method based on deep learning, including the following steps:

[0072] Step 1: Input the video information into the feature extraction network to obtain multi-scale feature representation of time and content information;

[0073] Step 2, inputting the multi-scale feature information into a multi-scale leaf-level classification network, and using the leaf-level classification network to classify the extracted multi-scale feature representations to ensure that the local and global information of the image can be accurately captured at different scales;

[0074] Step 3: Calculate the correlation volume between input frames through the full pixel association network, and obtain the optical flow estimation through iterative update;

[0075] Step 4: Optimize the video conversion effect in time and space dimensions through contrastive learning; ensure the overall consistency of the generated video sequence through cycle consistency optimization;

[0076] Step 5: Use the generator to generate the target image, and use the discriminator to evaluate the quality of the generated image;

[0077] Step 6, the generated image is converted into video output.

[0078] The feature extraction network usually consists of multiple convolutional layers, batch normalization layers, and activation function layers (such as ReLU). These layers convert the original image into a high-dimensional feature representation by extracting and processing the information in the input image layer by layer. The network structure can be divided into the following main parts: initial convolution layer: used to capture the basic features of the image, such as edges, corners, etc.; intermediate convolution layer: further extract higher-level features, such as texture, shape, etc.; multi-scale feature layer: through convolution kernels of different sizes and pooling operations, extract the features of the image at different scales; feature fusion layer: fuse features of different scales to generate the final high-dimensional feature vector.

[0079] Specifically, the input of the feature extraction network is a single-frame video image I, the size of which is H×W×3, where H and W represent the height and width of the image, respectively, and 3 represents the RGB color channel. The output is a set of multi-scale feature maps. The calculation steps of the feature extraction network include the following:

[0080] Step 101, performing a convolution operation on the input image through an initial convolution layer to extract basic spatial features and obtain an initial feature map;

[0081] Step 102, performing multiple convolutions on the initial feature map to generate a series of intermediate feature maps;

[0082] Step 103, extracting features at different scales, and generating multi-scale feature maps through convolution kernels or pooling operations of different sizes;

[0083] Step 104, the feature maps extracted at different scales are fused to generate a final multi-scale feature map.

[0084] The main function of the leaf classification network is to further process the multi-scale features generated by the feature extraction network to identify and classify different structural features in the image. These features are regarded as leaf layers (i.e. leaves on the complex projection plane), and the network classifies them by identifying the properties of these leaf layers (such as whether they are convex). The classification results will be used to guide the subsequent video image conversion process, making the converted image more accurate in structure and details.

[0085] Specifically, the method of using a leaf-level classification network to classify the extracted multi-scale feature information includes the following steps:

[0086] Step 201, performing a convolution operation on the input multi-scale feature map to capture local patterns in the feature map and generate a preliminary leaf layer representation;

[0087] Step 202, pooling the convolved feature map to reduce the dimension of the feature map while retaining the most important spatial information;

[0088] Step 203, flattening the pooled feature map and further processing it through a fully connected layer to extract the global pattern;

[0089] Step 204, input the flattened feature vector to the classification layer, and output the category probability of the leaf layer through the Softmax activation function.

[0090] The classification result is a leaf-level classification of each pixel in the input feature map, and the output is a category probability distribution or category label. These classification results will be used to guide subsequent video image conversion, so that the structure and details of the original image can be accurately maintained during the conversion process.

[0091] The core function of the full pixel association network is to perform global association calculations on all pixels in an input image frame pair, thereby accurately estimating the optical flow, that is, the motion vector of each pixel between frames.

[0092] Specifically, the method of calculating the correlation volume between input frames through a full pixel association network and obtaining the optical flow estimation through iterative updating includes the following steps:

[0093] Step 301: input image frame I 1 and I 2 Extract multi-scale feature maps as the basis for association calculation;

[0094] Step 302, calculating the correlation of all pixel pairs between pairs of input image frames to generate a 4D correlation volume;

[0095] Step 303, processing the 4D correlation volume at different scales, and continuously iteratively updating the optical flow estimation through the recurrent unit GRU;

[0096] Step 304 , based on the updated 4D correlation volume, the updated optical flow estimate is processed using a convolution operation to generate a final optical flow field, ie, a motion vector for each pixel.

[0097] Specifically, in step 302, for each pixel in each feature map, the inner product with all pixels in another feature map is calculated to generate a 4D correlation volume C, which is calculated as follows:

[0098]

[0099] Among them, C ijkl Indicates the (i,j)th pixel in I 1 and the (k,l)th pixel in I 2 The correlation in , h is the index of the feature dimension, f 1 and f 2 From I 1 and I 2 The extracted multi-scale feature map, f 1 (i,j,h) means I 1 The feature value of the hth feature dimension of the (i, j)th pixel in the 2 (k,l,h) represents I 2 The feature value of the hth feature dimension of the (k, l)th pixel in ;

[0100] In step 303, the relevant volumes of different scales are matched and associated, and the optical flow estimation is cyclically updated to pool the relevant volume C to generate multi-scale relevant volumes of different resolutions. The recurrent neural network GRU is used to iteratively update the relevant volumes, and the optical flow estimation is updated each time. The calculation formula is: k+1 =f k +Δf k , where f k represents the optical flow estimate of the kth iteration, Δf k is the updated optical flow increment, f k+1 represents the optical flow estimate at the k+1th iteration.

[0101] The generation of a 4D correlation volume is one of the core steps, which is used to capture the relationship between all pixel pairs between two consecutive image frames. This process generates a 4D tensor containing global correlation information by calculating the similarity between the feature vectors of each pixel.

[0102] Furthermore, the generation of the 4D correlation volume comprises the following steps:

[0103] Extract two consecutive image frames I 1 and I 2 The feature vector of each pixel position in the multi-scale feature map is: Among them, f 1 (i,j) represents image frame I 1 The feature vector at position (i, j) in 2 (k,l) represents the image frame I 2 The feature vector of position (k, l) in the middle, D is the feature dimension, represents a real number;

[0104] Calculate the similarity between two feature vectors, using the inner product as a measure of similarity; for each pair of pixel positions (i, j) and (k, l), calculate the inner product of their feature vectors, Among them, C ijkl is an element in the 4D correlation volume, indicating the similarity between the pixel pairs (i, j) and (k, l);

[0105] The similarity calculation results of all pixel pairs are organized into a 4D tensor, i.e., a 4D correlation volume; by traversing I 1 and I 2 All pixel positions in the image are calculated to obtain the 4D correlation volume C of the entire image frame;

[0106] Generate 4D related volumes of different resolutions through pooling operations to capture motion information at different scales; perform pooling operations on the 4D related volumes C to generate 4D related volumes C of different resolutions s , Among them, C s It is the pooled 4D correlation volume, which contains pixel correlation information at a lower resolution.

[0107] Furthermore, step 304 of generating the final optical flow field includes the following steps:

[0108] Obtain 4D correlation volumes C at different resolutions s ;

[0109] Initialize the optical flow estimate f 0 , usually initialized to a zero vector or an optical flow field based on the previous frame;

[0110] Use the recurrent neural network GRU to iteratively update the optical flow estimate at different scales to gradually approach the real optical flow field; at each scale s, use the current optical flow estimate f k The corresponding pixel position, from the 4D correlation volume C s Retrieve relevant information from the , and update the optical flow estimate through the GRU unit;

[0111] After updating the optical flow estimate at the current scale, it is upsampled to a higher resolution so that it can continue to be iteratively updated at a higher scale; after completing the optical flow update at each scale s, the optical flow estimate is upsampled to a higher resolution using bilinear interpolation or convolution upsampling. is the optical flow estimate after sampling at scale s;

[0112] After the iterative update of all scales is completed, the final optical flow estimation result is output; after the last optical flow update is completed at the highest resolution, the final optical flow field f is output final .

[0113] In the new method of video image spatial transformation, the multi-scale association and update step is one of the core processes, which is used to gradually optimize the optical flow estimation. Optical flow estimation is the key step to finally generate the motion vector of each pixel. This step is based on the multi-scale association and updated correlation volume and optical flow estimation results, and finally outputs an accurate optical flow field to guide the transformation of the video image.

[0114] Preferably, the optical flow estimation comprises the following steps:

[0115] Based on the updated optical flow estimate f final And the matching information in the relevant volume is used to further refine the optical flow field to ensure high-precision motion estimation; for each pixel position (i, j), the updated optical flow estimate f is used final (i,j) in the feature map f 2 Locate the corresponding position in the image and extract the corresponding matching information from the relevant volume C; Among them, f refined (i, j) is the refined optical flow vector. The Refine operation refines the optical flow based on the information in the relevant volume.

[0116] The refined optical flow field is reverse mapped and checked for consistency to ensure the bidirectional consistency of the optical flow; for each pixel position (i, j), according to the optical flow vector f refined (i,j) calculates its corresponding position in the previous frame (i ′ ,j ′ ); Check the consistency of the forward optical flow and the reverse optical flow to ensure that the relative error is within a certain range;

[0117] Apply smoothing and regularization operations to eliminate noise and discontinuity in the optical flow field and ensure the smoothness of the optical flow field; use Gaussian filtering or global optimization methods to smooth the refined optical flow field, usually;

[0118] The optical flow field after refinement, consistency check and smoothing is the final output optical flow field.

[0119] Specifically, the optimization of the video conversion effect in time and space dimensions through contrastive learning includes the following steps:

[0120] Calculate the temporal similarity loss between consecutive frames to ensure the smoothness and consistency of inter-frame motion; for two adjacent frames I t and I t+1 , according to the optical flow estimation f(t), I t+1 The feature vector of I is mapped back to t The feature space of , and calculate the similarity loss, Among them, F 1 (t) and F 2 (t+1) represents the video frame feature graph at time t and t+1 respectively, is the temporal similarity loss, T is the total duration of the video;

[0121] Based on the temporal similarity loss, the generator is optimized through back propagation and the parameters of the video conversion model are adjusted. The Adam optimization algorithm is used to minimize the temporal similarity loss and adjust the generator parameters.

[0122] Calculate the spatial similarity loss within the same frame to ensure the consistency of spatial structure and details within the frame;

[0123] In each frame I t In the 1 ,j 1 ) and (i 2 ,j 2 ), calculate the similarity between the feature vectors of these pixel pairs; Among them, F 1 (t,i 1 ,j 1 ) means that at time t, position (i 1 ,j 1 ), Spatial similarity loss;

[0124] Based on the spatial similarity loss, the generator is optimized through back propagation and the parameters of the video conversion model are adjusted; the optimization algorithm is used to minimize the spatial similarity loss and adjust the generator parameters;

[0125] Combine temporal similarity and spatial similarity losses to form the final optimization goal; combine temporal and spatial similarity losses to construct the total loss function Among them, λ time and λ space are the weights of temporal similarity and spatial similarity losses, respectively;

[0126] Minimize the total loss function and optimize all parameters of the video conversion model.

[0127] Specifically, the overall consistency of the generated video sequence is ensured by cycle consistency optimization, comprising the following steps:

[0128] Use the generator G to transform the original image frame I t Convert to the image frame in the target state

[0129] Using the reverse generator G -1 The generated image frame Convert back to the original state to get the reconstructed image frame

[0130] Calculate the original image frame I t and the reconstructed image frame The pixel-level difference between them is used as the cycle consistency loss;

[0131] Minimizing the cycle consistency loss via backpropagation Among them, I t (i,j) and Represents image frame I t and The pixel value optimization generator G and the reverse generator G at position (i, j) in -1 Parameters.

[0132] Specifically, the use of the generator to generate the target image described in step 5 includes the following steps:

[0133] The generator first extracts multi-scale features from the input image, and then combines the classification results of the leaf-level classification network to structurally constrain and fuse these features to ensure that the generator retains the spatial geometric features and hierarchical structure of the image when generating the image;

[0134] After integrating the spatial structure information, the generator further refers to the optical flow field information provided by the full pixel association network. Guided by the optical flow, the target image generated by the generator maintains motion consistency in the time series and performs motion compensation and adjustment on the dynamically changing parts.

[0135] Finally, the generator uses the information of spatial structure and temporal dynamic fusion to generate the target image through a series of convolution and deconvolution operations.

[0136] Specifically, the step 5 of evaluating the quality of the generated image by the discriminator includes the following steps:

[0137] Spatial structure check: the discriminator compares the features of the generated image with the output of the leaf-level classification network to check whether the image is consistent with the original image in terms of spatial structure;

[0138] Temporal dynamics check, the discriminator uses the optical flow information of the full pixel association network to evaluate the motion consistency between the generated image and adjacent frames;

[0139] The discriminator comprehensively evaluates the results of spatial structure inspection and temporal dynamic inspection, generates a final discrimination result, and outputs the quality score of the image. If the generated image performs well in both aspects, the discriminator will give a higher quality score, otherwise it will give a lower quality score.

[0140] The leaf-level classification network and the full-pixel association network play different but complementary roles in the spatial transformation of video images. The relationship between the two can be summarized as follows.

[0141] Division of structure and motion: The leaf-level classification network focuses on the spatial structure of the image, that is, identifying and classifying static geometric features in the image. These feature classifications are used to guide local transformations of the image, ensuring that the integrity of the structure and the accuracy of the details are maintained during the transformation process. The full pixel association network focuses on dynamic changes in time, that is, capturing and calculating the movement of each pixel in the image between frames. Optical flow information is crucial to maintaining coherence and spatiotemporal consistency between video frames.

[0142] Interaction and collaborative work: Leaf-level classification guides optical flow estimation: The structural information provided by the leaf-level classification network can be used to guide the optical flow estimation of the full-pixel association network. For example, the leaf-level classification results can help determine which areas require higher accuracy in optical flow estimation (such as boundaries and convex areas), thereby optimizing the optical flow calculation and avoiding motion artifacts in areas with complex structures. Optical flow feedback improves leaf-level classification: On the other hand, the optical flow field calculated by the full-pixel association network can be fed back to the leaf-level classification network to help update and adjust the leaf-level classification results. Especially when processing dynamic scenes, optical flow information can help identify moving boundaries or dynamically changing areas in the image, thereby updating the leaf-level classification of these areas and improving the overall classification accuracy.

[0143] Comprehensively improve the conversion effect: The leaf-level classification network ensures the geometric fidelity of static structures, while the full-pixel association network ensures the temporal and spatial consistency in dynamic scenes. The synergy of the two makes the entire video image space conversion process more accurate, and the generated images can achieve high-quality performance in both static and dynamic aspects.

[0144] The generator generates high-quality target images by combining the spatial structure information provided by the leaf-level classification network and the temporal dynamic information provided by the full-pixel association network. The leaf-level classification network ensures that the generated image retains the structure and geometric features of the original image, while the full-pixel association network ensures that the generated image is temporally consistent with the previous and next frames. Through this integration, the generator is able to generate high-quality images with rich details and natural dynamics during the conversion process.

[0145] The discriminator comprehensively evaluates the generated image by utilizing the spatial structure information from the leaf-level classification network and the temporal dynamic information from the full-pixel association network. Through the consistency check of the spatial structure, the discriminator ensures that the geometric structure of the generated image is consistent with the original image; through the temporal dynamic consistency check, the discriminator ensures the motion coherence of the generated image between frames. Finally, the discriminator judges the quality of the generated image based on this comprehensive information, outputs its "authenticity" probability, and provides feedback for the training of the generative adversarial network, thereby improving the quality of the generated image.

[0146] As used herein, the word "preferred" is intended to be used as an example, instance, or illustration. Any aspect or design described herein as "preferred" is not necessarily to be construed as being more advantageous than other aspects or designs. On the contrary, the use of the word "preferred" is intended to present concepts in a specific way. The term "or" as used in this application is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X uses A or B" means any one of the naturally included permutations. That is, if X uses A; X uses B; or X uses both A and B, then "X uses A or B" is satisfied in any of the foregoing examples.

[0147] Moreover, although the present disclosure has been shown and described with respect to one or implementations, those skilled in the art will think of equivalent variations and modifications based on the reading and understanding of this specification and the accompanying drawings. The present disclosure includes all such modifications and variations, and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the above-mentioned components (such as elements, etc.), the terms used to describe such components are intended to correspond to any component (unless otherwise indicated) that performs the specified function of the component (such as it is functionally equivalent), even if the structure is not equivalent to the disclosed structure of the function in the exemplary implementation of the present disclosure shown herein. In addition, although the specific features of the present disclosure have been disclosed with respect to only one of several implementations, such features can be combined with one or other features of other implementations that may be desired and advantageous for a given or specific application. Moreover, insofar as the terms "including", "having", "containing" or their variations are used in specific embodiments or claims, such terms are intended to be included in a manner similar to the term "comprising".

[0148] The functional units in the embodiments of the present invention may be integrated into a processing module, or each unit may exist physically separately, or multiple or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc. The above-mentioned devices or systems may execute the storage method in the corresponding method embodiment.

[0149] To sum up, the above embodiment is an implementation mode of the present invention, but the implementation mode of the present invention is not limited by the embodiment. Any other changes, modifications, substitutions, combinations, and simplifications that deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A video image space conversion method based on deep learning, characterized in that: The following steps are involved: Step 1: Input the video information into the feature extraction network to obtain multi-scale feature information of time and content information; Step 2, inputting the multi-scale feature information into a multi-scale leaf-level classification network, and using the leaf-level classification network to classify the extracted multi-scale feature information to ensure that the local and global information of the image can be accurately captured at different scales; Step 3: Calculate the correlation volume between input frames through the full pixel association network, and obtain the optical flow estimation through iterative update; Step 4: Optimize the video conversion effect in time and space dimensions through contrastive learning; ensure the overall consistency of the generated video sequence through cycle consistency optimization; Step 5: Use the generator to generate the target image, and use the discriminator to evaluate the quality of the generated image; Step 6: The generated high-quality images are converted into video output.

2. The video image space conversion method based on deep learning according to claim 1 is characterized in that: The input of the feature extraction network is a single-frame video image I, with a size of H×W×3, where H and W represent the height and width of the image respectively, and 3 represents the RGB color channel. The output is a set of multi-scale feature maps. The calculation steps of the feature extraction network include the following: Step 101, performing a convolution operation on the input image through an initial convolution layer to extract basic spatial features and obtain an initial feature map; Step 102, performing multiple convolutions on the initial feature map to generate a series of intermediate feature maps; Step 103, extracting features at different scales, and generating multi-scale feature maps through convolution kernels or pooling operations of different sizes; Step 104, the feature maps extracted at different scales are fused to generate a final multi-scale feature map.

3. The video image space conversion method based on deep learning according to claim 1, characterized in that: The method of using the leaf-level classification network to classify the extracted multi-scale feature information includes the following steps: Step 201, performing a convolution operation on the input multi-scale feature map to capture local patterns in the feature map and generate a preliminary leaf layer representation; Step 202, pooling the convolved feature map to reduce the dimension of the feature map while retaining the most important spatial information; Step 203, flattening the pooled feature map and further processing it through a fully connected layer to extract the global pattern; Step 204, input the flattened feature vector to the classification layer, and output the category probability of the leaf layer through the Softmax activation function.

4. The video image space conversion method based on deep learning according to claim 1, characterized in that: The method of calculating the correlation volume between input frames through the full pixel association network and obtaining the optical flow estimation through iterative updating includes the following steps: Step 301, extracting multi-scale feature maps of the input image frames I1 and I2 as a basis for association calculation; Step 302, calculating the correlation of all pixel pairs between pairs of input image frames to generate a 4D correlation volume; Step 303, processing the 4D correlation volume at different scales, and continuously iteratively updating the optical flow estimation through the recurrent unit GRU; Step 304 , based on the updated 4D correlation volume, the updated optical flow estimate is processed using a convolution operation to generate a final optical flow field, ie, a motion vector for each pixel.

5. The video image space conversion method based on deep learning according to claim 4 is characterized in that: In step 302, for each pixel in each feature map, the inner product with all pixels in another feature map in the image frame pair is calculated to generate a 4D correlation volume C, which is calculated as follows: Among them, C ijkl represents the correlation between the (i,j)th pixel in I1 and the (k,l)th pixel in I2, h is the index of the feature dimension, f1 and f2 are multi-scale feature maps extracted from I1 and I2 respectively, f1(i,j,h) represents the feature value of the hth feature dimension of the (i,j)th pixel in I1, and f2(k,l,h) represents the feature value of the hth feature dimension of the (k,l)th pixel in I2; In step 303, the relevant volumes of different scales are matched and associated, and the optical flow estimation is cyclically updated to pool the relevant volume C to generate multi-scale relevant volumes of different resolutions. The recurrent neural network GRU is used to iteratively update the relevant volumes, and the optical flow estimation is updated each time. The calculation formula is: k+1 =f k +Δf k , where f k represents the optical flow estimate of the kth iteration, Δf k is the updated optical flow increment, f k+1 represents the optical flow estimate at the k+1th iteration.

6. The method for video image space conversion based on deep learning according to claim 5, characterized in that: The generation of the 4D correlation volume comprises the following steps: Extract the feature vector of each pixel position in the multi-scale feature map of two consecutive image frames I1 and I2: Among them, f1(i,j) represents the feature vector of position (i,j) in image frame I1, f2(k,l) represents the feature vector of position (k,l) in image frame I2, D is the feature dimension, represents a real number; Calculate the similarity between two feature vectors, using the inner product as a measure of similarity; for each pair of pixel positions (i, j) and (k, l), calculate the inner product of their feature vectors, Among them, C ijkl is an element in the 4D correlation volume, indicating the similarity between the pixel pairs (i, j) and (k, l); The similarity calculation results of all pixel pairs are organized into a 4D tensor, namely, a 4D correlation volume; the 4D correlation volume C of the entire image frame is calculated by traversing all pixel positions in I1 and I2; Generate 4D related volumes of different resolutions through pooling operations to capture motion information at different scales; perform pooling operations on the 4D related volumes C to generate 4D related volumes C of different resolutions s , Among them, C s is the pooled 4D correlation volume, which contains pixel correlation information at a lower resolution. It is C s An element in .

7. The video image space conversion method based on deep learning according to claim 5 or 6, characterized in that: Step 304 generates the final optical flow field, including the following steps: Obtain 4D correlation volumes C at different resolutions s ; Initialize the optical flow estimate f0, usually initialized to a zero vector or an optical flow field based on the previous frame; Use the recurrent neural network GRU to iteratively update the optical flow estimation at different scales to gradually approach the real optical flow field; at each scale s, use the current optical flow estimation f k The corresponding pixel position, from the 4D correlation volume C s Retrieve relevant information from the , and update the optical flow estimate through the GRU unit; After updating the optical flow estimate at the current scale, it is upsampled to a higher resolution so that it can continue to be iteratively updated at a higher scale; after completing the optical flow update at each scale s, the optical flow estimate is upsampled to a higher resolution using bilinear interpolation or convolution upsampling. is the optical flow estimate after sampling at scale s; After the iterative update of all scales is completed, the final optical flow estimation result is output; after the last optical flow update is completed at the highest resolution, the final optical flow field f is output final .

8. The method for video image space conversion based on deep learning according to claim 7, characterized in that: The optical flow estimation comprises the following steps: Based on the updated optical flow estimate f final And the matching information in the relevant volume is used to further refine the optical flow field to ensure high-precision motion estimation; for each pixel position (i, j), the updated optical flow estimate f is used final (i, j) locate the corresponding position in the feature map f2 and extract the corresponding matching information from the correlation volume C; Among them, f refined (i, j) is the refined optical flow vector. The Refine operation refines the optical flow based on the information in the relevant volume. The refined optical flow field is reverse mapped and checked for consistency to ensure the bidirectional consistency of the optical flow; for each pixel position (i, j), according to the optical flow vector f refined (i,j) calculates its corresponding position in the previous frame (i ′ ,j ′ ); Check the consistency of the forward optical flow and the reverse optical flow to ensure that the relative error is within a certain range; Apply smoothing and regularization operations to eliminate noise and discontinuity in the optical flow field and ensure the smoothness of the optical flow field; use Gaussian filtering or global optimization methods to smooth the refined optical flow field, usually; The optical flow field after refinement, consistency check and smoothing is the final output optical flow field.

9. The video image space conversion method based on deep learning according to claim 1, characterized in that: The method of optimizing the video conversion effect in time and space dimensions through contrastive learning includes the following steps: Calculate the temporal similarity loss between consecutive frames to ensure the smoothness and consistency of inter-frame motion; for two adjacent frames I t and I t+1 , according to the optical flow estimation f(t), I t+1 The feature vector of I is mapped back to t The feature space of , and calculate the similarity loss, Among them, F1(t) and F2(t+1) represent the video frame feature graphs at time t and t+1 respectively. is the temporal similarity loss, T is the total duration of the video; Based on the temporal similarity loss, the generator is optimized through back propagation and the parameters of the video conversion model are adjusted. The Adam optimization algorithm is used to minimize the temporal similarity loss and adjust the generator parameters. Calculate the spatial similarity loss within the same frame to ensure the consistency of spatial structure and details within the frame; In each frame I t In, select multiple pixel pairs (i1, j1) and (i2, j2), and calculate the similarity between the feature vectors of these pixel pairs; Among them, F1(t,i1,j1) represents the feature vector of position (i1,j1) at time t, is the spatial similarity loss; Based on the spatial similarity loss, the generator is optimized through back propagation and the parameters of the video conversion model are adjusted; the optimization algorithm is used to minimize the spatial similarity loss and adjust the generator parameters; Combine temporal similarity and spatial similarity losses to form the final optimization goal; combine temporal and spatial similarity losses to construct the total loss function Among them, λ time and λ space are the weights of temporal similarity and spatial similarity losses, respectively; Minimize the total loss function and optimize all parameters of the video conversion model; The method of ensuring the overall consistency of the generated video sequence through cycle consistency optimization includes the following steps: Use the generator G to transform the original image frame I t Convert to the image frame in the target state Using the reverse generator G -1 The generated image frame Convert back to the original state to get the reconstructed image frame Calculate the original image frame I t and the reconstructed image frame The pixel-level difference between them is used as the cycle consistency loss; Minimizing the cycle consistency loss via backpropagation Among them, I t (i,j) and Represents image frame I t and The pixel value optimization generator G and the reverse generator G at position (i, j) in -1 Parameters.

10. The video image space conversion method based on deep learning according to claim 1, characterized in that: Generating a target image using the generator as described in step 5 includes the following steps: The generator first extracts multi-scale features from the input image, and then combines the classification results of the leaf-level classification network to structurally constrain and fuse these features to ensure that the generator retains the spatial geometric features and hierarchical structure of the image when generating the image; After integrating the spatial structure information, the generator further refers to the optical flow field information provided by the full pixel association network. Guided by the optical flow, the target image generated by the generator maintains motion consistency in the time series and performs motion compensation and adjustment on the dynamically changing parts. Finally, the generator uses the information of spatial structure and temporal dynamic fusion to generate the target image through a series of convolution and deconvolution operations; The evaluation of the quality of the generated image by the discriminator described in step 5 includes the following steps: Spatial structure check: the discriminator compares the features of the generated image with the output of the leaf-level classification network to check whether the image is consistent with the original image in terms of spatial structure; Temporal dynamics check, the discriminator uses the optical flow information of the full pixel association network to evaluate the motion consistency between the generated image and adjacent frames; The discriminator comprehensively evaluates the results of spatial structure inspection and temporal dynamic inspection, generates a final discrimination result, and outputs the quality score of the image. If the generated image performs well in both aspects, the discriminator will give a higher quality score, otherwise it will give a lower quality score.

Images