A data processing method and computer device

Abstract

The embodiment of the present application discloses a data processing method and computer device based on artificial intelligence, wherein the method includes: obtaining a first video frame and a second video frame in a video; the video scene corresponding to the first video frame is the same as the video scene corresponding to the second video frame; performing feature convolution processing on the first initial video feature corresponding to the first video frame to generate the first video feature to be fused corresponding to the first video frame; performing feature splicing processing on the first initial video feature corresponding to the first video frame and the second initial video feature corresponding to the second video frame to generate the first video feature to be processed; performing feature downsampling processing on the first video feature to be processed to generate the second video feature to be processed; generating an aligned video frame for the second video frame according to the second video feature to be processed and the first video feature to be fused. By adopting the present application, the recognition accuracy of optical flow data can be aligned, and then a high-accuracy aligned video frame can be generated.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a data processing method and a computer device. Background Art

[0002] Optical flow estimation is an important direction in computer vision research. It is the basis for solving many other problems in video tasks, such as motion estimation, motion segmentation, and behavior recognition. As a basic technology, optical flow estimation plays an important role in conjunction with other technologies, so it has attracted much attention.

[0003] Optical flow estimation can be achieved through video frame alignment (inter-frame alignment) methods, which can be specifically used to estimate the aligned optical flow data of the optical flow data in another image (video frame) through the optical flow data in one image (video frame). Most of the existing inter-frame alignment methods are based on the prior information of image indicators, that is, artificial analysis of the prior information, determining the law of the indicator, and estimating the optical flow difference in the image based on the law, for example, based on the brightness invariance of the same object in two images, estimating the aligned optical flow data of the image. However, humans can only observe the surface laws and it is difficult to discover the hidden factors in the deep level. Therefore, the existing methods may identify low-accuracy aligned optical flow data, thereby generating low-accuracy aligned video frames. Summary of the invention

[0004] The embodiments of the present application provide a data processing method and a computer device, which can improve the recognition accuracy of aligned optical flow data, and further generate aligned video frames with high accuracy.

[0005] On the one hand, an embodiment of the present application provides a data processing method, including:

[0006] Acquire a first video frame and a second video frame in a video; the video scene corresponding to the first video frame is the same as the video scene corresponding to the second video frame;

[0007] Performing feature convolution processing on a first initial video feature corresponding to the first video frame to generate a first to-be-fused video feature corresponding to the first video frame;

[0008] Performing feature splicing processing on a first initial video feature corresponding to the first video frame and a second initial video feature corresponding to the second video frame to generate a first video feature to be processed;

[0009] Performing feature downsampling processing on the first video feature to be processed to generate a second video feature to be processed;

[0010] An aligned video frame for the second video frame is generated according to the second video feature to be processed and the first video feature to be fused.

[0011] On the one hand, an embodiment of the present application provides a data processing method, including:

[0012] Acquire a first sample video frame and a second sample video frame in the sample video; the sample video scene corresponding to the first sample video frame is the same as the sample video scene corresponding to the second sample video frame;

[0013] The first sample video frame and the second sample video frame are respectively input into the video frame alignment initial model, and in the video frame alignment initial model, a feature convolution process is performed on the first initial sample video feature corresponding to the first sample video frame to generate a first sample video feature to be fused corresponding to the first sample video frame;

[0014] Performing feature splicing processing on a first initial sample video feature corresponding to the first sample video frame and a second initial sample video feature corresponding to the second sample video frame to generate a first sample video feature to be processed;

[0015] Performing feature downsampling processing on the first sample video feature to be processed to generate a second sample video feature to be processed;

[0016] Generate an aligned estimated video frame for the second sample video frame according to the second sample video feature to be processed and the first sample video feature to be fused;

[0017] Obtaining a mean square error value between the second sample video frame and the aligned estimated video frame, and determining the mean square error value as a loss value of the initial model for video frame alignment;

[0018] The model parameters in the initial model of the video frame alignment are adjusted according to the loss value to generate a video frame alignment model; the video frame alignment model is used to perform inter-frame alignment processing on video frames in the video.

[0019] An embodiment of the present application provides a data processing device, including:

[0020] A first acquisition module is used to acquire a first video frame and a second video frame in a video; the video scene corresponding to the first video frame is the same as the video scene corresponding to the second video frame;

[0021] A first generating module, configured to perform feature convolution processing on a first initial video feature corresponding to a first video frame to generate a first to-be-fused video feature corresponding to the first video frame;

[0022] A second generating module is used to perform feature splicing processing on the first initial video feature corresponding to the first video frame and the second initial video feature corresponding to the second video frame to generate a first video feature to be processed;

[0023] A third generating module is used to perform feature downsampling processing on the first video feature to be processed to generate a second video feature to be processed;

[0024] The fourth generating module is used to generate an aligned video frame for the second video frame according to the second video feature to be processed and the first video feature to be fused.

[0025] The third generation module includes:

[0026] The acquisition model unit is used to acquire a video frame alignment model; the video frame alignment model includes A feature downsampling layers, and the A feature downsampling layers include feature downsampling layers B c And feature downsampling layer B c+1 , A is a positive integer, c is a positive integer and c is less than A;

[0027] The first processing unit is used to downsample the feature layer B c In the process, the first input feature is downsampled to generate the first transition video feature D c ; If the feature downsampling layer B c is the first feature downsampling layer among A feature downsampling layers, then the first input feature is the first video feature to be processed; if feature downsampling layer B c If it is not the first feature downsampling layer among A feature downsampling layers, then the first input feature is, feature downsampling layer B c The first transition video feature output by the previous feature downsampling layer;

[0028] The second processing unit is used to convert the first transition video feature D c As feature downsampling layer B c+1 The first input feature of the feature downsampling layer B c+1 In the first transition video feature D c Perform feature downsampling to generate the first transition video feature D c+1 ;

[0029] The first determination unit is used to determine if the feature downsampling layer B c+1 If it is not the last feature downsampling layer among the A feature downsampling layers, the first transition video feature D c+1 Determined as, feature downsampling layer B c+1 The first input feature of the next feature downsampling layer;

[0030] The second determination unit is used to determine if the feature downsampling layer B c+1 is the last feature downsampling layer among the A feature downsampling layers, then the first transition video feature D c+1 Determined as the second video feature to be processed.

[0031] Among them, the feature downsampling layer B c Includes a first convolution component and a feature pooling component;

[0032] The first processing unit comprises:

[0033] A first processing subunit, configured to input the first input feature into a first convolution component, and perform feature convolution processing on the first input feature in the first convolution component to generate a video feature to be pooled;

[0034] The first processing subunit is further used to input the video features to be pooled into the feature pooling component; the feature pooling component carries the pooling size;

[0035] The second processing subunit is used to perform feature downsampling processing on the pooled video features in the feature pooling component according to the pooling size to generate the first transition video feature D c ; The feature resolution corresponding to the first input feature is equal to the pooling size and the first transition video feature D c The product of the corresponding feature resolution.

[0036] Among them, the first convolution component includes the first convolution component G h , and the first convolutional component G h The associated activation component, the first convolution component G h+1 , and the first convolutional component G h+1 The associated fusion component and the first convolution component G h+1 The associated activation component; h is a positive integer, and h is less than the total number of the first convolutional components;

[0037] The first processing subunit comprises:

[0038] The first input subunit is used to input the first input feature into the first convolution component G h ; The first convolutional component G h Includes E convolution channels, where E is a positive integer;

[0039] The first input subunit is also used to pass through the first convolution component G h The E convolution channels in the first input feature are processed by feature convolution to generate the first video feature I to be activated. h ; The first video feature to be activated I h The corresponding number of feature channels is equal to E;

[0040] The second input subunit is used to input the first video feature I to be activated h Input to the first convolution component G h The associated activation component is connected to the first convolution component G h The associated activation component, for the first video feature I to be activated h Perform feature nonlinear processing to generate the first convolution video feature J h ;

[0041] The third input subunit is used to convert the first convolutional video feature J h Input to the first convolutional component G h+1 ; The first convolutional component G h+1 Includes E convolution channels;

[0042] The third input subunit is also used to pass through the first convolution component G h+1 The E convolution channels in the first convolution video feature J h Perform feature convolution processing to generate the first video feature I to be activated h+1 ; The first video feature to be activated I h+1 The corresponding number of feature channels is equal to E;

[0043] The fourth input subunit is used to input the first video feature I to be activated h+1 And the first convolutional video feature J h Input to the first convolution component G h+1 The associated fusion component, in conjunction with the first convolutional component G h+1 In the associated fusion component, the first video feature I to be activated h+1 And the first convolutional video feature J h Perform feature fusion processing to generate the first video feature I to be activated h+2 ;

[0044] The fifth input subunit is used to input the first video feature I to be activated h+2 Input to the first convolution component G h+1 The associated activation component is connected to the first convolution component G h+1 The associated activation component, for the first video feature I to be activated h+2 Perform nonlinear feature processing to generate video features to be pooled.

[0045] The video frame alignment model also includes A feature upsampling layers, a first feature convolution layer, and a feature fusion layer;

[0046] The fourth generation module includes:

[0047] A third processing unit is used to perform feature upsampling processing on the second video feature to be processed through A feature upsampling layers to generate a third video feature to be processed, and input the third video feature to be processed into the first feature convolution layer; the feature resolution corresponding to the third video feature to be processed is equal to the feature resolution corresponding to the first video feature to be processed;

[0048] The fourth processing unit is used to perform feature convolution processing on the third video feature to be processed through the first feature convolution layer to generate a second video feature to be fused; the number of feature channels corresponding to the second video feature to be fused is equal to the number of feature channels corresponding to the first video feature to be fused;

[0049] a fifth processing unit, configured to input the first video feature to be fused and the second video feature to be fused into a feature fusion layer, and perform feature fusion processing on the first video feature to be fused and the second video feature to be fused in the feature fusion layer to generate a normalized video feature for the second video frame;

[0050] The sixth processing unit is used to perform feature denormalization processing on the normalized video features to generate aligned video frames.

[0051] Among them, A feature upsampling layers include feature upsampling layer K m And the feature upsampling layer K m+1 , m is a positive integer and m is less than A;

[0052] The third processing unit comprises:

[0053] The third processing subunit is used to sample the feature layer K m In the process, the second input feature is subjected to feature upsampling to generate the second transition video feature L m ; If the feature upsampling layer K m is the first feature upsampling layer among the A feature upsampling layers, then the second input feature is the second video feature to be processed; if the feature upsampling layer K m If it is not the first feature upsampling layer among the A feature upsampling layers, then the second input feature is, feature upsampling layer K m The second transition video feature output by the previous feature upsampling layer;

[0054] The fourth processing subunit is used to convert the second transition video feature L m As the feature upsampling layer K m+1 The second input feature of m+1 In the second transition video feature L m Perform feature upsampling to generate the second transition video feature L m+1 ;

[0055] The first determination subunit is used to determine if the feature upsampling layer K m+1 If it is not the last feature upsampling layer among the A feature upsampling layers, the second transition video feature L m+1 Determined as, the feature upsampling layer K m+1 The second input feature of the next feature upsampling layer;

[0056] The second determination subunit is used to determine if the feature upsampling layer K m+1 is the last feature upsampling layer in the A feature upsampling layers, then the second transition video feature L m+1 Determined as the third video feature to be processed.

[0057] Among them, the feature upsampling layer K m including a second convolution component and a feature upsampling component;

[0058] The third processing subunit includes:

[0059] A sixth input subunit, configured to input the second input feature into the second convolution component, and perform feature convolution processing on the second input feature in the second convolution component to generate a video feature to be upsampled;

[0060] A seventh input subunit, configured to input the to-be-upsampled video features into a feature upsampling component; the feature upsampling component carries an upsampling size;

[0061] The seventh input subunit is further configured to perform feature upsampling processing on the to-be-upsampled video feature in the feature upsampling component according to the upsampling size to generate a second transition video feature L m ; The second transition video feature L m The corresponding feature resolution is equal to the product of the feature resolution corresponding to the second input feature and the upsampling size.

[0062] Among them, the second convolution component includes the second convolution component N o , and the second convolutional component N o The associated activation component, the second convolution component N o+1 , and the second convolutional component N o+1 The associated fusion component and the second convolution component N o+1 The associated activation component; o is a positive integer, and o is less than the total number of the second convolution components;

[0063] The sixth input subunit is specifically used to input the second input feature into the second convolution component N o ; The second convolutional component N o Includes P convolution channels, where P is a positive integer;

[0064] The sixth input subunit is further specifically used to pass the second convolution component N o The P convolution channels in the image are used to perform feature convolution processing on the second input feature to generate the second video feature Q to be activated. o ; The second video feature Q to be activated o The corresponding number of feature channels is equal to P;

[0065] The sixth input subunit is further configured to input the second video feature Q to be activated o Input to the second convolution component N o The associated activation component is connected to the second convolution component N o The associated activation component is for the second video feature Q to be activated o Perform feature nonlinear processing to generate the second convolution video feature R o ;

[0066] The sixth input subunit is further specifically used to convert the second convolutional video feature R o Input to the second convolutional component N o+1 ; The second convolutional component N o+1 Includes P convolution channels;

[0067] The sixth input subunit is further specifically used to pass the second convolution component N o+1 The P convolution channels in the second convolution video feature R o Perform feature convolution processing to generate the second video feature Q to be activated o+1 ; The second video feature Q to be activated o+1 The corresponding number of feature channels is equal to P;

[0068] The sixth input subunit is further configured to input the second video feature Q to be activated o+1 And the second convolutional video feature R o Input to the second convolution component N o+1 The associated fusion component, in conjunction with the second convolutional component N o+1 In the associated fusion component, the second video feature Q to be activated o+1 And the second convolutional video feature R o Perform feature fusion processing to generate the second video feature Q to be activated o+2 ;

[0069] The sixth input subunit is further configured to input the second video feature Q to be activated o+2 Input to the second convolution component N o+1 The associated activation component is connected to the second convolution component N o+1 The associated activation component is for the second video feature Q to be activated o+2 Perform nonlinear feature processing to generate video features to be upsampled.

[0070] Among them, the first feature convolution layer includes a feature convolution component, a feature activation component and a feature normalization component;

[0071] The fourth processing unit comprises:

[0072] The first generating subunit is used to input the third video feature to be processed into the feature convolution component, and in the feature convolution component, perform feature convolution processing on the third video feature to be processed to generate a fourth video feature to be processed; the number of feature channels corresponding to the fourth video feature to be processed is equal to the number of feature channels corresponding to the first video feature to be fused;

[0073] The second generating subunit is used to input the fourth to-be-processed video feature into the feature activation component, and in the feature activation component, perform feature nonlinear processing on the fourth to-be-processed video feature to generate a fifth to-be-processed video feature;

[0074] The third generating subunit is used to input the fifth video feature to be processed into the feature normalization component, and in the feature normalization component, perform a feature normalization operation on the fifth video feature to be processed to generate a second video feature to be fused.

[0075] Among them, the video frame alignment model also includes a feature splicing layer;

[0076] The second generation module includes:

[0077] A first generating unit is used to perform normalization processing on the first video frame to generate a first normalized video frame, and to perform normalization processing on the second video frame to generate a second normalized video frame; the number of video frame channels corresponding to the first normalized video frame is equal to the number of video frame channels corresponding to the second normalized video frame;

[0078] The first generating unit is further used to input the first initial video feature corresponding to the first normalized video frame and the second initial video feature corresponding to the second normalized video frame into the feature splicing layer;

[0079] The second generating unit is used to perform feature stitching processing on the first initial video feature corresponding to the first normalized video frame and the second initial video feature corresponding to the second normalized video frame in the feature stitching layer to generate a first video feature to be processed; the number of feature channels corresponding to the first video feature to be processed is equal to twice the number of video frame channels corresponding to the first normalized video frame.

[0080] The video frame alignment model further includes a second feature convolution layer and a third feature convolution layer;

[0081] The first generation module includes:

[0082] a third generating unit, configured to perform normalization processing on the first video frame to generate a first normalized video frame;

[0083] The fourth generating unit is used to input the first initial video feature corresponding to the first normalized video frame into the second feature convolution layer, and in the second feature convolution layer, perform feature dimension increase processing on the first initial video feature corresponding to the first normalized video frame to generate an increased dimension video feature; the number of feature channels corresponding to the increased dimension video feature is greater than the number of video frame channels corresponding to the first normalized video frame;

[0084] The fifth generating unit is used to input the increased-dimensional video features into the third feature convolution layer, and in the third feature convolution layer, perform feature dimensionality reduction processing on the increased-dimensional video features to generate the first video features to be fused; the number of feature channels corresponding to the first video features to be fused is equal to the number of video frame channels corresponding to the first normalized video frame.

[0085] The data processing device further includes:

[0086] A second acquisition module is used to acquire an image super-resolution model for enhancing the clarity of video frames;

[0087] The fifth generation module is used to input the second video frame and the aligned video frame into the image super-resolution model. In the image super-resolution model, based on the aligned video frame, the second video frame is subjected to resolution enhancement processing to generate an enhanced resolution video frame corresponding to the second video frame.

[0088] Wherein, in the data processing device, the inter-frame difference between the aligned video frame and the first video frame is smaller than the inter-frame difference between the second video frame and the first video frame.

[0089] An embodiment of the present application provides a data processing device, including:

[0090] A first acquisition module is used to acquire a first sample video frame and a second sample video frame in the sample video; the sample video scene corresponding to the first sample video frame is the same as the sample video scene corresponding to the second sample video frame;

[0091] A first generating module is used to input the first sample video frame and the second sample video frame into the video frame alignment initial model respectively, and in the video frame alignment initial model, perform feature convolution processing on the first initial sample video feature corresponding to the first sample video frame to generate a first sample video feature to be fused corresponding to the first sample video frame;

[0092] A second generating module is used to perform feature splicing processing on the first initial sample video feature corresponding to the first sample video frame and the second initial sample video feature corresponding to the second sample video frame to generate a first sample video feature to be processed;

[0093] A third generating module is used to perform feature downsampling processing on the first sample video feature to be processed to generate a second sample video feature to be processed;

[0094] A fourth generating module, used to generate an alignment estimation video frame for the second sample video frame according to the second sample video feature to be processed and the first sample video feature to be fused;

[0095] A second acquisition module is used to obtain a mean square error value between the second sample video frame and the aligned estimated video frame, and determine the mean square error value as a loss value of the initial model of the video frame alignment;

[0096] The fifth generation module is used to adjust the model parameters in the initial model of the video frame alignment according to the loss value to generate a video frame alignment model; the video frame alignment model is used to perform inter-frame alignment processing on the video frames in the video.

[0097] On one hand, the present application provides a computer device, including: a processor, a memory, and a network interface;

[0098] The above-mentioned processor is connected to the above-mentioned memory and the above-mentioned network interface, wherein the above-mentioned network interface is used to provide a data communication function, the above-mentioned memory is used to store a computer program, and the above-mentioned processor is used to call the above-mentioned computer program so that the computer device executes the method in the embodiment of the present application.

[0099] On one hand, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. The computer program is suitable for being loaded by a processor and executing the method in the embodiment of the present application.

[0100] On the one hand, an embodiment of the present application provides a computer program product or a computer program, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method in the embodiment of the present application.

[0101] In an embodiment of the present application, a first video frame and a second video frame in the same video scene are obtained to determine that the moving objects in the two video frames remain consistent; a feature convolution process is performed on a first initial video feature corresponding to the first video frame to extract hidden features of the first video frame and generate a first video feature to be fused; further, a feature splicing process is performed on the first initial video feature corresponding to the first video frame and a second initial video feature corresponding to the second video frame to generate a first video feature to be processed including the first initial video feature and the second initial video feature; a feature downsampling process is performed on the first video feature to be processed to reduce the motion difference between the first initial video feature and the second initial video feature and generate a second video feature to be processed; further, an aligned video frame for the second video frame is generated based on the second video feature to be processed and the first video feature to be fused. From the above, it can be seen that the present application can reduce the inter-frame motion difference between the first video frame and the second video frame by performing feature downsampling processing on the first to-be-processed video features including the first initial video features and the second initial video features, thereby achieving high-accuracy recognition of aligned optical flow data. Therefore, the present application can improve the recognition accuracy of aligned optical flow data, and further generate high-accuracy aligned video frames. BRIEF DESCRIPTION OF THE DRAWINGS

[0102] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0103] Figure 1 It is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0104] Figure 2 It is a schematic diagram of a data processing scenario provided by an embodiment of the present application;

[0105] Figure 3 It is a flowchart of a data processing method provided in an embodiment of the present application;

[0106] Figure 4 It is a structural diagram of a video frame alignment model provided in an embodiment of the present application;

[0107] Figure 5 It is a structural schematic diagram of a feature downsampling layer provided in an embodiment of the present application;

[0108] Figure 6 It is a structural schematic diagram of a feature upsampling layer provided in an embodiment of the present application;

[0109] Figure 7 It is a schematic diagram of the structure of a first feature convolution layer provided in an embodiment of the present application;

[0110] Figure 8 It is a flowchart of a data processing method provided in an embodiment of the present application;

[0111] Fig. 9 It is a flowchart of a data processing method provided in an embodiment of the present application;

[0112] Fig.10 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0113] Fig.11 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0114] Fig.12 is a structural schematic diagram of a computer device provided in an embodiment of the present application;

[0115] Fig.13 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0116] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0117] To facilitate understanding, some nouns are briefly explained as follows:

[0118] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0119] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0120] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further perform graphics processing so that the computer processes the images into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0121] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0122] The smallest unit that constitutes a TV show (i.e., a video clip) is a shot. The sequence of shots formed by connecting shots together can be called a paragraph. Each paragraph has a single, relatively complete meaning, such as expressing an action process, expressing a correlation, expressing a meaning, etc. It is a complete narrative level in a TV show, just like a scene in a drama or a chapter in a novel. When paragraphs are connected together, a complete TV show is formed. Therefore, paragraphs are the most basic structural form of a TV show, and the structural level of a TV show in terms of content is expressed through paragraphs. It can be understood that a paragraph corresponds to a video scene. In this application, the same video scene mainly means that the paragraphs are the same. In the same video scene, each frame contains the same target object, which can be a person or an object. The target object can undergo motion changes, that is, there are object motion differences between frames.

[0123] The solution provided in the embodiments of the present application involves artificial intelligence computer vision technology and deep learning technology, which is specifically illustrated by the following embodiments.

[0124] See also Figure 1 , Figure 1 Schematic diagram of a system architecture provided by an embodiment of the present application. Figure 1 As shown, the system may include a server 10a and a user terminal cluster, and the user terminal cluster may include: user terminal 10b, user terminal 10c, ..., user terminal 10d. It can be understood that the above system may include one or more user terminals, and the number of user terminals will not be limited here.

[0125] Among them, there may be a communication connection between user terminal clusters, for example, there is a communication connection between user terminal 10b and user terminal 10c, and there is a communication connection between user terminal 10b and user terminal 10d. At the same time, any user terminal in the user terminal cluster may have a communication connection with server 10a, for example, there is a communication connection between user terminal 10b and server 10a, and there is a communication connection between user terminal 10c and server 10a. Among them, the above-mentioned communication connection does not limit the connection method, and can be directly or indirectly connected through a wired communication method, or directly or indirectly connected through a wireless communication method, or through other methods, and this application does not limit it here.

[0126] It should be understood that Figure 1 Each user terminal in the user terminal cluster shown in the figure may be installed with an application client. When the application client runs in each user terminal, it may be respectively connected to the above-mentioned Figure 1The server 10a shown in the figure performs data interaction, that is, the above-mentioned communication connection. Among them, the application client can be a social client, a multimedia client (such as a video client), an entertainment client (such as a game client), an education client, a live broadcast client, etc., which has an image loading function. Among them, the application client can be an independent client or an embedded sub-client integrated in a client (such as a social client, an education client, and a multimedia client, etc.), which is not limited here.

[0127] The server 10a provides services for the user terminal cluster through the communication connection function. When the user terminal (which can be user terminal 10b, user terminal 10c or user terminal 10d) obtains the video and needs to process the video, for example, to perform frame alignment processing on the video frames in the video, the user terminal can send the video or the video frames contained in the video to the server 10a. After the server 10a receives the video sent by the user terminal or the video frame corresponding to the video, it performs frame alignment processing on the first video frame and the second video frame in the same video scene in the video based on the video frame alignment model trained in advance, and obtains the aligned video frame corresponding to the second video frame. Subsequently, the server 10a can associate and send the video frames to be aligned (including the second video frame) and the aligned video frames corresponding to the video frames to be aligned to the user terminal.

[0128] After the user terminal receives the video frames to be aligned and the aligned video frames sent by the server 10a, the video frames to be aligned and the aligned video frames can be displayed on the corresponding screen. If the user terminal stores a trained video frame alignment model locally, the video frames in the video can be aligned locally in the user terminal, and the alignment process is consistent with the process of the server 10a performing the inter-frame alignment process. Among them, since the training of the video frame alignment model involves a large amount of offline calculations, the video frame alignment model local to the user terminal can be sent to the user terminal after the server 10a completes the training.

[0129] in, Figure 1 The server 10a, user terminal 10b, user terminal 10c, ..., user terminal 10d may include a mobile phone, a tablet computer, a laptop computer, a PDA, a smart speaker, a mobile Internet device (MID), a POS (Point Of Sales), a wearable device (such as a smart watch, a smart bracelet, etc.), etc.

[0130] It should be noted that the above-mentioned server 10a, user terminal 10b, user terminal 10c, ..., user terminal 10d can all be blockchain nodes in the blockchain network, and the data described in the full text (such as the above-mentioned second video frame and aligned video frame) can be stored. The storage method can be that the blockchain node generates blocks according to the data and adds the blocks to the blockchain for storage.

[0131] Blockchain is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism and encryption algorithm. It is mainly used to organize data in chronological order and encrypt it into a ledger to make it impossible to be tampered with or forged. At the same time, it can verify, store and update data. Blockchain is essentially a decentralized database. Each node in the database stores the same blockchain. The blockchain network can divide nodes into core nodes, data nodes and light nodes. Core nodes, data nodes and light nodes together constitute blockchain nodes. Among them, the core node is responsible for the consensus of the entire blockchain network, that is, the core node is the consensus node in the blockchain network. The process of writing transaction data in the blockchain network into the ledger can be that the data node or light node in the blockchain network obtains the transaction data, and transmits the transaction data in the blockchain network (that is, the node transmits it in the form of a baton) until the consensus node receives the transaction data, and the consensus node then packs the transaction data into a block, executes consensus on the block, and writes the transaction data into the ledger after the consensus is completed. Here, the transaction data of the second video frame and the aligned video frame are used as examples. After reaching a consensus on the transaction data, the server 10a (blockchain node) generates a block based on the transaction data and stores the block in the blockchain network. As for the reading of the transaction data (i.e., the second video frame and the aligned video frame), the blockchain node can obtain the block containing the transaction data in the blockchain network, and further obtain the transaction data in the block.

[0132] It is understandable that the data processing method provided in the embodiment of the present application can be executed by a computer device, and the computer device includes but is not limited to a user terminal or a server. The above-mentioned server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The above-mentioned user terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a car terminal, a smart TV, etc., but is not limited to this. The user terminal and the server can be directly or indirectly connected by wired or wireless communication, and this application is not limited here.

[0133] For further information, see Figure 2 , Figure 2 Schematic diagram of a data processing scenario provided by an embodiment of the present application. Figure 2 As shown, video 20a includes at least two video frames, and the at least two video frames may include a first video frame 201a, ..., and a second video frame 202a, wherein the first video frame 201a and the second video frame 202a may be adjacent frames, for example, in video 20a, the next frame of the first video frame 201a is the second video frame 202a, and optionally, the first video frame 201a and the second video frame 202a may not be adjacent frames, for example, in video 20a, the next frame of the first video frame 201a is not the second video frame 202a, and the adjacent relationship between the first video frame 201a and the second video frame 202a is not limited here, but the video scenes to which the two frames belong are the same video scene, such as Figure 2 As shown, the first video frame 201a and the second video frame 202a both include a store, an escalator, and a portrait (i.e. Figure 2 little girl in the film).

[0134] The computer device obtains the first video frame 201a and the second video frame 202a from the video 20a, and obtains the pre-trained video frame alignment model 20c. Please refer to Figure 2 , the video frame alignment model 20c provided in the embodiment of the present application may include two branch networks, namely Figure 2 The illustrated branch network 201b and the branch network 202b. The computer device inputs the first video frame 201a to the branch network 201b, and inputs the first video frame 201a and the second video frame 202a to the branch network 202b respectively.

[0135] In the branch network 201b, the computer device performs feature extraction on the first video frame 201a to generate a first video feature 201c to be fused corresponding to the first video frame 201a; in the branch network 202b, feature extraction is performed on the initial video features corresponding to the first video frame 201a and the second video frame 202a, respectively, to generate a second video feature 202c to be fused for the second video frame 202a; in the video frame alignment model 20c, the computer device performs feature fusion processing on the first video feature 201c to be fused and the second video feature 202c to be fused to obtain a normalized video feature 201d for the second video frame 202a. The computer device performs denormalization processing on the normalized video feature 201d to obtain an aligned video frame 201e for the second video frame 202a. Among them, the static objects (such as Figure 2There is no difference between the example store and escalator in the example, but there is an inter-frame difference between the aligned video frame 201e and the first video frame 201a (such as Figure 2 example girl), is smaller than the inter-frame difference between the second video frame 202a and the first video frame 201a.

[0136] The internal structure of the video frame alignment model 20c is not described here, and can be found in the following Figure 3 A detailed description of the model in the corresponding embodiment.

[0137] For further information, see Figure 3 , Figure 3 is a flow chart of a data processing method provided in an embodiment of the present application. The data processing method may be executed by a computer device, which may include Figure 1 The user terminals and servers in Figure 3 As shown, the data processing process may include the following steps.

[0138] Step S101, obtaining a first video frame and a second video frame in a video; a video scene corresponding to the first video frame is the same as a video scene corresponding to the second video frame.

[0139] Specifically, the video in the embodiments of the present application can be a video of any format and any size, such as the video format can be Moving Picture Experts Group 4 (MP4 for short, a set of compression encoding standard formats for audio and video information), or it can be an Audio Video Interleaved (AVI) format; the video size can be 512*384, 1024*534, etc.

[0140] The computer device obtains the first video frame and the second video frame from the video. It should be noted that the embodiment of the present application does not limit the adjacent relationship between the first video frame and the second video frame, and can be set according to the actual application scenario. When the video scene corresponding to the first video frame is the same as the video scene corresponding to the second video frame, the two video frames can be non-adjacent video frames or adjacent video frames.

[0141] The embodiment of the present application does not limit the technology for acquiring video frames in a video, and the technology may be a computer programming language (such as Python-OpenCv), etc. The embodiment of the present application does not limit the technology for performing scene detection on a video, and the technology may be a histogram-based technology, a pixel-based technology, a deep learning-based technology, etc.

[0142] It can be understood that a video frame is an image. The embodiment of the present application does not limit the image format of the video frame. The video frame can be an image in any format, such as Portable Network Graphics (PNG), Bitmap (BMP), Joint Photographic Experts Group (one of the most commonly used image file formats, JPEG), etc.

[0143] In addition, the video frame can be a color image (RGB) having three color channels of red, green, and blue; the video frame can be a grayscale image having a brightness channel (Y component). The embodiment of the present application does not limit the image type to which the video frame belongs and can be set according to the actual application scenario.

[0144] Step S102: performing feature convolution processing on the first initial video feature corresponding to the first video frame to generate a first video feature to be fused corresponding to the first video frame.

[0145] Specifically, the video frame alignment model includes a second feature convolution layer and a third feature convolution layer; the first video frame is normalized to generate a first normalized video frame; the first initial video feature corresponding to the first normalized video frame is input into the second feature convolution layer, and in the second feature convolution layer, the first initial video feature corresponding to the first normalized video frame is subjected to feature dimension increase processing to generate an increased dimension video feature; the number of feature channels corresponding to the increased dimension video feature is greater than the number of video frame channels corresponding to the first normalized video frame; the increased dimension video feature is input into the third feature convolution layer, and in the third feature convolution layer, the increased dimension video feature is subjected to feature dimension reduction processing to generate a first video feature to be fused; the number of feature channels corresponding to the first video feature to be fused is equal to the number of video frame channels corresponding to the first normalized video frame.

[0146] The computer device first normalizes the first video frame, that is, the pixel values ​​corresponding to each pixel point in the first video frame are processed to be between [0, 1], and a first normalized video frame corresponding to the first video frame is generated; for eight-bit pixels, each pixel value is divided by 255, and for 10-bit pixels, each pixel value is divided by 1023. The same can be said for other bit pixels, so they are not described one by one. Similarly, the computer device normalizes the second video frame to generate a second normalized video frame corresponding to the second video frame.

[0147] Please also see Figure 4 , Figure 4 Schematic diagram of a video frame alignment model provided in an embodiment of the present application. Figure 4As shown, the video frame alignment model may include a second feature convolution layer 40b and a third feature convolution layer 40c. The computer device inputs the first initial video feature 401a corresponding to the first normalized video frame into the second feature convolution layer 40b, as shown in FIG. Figure 4 As shown, the number of feature channels corresponding to the first initial video feature 401a is C. It can be understood that when the first video frame is an RGB image, its channel number is 3, then C is equal to 3; when the first video frame is a y component, its channel number is 1, then C is equal to 1.

[0148] In the second feature convolution layer 40b, the computer device not only performs feature convolution processing on the first initial video feature 401a, but also performs feature dimension increase processing on the first initial video feature 401a, such as Figure 4 As shown, the second feature convolution layer 40b includes D convolution kernels, and D can be much larger than C (equal to the number of video frame channels corresponding to the first normalized video frame). Therefore, through the second feature convolution layer 40b, an increased-dimensional video feature 402b can be generated; the computer device inputs the increased-dimensional video feature 402b to the third feature convolution layer 40c, as shown in FIG. Figure 4 As shown, the number of convolution kernels of the third feature convolution layer 40c is equal to C. Therefore, in the third feature convolution layer 40c, not only the feature convolution processing is performed on the increased-dimensional video feature 402b, but also the feature dimensionality reduction processing can be performed on the increased-dimensional video feature 402b to generate the first video feature 401c to be fused. Obviously, the number of feature channels corresponding to the first video feature 401c to be fused is equal to the number of video frame channels corresponding to the first normalized video frame, and both are C.

[0149] The second feature convolution layer 40b and the third feature convolution layer 40c both include an activation function. The embodiment of the present application does not limit the type of the activation function, which may be a Sigmod function, a Tanh function, a Rectified Linear Unit (Relu) function, or the like.

[0150] It can be understood that the embodiment of the present application can extract features from the first initial video feature 401a through the second feature convolution layer 40b and the third feature convolution layer 40c, and perform a dimensionality increase operation on the first initial video feature 401a through the second feature convolution layer 40b, so that the detailed content in the first video frame can be learned; through the third feature convolution layer 40c, a dimensionality reduction operation is performed on the increased-dimensional video feature 402b to facilitate subsequent feature fusion processing.

[0151] Understandably, Figure 4Only one pair of feature convolution layers is described in the figure (a second feature convolution layer 40b for dimensionality increase operation, and a third feature convolution layer 40c for dimensionality reduction operation). However, in actual applications, feature convolution layer pairs can be added according to the scenario. It is only necessary to ensure that the number of layers of the second feature convolution layer for dimensionality increase operation is equal to the number of layers of the third feature convolution layer for dimensionality reduction operation. Optionally, the number of layers of the second feature convolution layer for dimensionality increase operation may not be equal to the number of layers of the third feature convolution layer for dimensionality reduction operation. It is only necessary to ensure that the number of feature channels corresponding to the first video feature to be fused in the final output is equal to the number of video frame channels corresponding to the first normalized video frame.

[0152] Step S103, performing feature splicing processing on the first initial video feature corresponding to the first video frame and the second initial video feature corresponding to the second video frame to generate a first video feature to be processed.

[0153] Specifically, the video frame alignment model includes a feature stitching layer; the first video frame is normalized to generate a first normalized video frame, and the second video frame is normalized to generate a second normalized video frame; the number of video frame channels corresponding to the first normalized video frame is equal to the number of video frame channels corresponding to the second normalized video frame; the first initial video feature corresponding to the first normalized video frame and the second initial video feature corresponding to the second normalized video frame are input into the feature stitching layer; in the feature stitching layer, the first initial video feature corresponding to the first normalized video frame and the second initial video feature corresponding to the second normalized video frame are feature stitched to generate a first video feature to be processed; the number of feature channels corresponding to the first video feature to be processed is equal to twice the number of video frame channels corresponding to the first normalized video frame.

[0154] The process of the computer device performing normalization processing on the first video frame and the second video frame respectively can be referred to the description in step S102, which will not be described again here.

[0155] Please see again Figure 4 , the video frame alignment model may include a feature splicing layer 40d, the computer device inputs the first initial video feature 401a and the second initial video feature 402a into the feature splicing layer 40d; in the feature splicing layer 40d, the first initial video feature 401a and the second initial video feature 402a are subjected to feature splicing processing to generate a first video feature to be processed 40e; obviously, the number of feature channels corresponding to the first video feature to be processed 40e is equal to twice the number of video frame channels corresponding to the first normalized video frame. For example, if the first video frame and the second video frame are both RGB images, the number of video frame channels corresponding to the two frames is 3, and the number of feature channels corresponding to the first video feature to be processed is equal to 6.

[0156] It is understandable that the computer device may splice the first initial video feature 401a to the back of the second initial video feature 402a, or may splice the second initial video feature 402a to the back of the first initial video feature 401a.

[0157] Step S104: perform feature downsampling processing on the first video feature to be processed to generate a second video feature to be processed.

[0158] Specifically, a video frame alignment model is obtained; the video frame alignment model includes A feature downsampling layers, and the A feature downsampling layers include feature downsampling layers B c And feature downsampling layer B c+1 , A is a positive integer, c is a positive integer and c is less than A; in the feature downsampling layer B c In the process, the first input feature is downsampled to generate the first transition video feature D c ; If the feature downsampling layer B c is the first feature downsampling layer among A feature downsampling layers, then the first input feature is the first video feature to be processed; if feature downsampling layer B c If it is not the first feature downsampling layer among A feature downsampling layers, then the first input feature is, feature downsampling layer B c The first transition video feature D is output by the previous feature downsampling layer; c As feature downsampling layer B c+1 The first input feature of the feature downsampling layer B c+1 In the first transition video feature D c Perform feature downsampling to generate the first transition video feature D c+1 ; If the feature downsampling layer B c+1 If it is not the last feature downsampling layer among the A feature downsampling layers, the first transition video feature D c+1 Determined as, feature downsampling layer B c+1 The first input feature of the next feature downsampling layer; if the feature downsampling layer B c+1 is the last feature downsampling layer among the A feature downsampling layers, then the first transition video feature D c+1 Determined as the second video feature to be processed.

[0159] Among them, the feature downsampling layer B c Including a first convolution component and a feature pooling component; generating a first transition video feature D cThe specific process may include: inputting the first input feature into the first convolution component, performing feature convolution processing on the first input feature in the first convolution component to generate a video feature to be pooled; inputting the video feature to be pooled into the feature pooling component; the feature pooling component carries the pooling size; according to the pooling size, performing feature downsampling processing on the video feature to be pooled in the feature pooling component to generate a first transition video feature D c ; The feature resolution corresponding to the first input feature is equal to the pooling size and the first transition video feature D c The product of the corresponding feature resolution.

[0160] Among them, the first convolution component includes the first convolution component G h , and the first convolutional component G h The associated activation component, the first convolution component G h+1 , and the first convolutional component G h+1 The associated fusion component and the first convolution component G h+1 The associated activation component; h is a positive integer, and h is less than the total number of the first convolutional components; the specific process of generating the video feature to be pooled may include: inputting the first input feature into the first convolutional component G h ; The first convolutional component G h It includes E convolution channels, where E is a positive integer; through the first convolution component G h The E convolution channels in the first input feature are processed by feature convolution to generate the first video feature I to be activated. h ; The first video feature to be activated I h The corresponding feature channel number is equal to E; the first video feature I to be activated h Input to the first convolution component G h The associated activation component is connected to the first convolution component G h The associated activation component, for the first video feature I to be activated h Perform feature nonlinear processing to generate the first convolution video feature J h ; The first convolutional video feature J h Input to the first convolutional component G h+1 ; The first convolutional component G h+1 Includes E convolution channels; through the first convolution component G h+1 The E convolution channels in the first convolution video feature J h Perform feature convolution processing to generate the first video feature I to be activated h+1 ; The first video feature to be activated I h+1 The corresponding feature channel number is equal to E; the first video feature I to be activated h+1 And the first convolutional video feature J h Input to the first convolution component Gh+1 The associated fusion component, in conjunction with the first convolutional component G h+1 In the associated fusion component, the first video feature I to be activated h+1 And the first convolutional video feature J h Perform feature fusion processing to generate the first video feature I to be activated h+2 ; Set the first video feature to be activated I h+2 Input to the first convolution component G h+1 The associated activation component is connected to the first convolution component G h+1 The associated activation component, for the first video feature I to be activated h+2 Perform nonlinear feature processing to generate video features to be pooled.

[0161] Please see again Figure 4 After generating the first video feature 40e to be processed, the computer device inputs the first video feature 40e to be processed into A feature down-sampling layers 40f, wherein the feature down-sampling layer can also be called a residual feature down-sampling block (RFDB). Please refer to Figure 5 , Figure 5 Schematic diagram of a feature downsampling layer provided in an embodiment of the present application. Figure 5 As shown, the computer device inputs the first to-be-processed video feature 40e into the first feature downsampling layer (equal to the feature downsampling layer B described above). c＝1 ), the first feature downsampling layer may include a first convolution component 4011f and a feature pooling component 4013f. The structure of the first convolution component 4011f is as follows: Figure 5 As shown on the right side of , it can include a first convolution component G1, an activation component associated with the first convolution component G1 (equivalent to Figure 5 The activation function 1 in ), the first convolution component G2, the fusion component associated with the first convolution component G2, and the activation component associated with the first convolution component G2 (equivalent to Figure 5 The activation function in 2).

[0162] The computer device first inputs the first video feature 40e to be processed (the corresponding feature channel number is 2C) into the first convolution component G1. The embodiment of the present application does not limit the size of the convolution kernel of the first convolution component G1, and the size of the convolution kernel should be determined according to the actual application. Similarly, the number of convolution kernels E (i.e., the number of convolution channels) of the first convolution component G1 should be set according to the actual application. It can be understood that in order to capture features, E can be much larger than C. For example, the first convolution component G1 includes a 3x3 convolution kernel with a channel number E of 64. Since the linear model has insufficient expressive power, the computer device needs to use an activation function to add nonlinear factors, so the video feature to be activated output by the first convolution component G1 is input into the activation function 1. In the embodiment of the present application, if the type of activation function is not clearly marked, the type of activation function is not limited, and it can be a Sigmod function, a Tanh function, a rectified linear unit (Rectified Linear Unit, Relu) function, a LeakRelu activation function, etc.

[0163] Further, the computer device inputs the convolution video feature output by the activation function 1 into the first convolution component G2, wherein the total number of convolution channels corresponding to the first convolution component G2 is equal to the total number of convolution channels corresponding to the first convolution component G1; through the E convolution channels in the first convolution component G2, the convolution video feature output by the activation function 1 is subjected to feature convolution processing to generate the first video feature to be activated I2. Figure 5 As shown, the embodiment of the present application uses a residual connection (i.e., a fusion component) to perform feature fusion processing on the input of the first convolution component G2 (including the convolution video features output by the above-mentioned activation function 1) and the output (the first video feature to be activated I2) to generate the first video feature to be activated I3. The shallow residual design benefits from residual learning and is lightweight enough to improve the feature learning ability of the network structure.

[0164] The computer device uses activation function 2 to enhance nonlinearity, that is, the first video feature I3 to be activated is input into the activation component 2 to generate the video feature 4012f to be pooled. Finally, the computer device uses a feature pooling component 4013f to perform feature downsampling processing on the video feature 4012f to be pooled, which can not only extract features, but also reduce the position offset caused by the movement of objects between the first video frame and the second video frame, and obtain the second video feature 401f to be processed. In the embodiment of the present application, the pooling size and pooling type of the feature pooling component 4013f are not limited, and can be set according to the actual application scenario. In general, the pooling size can be set to 2, that is, the resolution of the video feature 4012f can be reduced. In general, the pooling type can be set to maximum pooling.

[0165] It is understandable that in order to control complexity, the computer device can set the A feature downsampling layers in the video frame alignment model to 1 feature downsampling layer. In this case, the first transition video feature described above does not exist, and the first input feature is fixed to the first video feature to be processed; it is understandable that A can be a positive integer greater than 1, for example, A is equal to 2, then the first video feature to be processed 40e can be feature downsampled twice, in which case the feature resolution (feature size) corresponding to the second video feature to be processed 401f should be reduced by two pooling sizes.

[0166] Optional, in Figure 5 In the example structure, the number of convolution channels corresponding to the feature pooling component 4013f is equal to the number of convolution channels corresponding to the first convolution component 4011f. In actual applications, the convolution channels corresponding to the two may not be equal.

[0167] Step S105: generating an aligned video frame for the second video frame according to the second video feature to be processed and the first video feature to be fused.

[0168] Specifically, the video frame alignment model also includes A feature upsampling layers, a first feature convolution layer, and a feature fusion layer; through the A feature upsampling layers, the second video feature to be processed is subjected to feature upsampling processing to generate a third video feature to be processed, and the third video feature to be processed is input into the first feature convolution layer; the feature resolution corresponding to the third video feature to be processed is equal to the feature resolution corresponding to the first video feature to be processed; through the first feature convolution layer, the third video feature to be processed is subjected to feature convolution processing to generate a second video feature to be fused; the number of feature channels corresponding to the second video feature to be fused is equal to the number of feature channels corresponding to the first video feature to be fused; the first video feature to be fused and the second video feature to be fused are input into the feature fusion layer, and in the feature fusion layer, the first video feature to be fused and the second video feature to be fused are subjected to feature fusion processing to generate a normalized video feature for the second video frame; the normalized video feature is subjected to feature denormalization processing to generate an aligned video frame. The inter-frame difference between the aligned video frame and the first video frame is less than the inter-frame difference between the second video frame and the first video frame.

[0169] Among them, A feature upsampling layers include feature upsampling layer K m And the feature upsampling layer K m+1 , m is a positive integer and m is less than A; the specific process of generating the third to-be-processed video feature may include: sampling layer K on the feature m In the process, the second input feature is subjected to feature upsampling to generate the second transition video feature L m ; If the feature upsampling layer K mis the first feature upsampling layer among the A feature upsampling layers, then the second input feature is the second video feature to be processed; if the feature upsampling layer K m If it is not the first feature upsampling layer among the A feature upsampling layers, then the second input feature is, feature upsampling layer K m The second transition video feature output by the previous feature upsampling layer; the second transition video feature L m As the feature upsampling layer K m+1 The second input feature of m+1 In the second transition video feature L m Perform feature upsampling to generate the second transition video feature L m+1 ; If the feature upsampling layer K m+1 If it is not the last feature upsampling layer among the A feature upsampling layers, the second transition video feature L m+1 Determined as, the feature upsampling layer K m+1 The second input feature of the next feature upsampling layer; if the feature upsampling layer K m+1 is the last feature upsampling layer in the A feature upsampling layers, then the second transition video feature L m+1 Determined as the third video feature to be processed.

[0170] Among them, the feature upsampling layer K m Including a second convolution component and a feature upsampling component; generating a second transition video feature L m The specific process may include: inputting the second input feature into the second convolution component, performing feature convolution processing on the second input feature in the second convolution component to generate a to-be-upsampled video feature; inputting the to-be-upsampled video feature into the feature upsampling component; the feature upsampling component carries the upsampling size; performing feature upsampling processing on the to-be-upsampled video feature in the feature upsampling component according to the upsampling size to generate a second transition video feature L m ; The second transition video feature L m The corresponding feature resolution is equal to the product of the feature resolution corresponding to the second input feature and the upsampling size.

[0171] Among them, the second convolution component includes the second convolution component N o , and the second convolutional component N o The associated activation component, the second convolution component N o+1 , and the second convolutional component N o+1 The associated fusion component and the second convolution component N o+1 The associated activation component; o is a positive integer, and o is less than the total number of the second convolution components; the specific process of generating the video feature to be upsampled may include: inputting the second input feature into the second convolution component No ; The second convolutional component N o It includes P convolution channels, where P is a positive integer; through the second convolution component N o The P convolution channels in the image are used to perform feature convolution processing on the second input feature to generate the second video feature Q to be activated. o ; The second video feature Q to be activated o The corresponding number of feature channels is equal to P; the second video feature to be activated Q o Input to the second convolution component N o The associated activation component is connected to the second convolution component N o The associated activation component is for the second video feature Q to be activated o Perform feature nonlinear processing to generate the second convolution video feature R o ; The second convolution video feature R o Input to the second convolutional component N o+1 ; The second convolutional component N o+1 It includes P convolution channels; through the second convolution component N o+1 The P convolution channels in the second convolution video feature R o Perform feature convolution processing to generate the second video feature Q to be activated o+1 ; The second video feature Q to be activated o+1 The corresponding number of feature channels is equal to P; the second video feature to be activated Q o+1 And the second convolutional video feature R o Input to the second convolution component N o+1 The associated fusion component, in conjunction with the second convolutional component N o+1 In the associated fusion component, the second video feature Q to be activated o+1 And the second convolutional video feature R o Perform feature fusion processing to generate the second video feature Q to be activated o+2 ; Set the second video feature Q to be activated o+2 Input to the second convolution component N o+1 The associated activation component is connected to the second convolution component N o+1 The associated activation component is for the second video feature Q to be activated o+2 Perform feature nonlinear processing to generate video features to be upsampled.

[0172] Among them, the first feature convolution layer includes a feature convolution component, a feature activation component and a feature normalization component; the specific process of generating the second video feature to be fused may include: inputting the third video feature to be processed into the feature convolution component, and performing feature convolution processing on the third video feature to be processed in the feature convolution component to generate a fourth video feature to be processed; the number of feature channels corresponding to the fourth video feature to be processed is equal to the number of feature channels corresponding to the first video feature to be fused; inputting the fourth video feature to be processed into the feature activation component, and performing feature nonlinear processing on the fourth video feature to be processed in the feature activation component to generate a fifth video feature to be processed; inputting the fifth video feature to be processed into the feature normalization component, and performing feature normalization operation on the fifth video feature to be processed in the feature normalization component to generate the second video feature to be fused.

[0173] Please see again Figure 4 After generating the second video feature 401f to be processed, the computer device inputs the second video feature 40e to be processed into A feature up-sampling layers 40g, wherein the feature down-sampling layer can also be called a residual feature up-sampling module (Residual Feature Up-sample Block, RFDB). Please refer to Figure 6 , Figure 6 Schematic diagram of a feature upsampling layer provided in an embodiment of the present application. Figure 6 As shown, the computer device inputs the second to-be-processed video feature 401f into the first feature upsampling layer (equal to the feature upsampling layer K described above). m＝1 ), the first feature upsampling layer may include a second convolution component 4011g and a feature upsampling component 4013g. The structure of the second convolution component 4011g is as follows: Figure 6 As shown on the right side of , it can include a second convolution component N1, an activation component associated with the second convolution component N1 (equivalent to Figure 6 The activation function 1 in ), the second convolution component N2, the fusion component associated with the second convolution component N2, and the activation component associated with the second convolution component N2 (equivalent to Figure 6 The activation function in 2).

[0174] The computer device first inputs the second video feature 401f to be processed into the second convolution component N1. The embodiment of the present application does not limit the size of the convolution kernel of the second convolution component N1. The size of the convolution kernel should be determined according to the actual application. Similarly, the number P of the convolution kernel of the second convolution component N1 (i.e., the number of convolution channels) should be set according to the actual application. It can be understood that in order to capture features, P can be much larger than C. For example, the second convolution component N1 includes a 3x3 convolution kernel with a channel number P of 32. Since the linear model has insufficient expression power, the computer device needs to use an activation function to add nonlinear factors, so the video feature to be activated output by the second convolution component N1 is input into the activation function 1.

[0175] Further, the computer device inputs the convolution video feature output by the activation function 1 into the second convolution component N2, wherein the total number of convolution channels corresponding to the second convolution component N2 is equal to the total number of convolution channels corresponding to the second convolution component N1; through the P convolution channels in the second convolution component N2, the convolution video feature output by the activation function 1 is subjected to feature convolution processing to generate the second video feature O2 to be activated. Figure 6 As shown, the embodiment of the present application uses a residual connection (i.e., a fusion component) to perform feature fusion processing on the input of the second convolution component N2 (including the convolution video features output by the above-mentioned activation function 1) and the output (the second video feature to be activated O2) to generate the second video feature to be activated O3.

[0176] Similarly, the computer device uses activation function 2 to enhance nonlinearity, that is, inputs the second video feature O3 to be activated into the activation component 2 to generate the video feature 4012g to be upsampled. Finally, the computer device uses a feature upsampling component 4013g to perform feature upsampling processing on the video feature 4012g to be upsampled, which can not only fuse features, but also eliminate feature offsets, realize inter-frame information alignment and fusion, and obtain the third video feature 401g to be processed, wherein the feature resolution corresponding to the third video feature 401g to be processed is equal to the feature resolution corresponding to the first video feature to be processed. It can be understood that, assuming Figure 5 The pooling size in is 2*2, then Figure 6 The upsampling size in is also 2*2.

[0177] Understandably, Figure 4The number of layers corresponding to the feature downsampling layer 40f and the feature downsampling layer 40g in the image processing module must be consistent, that is, the number of times the computer device performs feature downsampling processing on the first video feature 40e to be processed is equal to the number of times the computer device performs feature upsampling processing on the second video feature 401f to be processed, so as to restore the original size (resolution) corresponding to the first video feature 40e to be processed. In the feature upsampling component, the pixels (depth information) in the channel direction can be supplemented to the pixels (spatial information) in the width and height directions to achieve upsampling.

[0178] Combination Figure 5 as well as Figure 6 It can be seen that the feature downsampling layer 40f and the feature upsampling layer 40g in the embodiment of the present application are similar to the structure of the codec, which restores the original width and height dimensions, that is, the first video feature 40e to be processed is first subjected to feature downsampling processing, which can reduce the position offset between the first video frame and the second video frame due to the movement of the object, and then the second video feature 401f to be processed is subjected to feature upsampling processing, which is helpful to fuse the features, eliminate the feature offset, and realize the alignment and fusion of the information between the frames. Obviously, the embodiment of the present application achieves high performance through simple operations.

[0179] Optional, in Figure 6 In the example structure, the number of convolution channels corresponding to the feature upsampling component 4013g is equal to the number of convolution channels corresponding to the second convolution component 4011g. In actual applications, the convolution channels corresponding to the two may not be equal.

[0180] From the above, it can be seen that the number of feature channels corresponding to the third to-be-processed video feature 401g (equal to P) is not equal to the number of feature channels corresponding to the first to-be-fused video feature 401c (equal to C). Figure 4 As shown, the computer device inputs the third to-be-processed video feature 401g into the first feature convolution layer 40h, please refer to Figure 7 , Figure 7 is a schematic diagram of the structure of a first feature convolution layer provided in an embodiment of the present application. Figure 7 As shown, the first feature convolution layer 40h includes a feature convolution component (whose convolution channel number is equal to C), a feature activation component (equivalent to Figure 7The computer device inputs the third video feature 401g to be processed into the feature convolution component, which can not only extract the content in the third video feature 401g to be processed, but also play a role in dimension increase or dimension reduction. Specifically, if the number of feature channels corresponding to the third video feature 401g to be processed is greater than C, the feature convolution component plays a role in dimension reduction. If the number of feature channels corresponding to the third video feature 401g to be processed is less than C (for example, C is equal to 3, and the number of feature channels corresponding to the third video feature 401g to be processed is less than 3), the feature convolution component plays a role in dimension increase.

[0181] The computer device inputs the fourth video feature to be processed into the feature activation component. In order to make the network converge faster and reduce the number of iterative updates during training, the activation function here preferably uses the Tanh activation function. Since the output value range of the Tanh activation function is [-1, 1], in the feature activation component, after the fourth video feature to be processed is subjected to feature nonlinear processing, the feature value range of the generated fifth video feature to be processed is [-1, 1], and the output feature value range of the third feature convolution layer 40c is [0, 1]. For unification, the computer device performs a normalization process (De-process) operation on the fifth video feature to be processed to generate the second video feature to be fused 402c. The specific normalization operation is shown in Formula 1.

[0182] De-process(x)=(x+1) / 2 (1)

[0183] The x on the right side of formula (1) represents the feature value at each position in the fifth to-be-processed video feature, and the x on the left side represents the feature value after the feature value at each position is normalized.

[0184] Please see again Figure 4 The computer device performs feature fusion processing on the first video feature 401c to be fused and the second video feature 402c to be fused through the feature fusion layer 40k to obtain a normalized video feature 401k for the second video frame. Subsequently, the normalized video feature 401k is denormalized. For example, the previous normalization operation is to divide the pixel value by 255, then multiply it by 255 here to obtain an aligned video frame.

[0185] In an embodiment of the present application, a first video frame and a second video frame in the same video scene are obtained to determine that the moving objects in the two video frames remain consistent; a feature convolution process is performed on a first initial video feature corresponding to the first video frame to extract hidden features of the first video frame and generate a first video feature to be fused; further, a feature splicing process is performed on the first initial video feature corresponding to the first video frame and a second initial video feature corresponding to the second video frame to generate a first video feature to be processed including the first initial video feature and the second initial video feature; a feature downsampling process is performed on the first video feature to be processed to reduce the motion difference between the first initial video feature and the second initial video feature and generate a second video feature to be processed; further, an aligned video frame for the second video frame is generated based on the second video feature to be processed and the first video feature to be fused. From the above, it can be seen that the present application can reduce the inter-frame motion difference between the first video frame and the second video frame by performing feature downsampling processing on the first to-be-processed video features including the first initial video features and the second initial video features, thereby achieving high-accuracy recognition of aligned optical flow data. Therefore, the present application can improve the recognition accuracy of aligned optical flow data, and further generate high-accuracy aligned video frames.

[0186] For further information, see Figure 8 , Figure 8 is a flow chart of a data processing method provided in an embodiment of the present application. The data processing method may be executed by a computer device, which may include Figure 1 The user terminals and servers in Figure 8 As shown, the data processing process may include the following steps.

[0187] Step S201, obtaining a first video frame and a second video frame in a video; a video scene corresponding to the first video frame is the same as a video scene corresponding to the second video frame.

[0188] Step S202: performing feature convolution processing on the first initial video feature corresponding to the first video frame to generate a first video feature to be fused corresponding to the first video frame.

[0189] Step S203, performing feature splicing processing on the first initial video feature corresponding to the first video frame and the second initial video feature corresponding to the second video frame to generate a first video feature to be processed.

[0190] Step S204: perform feature downsampling processing on the first video feature to be processed to generate a second video feature to be processed.

[0191] Step S205, generating an aligned video frame for the second video frame according to the second video feature to be processed and the first video feature to be fused; the inter-frame difference between the aligned video frame and the first video frame is smaller than the inter-frame difference between the second video frame and the first video frame.

[0192] The specific process of step S201 to step S205 can be found in the above Figure 3 The description in the corresponding embodiment will not be repeated here.

[0193] Video information flow products are currently the hottest Internet products. With the development of the Internet, the demand for videos on mobile terminals or terminals is increasing. Optical flow estimation is the basis for solving many other problems in video tasks, such as motion estimation, motion segmentation, video super-resolution, etc. The aligned video frames generated by this application serve as a basis and play an important role in conjunction with other algorithms. For example, in video super-resolution, it is necessary to solve inter-frame alignment and use adjacent frame information to enhance the detail generation capability of the super-resolution network. For the specific process, please refer to steps S206-S207 below.

[0194] Step S206, obtaining an image super-resolution model for enhancing the clarity of video frames.

[0195] Step S207, input the second video frame and the aligned video frame into the image super-resolution model, and in the image super-resolution model, perform resolution enhancement processing on the second video frame based on the aligned video frame to generate an enhanced resolution video frame corresponding to the second video frame.

[0196] Specifically, based on a pre-trained video frame alignment model, a computer device performs inter-frame alignment processing on a first video frame and a second video frame in the same video scene in the video to obtain an aligned video frame corresponding to the second video frame; the computer device inputs the second video frame and the aligned video frame output by the video frame alignment model into a trained image super-resolution model, performs noise removal processing on the second video frame in the image super-resolution model, performs resolution enhancement processing on the second video frame after noise removal based on the aligned video frame, and generates an enhanced resolution video frame corresponding to the second video frame.

[0197] It can be understood that the above describes the process of enhancing the resolution of the second video frame by taking the first video frame and the second video frame as examples. The computer device can perform resolution enhancement processing on other video frames in the video through the above description, that is, first perform inter-frame alignment processing and then perform resolution enhancement processing.

[0198] It can be understood that after the video frames are aligned, the super-resolution processing can be performed on the video frames, which can mine the detailed texture in the video frames, and the overall picture clarity can be greatly improved. In addition, the aligned video frame corresponding to the second video frame includes not only the hidden features in the second video frame, but also the hidden features in the first video frame. Therefore, the effect of super-resolution processing of the second video frame using the aligned video frame is better than the effect of super-resolution processing of the second video frame using the first video frame.

[0199] In an embodiment of the present application, a first video frame and a second video frame in the same video scene are obtained to determine that the moving objects in the two video frames remain consistent; a feature convolution process is performed on a first initial video feature corresponding to the first video frame to extract hidden features of the first video frame and generate a first video feature to be fused; further, a feature splicing process is performed on the first initial video feature corresponding to the first video frame and a second initial video feature corresponding to the second video frame to generate a first video feature to be processed including the first initial video feature and the second initial video feature; a feature downsampling process is performed on the first video feature to be processed to reduce the motion difference between the first initial video feature and the second initial video feature and generate a second video feature to be processed; further, an aligned video frame for the second video frame is generated based on the second video feature to be processed and the first video feature to be fused. From the above, it can be seen that the present application can reduce the inter-frame motion difference between the first video frame and the second video frame by performing feature downsampling processing on the first to-be-processed video features including the first initial video features and the second initial video features, thereby achieving high-accuracy recognition of aligned optical flow data. Therefore, the present application can improve the recognition accuracy of aligned optical flow data, and further generate high-accuracy aligned video frames.

[0200] For further information, see Fig. 9 , Fig. 9 is a flow chart of a data processing method provided in an embodiment of the present application. The data processing method may be executed by a computer device, which may include Figure 1 The user terminals and servers in Fig. 9 As shown, the data processing process may include the following steps.

[0201] Step S301, obtaining a first sample video frame and a second sample video frame in a sample video; a sample video scene corresponding to the first sample video frame is the same as a sample video scene corresponding to the second sample video frame.

[0202] Specifically, the computer device obtains an initial model for video frame alignment and a sample video, decomposes the sample video into continuous sample video frames, performs video scene detection on multiple continuous sample video frames, aggregates sample video frames belonging to the same sample video scene, and extracts a first sample video frame and a second sample video frame in each sample video scene.

[0203] The information of the sample video and the specific process of obtaining the sample video frame can be found in the above Figure 3 The description of step S101 in the corresponding embodiment will not be repeated here.

[0204] Step S302, input the first sample video frame and the second sample video frame into the video frame alignment initial model respectively, in which the first initial sample video feature corresponding to the first sample video frame is subjected to feature convolution processing to generate the first sample video feature to be fused corresponding to the first sample video frame.

[0205] Step S303 , performing feature splicing processing on the first initial sample video feature corresponding to the first sample video frame and the second initial sample video feature corresponding to the second sample video frame to generate a first sample video feature to be processed.

[0206] Step S304: perform feature downsampling processing on the first sample video feature to be processed to generate a second sample video feature to be processed.

[0207] Step S305 : generating an aligned estimated video frame for the second sample video frame according to the second sample video feature to be processed and the first sample video feature to be fused.

[0208] The processing process of the sample video frame (including the first sample video frame and the second sample video frame) by the computer device is basically the same as the process of the video frame processing by the computer device. Therefore, the specific process of step S302 to step S305 can be referred to above. Figure 3 The description in the corresponding embodiment will not be repeated here.

[0209] Step S306, obtaining a mean square error value between the second sample video frame and the aligned estimated video frame, and determining the mean square error value as a loss value of the initial model for video frame alignment.

[0210] Step S307, adjusting the model parameters in the initial model of the video frame alignment according to the loss value to generate a video frame alignment model; the video frame alignment model is used to perform inter-frame alignment processing on the video frames in the video.

[0211] In an embodiment of the present application, the computer device uses a mean squared error loss (MSE) function to calculate the error between the second sample video frame and the aligned estimated video frame, and adjusts the model parameters in the video frame alignment initial model according to the above error. It can be understood that when the error is less than a preset model error, the video frame alignment model can be determined. Optionally, the computer device sets a preset number of iterations for the video frame alignment initial model and records the number of training iterations of the initial model. When the number of training iterations is equal to the preset number of iterations, the training of the initial model is stopped, and the model with the number of training iterations equal to the preset number of iterations is determined as the video frame alignment model.

[0212] Optionally, in addition to the above-mentioned mean squared error loss (MSE) function, the computer device may also adopt other error functions, such as a square loss function and a cross entropy loss function.

[0213] In summary, the embodiment of the present application proposes a fast video frame alignment algorithm that achieves considerable performance with low model complexity. The specific implementation is as follows: the first sample video feature to be processed, including the first initial sample video feature and the second initial sample video feature, is subjected to feature downsampling processing, which can reduce the inter-frame motion difference between the first sample video frame and the second sample video frame, and achieve high-accuracy recognition of aligned estimated optical flow data. Therefore, the use of this application can improve the recognition accuracy of aligned estimated optical flow data, and then generate highly accurate aligned estimated video frames. Through the error between the highly accurate aligned estimated video frame and the second sample video frame, the initial model of video frame alignment can be accurately adjusted, and the video frame alignment model can be efficiently generated.

[0214] For further information, see Fig.10 , Fig.10 : is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. The above-mentioned data processing device may be a computer program (including program code) running in a computer device, for example, the data processing device is an application software; the device may be used to execute the corresponding steps in the method provided in an embodiment of the present application. Fig.10 As shown, the data processing device 1 may include: a first acquisition module 11 , a first generation module 12 , a second generation module 13 , a third generation module 14 and a fourth generation module 15 .

[0215] A first acquisition module 11 is used to acquire a first video frame and a second video frame in a video; the video scene corresponding to the first video frame is the same as the video scene corresponding to the second video frame;

[0216] A first generating module 12 is used to perform feature convolution processing on the first initial video feature corresponding to the first video frame to generate a first to-be-fused video feature corresponding to the first video frame;

[0217] The second generating module 13 is used to perform feature splicing processing on the first initial video feature corresponding to the first video frame and the second initial video feature corresponding to the second video frame to generate a first video feature to be processed;

[0218] The third generating module 14 is used to perform feature downsampling processing on the first video feature to be processed to generate a second video feature to be processed;

[0219] The fourth generating module 15 is used to generate an aligned video frame for the second video frame according to the second video feature to be processed and the first video feature to be fused.

[0220] The specific functional implementation of the first acquisition module 11, the first generation module 12, the second generation module 13, the third generation module 14 and the fourth generation module 15 can be referred to above. Figure 3 Steps S101 to S105 in the corresponding embodiment are not described in detail here.

[0221] See also Fig.10 The third generating module 14 may include: a model acquiring unit 141 , a first processing unit 142 , a second processing unit 143 , a first determining unit 144 and a second determining unit 145 .

[0222] The acquisition model unit 141 is used to acquire a video frame alignment model; the video frame alignment model includes A feature downsampling layers, and the A feature downsampling layers include a feature downsampling layer B c And feature downsampling layer B c+1 , A is a positive integer, c is a positive integer and c is less than A;

[0223] The first processing unit 142 is used to perform feature downsampling at the layer B. c In the process, the first input feature is downsampled to generate the first transition video feature D c ; If the feature downsampling layer B c is the first feature downsampling layer among A feature downsampling layers, then the first input feature is the first video feature to be processed; if feature downsampling layer B c If it is not the first feature downsampling layer among A feature downsampling layers, then the first input feature is, feature downsampling layer B c The first transition video feature output by the previous feature downsampling layer;

[0224] The second processing unit 143 is used to convert the first transition video feature D c As feature downsampling layer Bc+1 The first input feature of the feature downsampling layer B c+1 In the first transition video feature D c Perform feature downsampling to generate the first transition video feature D c+1 ;

[0225] The first determining unit 144 is used to determine if the feature downsampling layer B c+1 If it is not the last feature downsampling layer among the A feature downsampling layers, the first transition video feature D c+1 Determined as, feature downsampling layer B c+1 The first input feature of the next feature downsampling layer;

[0226] The second determining unit 145 is used to determine if the feature downsampling layer B c+1 is the last feature downsampling layer among the A feature downsampling layers, then the first transition video feature D c+1 Determined as the second video feature to be processed.

[0227] The specific functional implementation of the acquisition model unit 141, the first processing unit 142, the second processing unit 143, the first determination unit 144 and the second determination unit 145 can be referred to above. Figure 3 The step S104 in the corresponding embodiment will not be described in detail here.

[0228] See also Fig.10 , feature downsampling layer B c Includes a first convolution component and a feature pooling component;

[0229] The first processing unit 142 may include: a first processing sub-unit 1421 and a second processing sub-unit 1422 .

[0230] The first processing subunit 1421 is used to input the first input feature into the first convolution component, and perform feature convolution processing on the first input feature in the first convolution component to generate a video feature to be pooled;

[0231] The first processing subunit 1421 is further used to input the video features to be pooled into the feature pooling component; the feature pooling component carries the pooling size;

[0232] The second processing subunit 1422 is used to perform feature downsampling processing on the video feature to be pooled in the feature pooling component according to the pooling size to generate the first transition video feature D c ; The feature resolution corresponding to the first input feature is equal to the pooling size and the first transition video feature D c The product of the corresponding feature resolution.

[0233] The specific functional implementation of the first processing subunit 1421 and the second processing subunit 1422 can be referred to above. Figure 3 The step S104 in the corresponding embodiment will not be described in detail here.

[0234] See also Fig.10 , the first convolution component includes the first convolution component G h , and the first convolutional component G h The associated activation component, the first convolution component G h+1 , and the first convolutional component G h+1 The associated fusion component and the first convolution component G h+1 The associated activation component; h is a positive integer, and h is less than the total number of the first convolutional components;

[0235] The first processing subunit 1421 may include a first input subunit 14211 , a second input subunit 14212 , a third input subunit 14213 , a fourth input subunit 14214 and a fifth input subunit 14215 .

[0236] The first input subunit 14211 is used to input the first input feature into the first convolution component G h ; The first convolutional component G h Includes E convolution channels, where E is a positive integer;

[0237] The first input subunit 14211 is also used to pass the first convolution component G h The E convolution channels in the first input feature are processed by feature convolution to generate the first video feature I to be activated. h ; The first video feature to be activated I h The corresponding number of feature channels is equal to E;

[0238] The second input subunit 14212 is used to input the first video feature I to be activated h Input to the first convolution component G h The associated activation component is connected to the first convolution component G h The associated activation component, for the first video feature I to be activated h Perform feature nonlinear processing to generate the first convolution video feature J h ;

[0239] The third input subunit 14213 is used to convert the first convolutional video feature J h Input to the first convolutional component G h+1 ; The first convolutional component G h+1 Includes E convolution channels;

[0240] The third input subunit 14213 is also used to pass through the first convolution component G h+1 The E convolution channels in the first convolution video feature J h Perform feature convolution processing to generate the first video feature I to be activated h+1 ; The first video feature to be activated I h+1 The corresponding number of feature channels is equal to E;

[0241] The fourth input subunit 14214 is used to input the first video feature I to be activated h+1 And the first convolutional video feature J h Input to the first convolution component G h+1 The associated fusion component, in conjunction with the first convolutional component G h+1 In the associated fusion component, the first video feature I to be activated h+1 And the first convolutional video feature J h Perform feature fusion processing to generate the first video feature I to be activated h+2 ;

[0242] The fifth input subunit 14215 is used to input the first video feature I to be activated h+2 Input to the first convolution component G h+1 The associated activation component is connected to the first convolution component G h+1 The associated activation component, for the first video feature I to be activated h+2 Perform nonlinear feature processing to generate video features to be pooled.

[0243] The specific functional implementation of the first input subunit 14211, the second input subunit 14212, the third input subunit 14213, the fourth input subunit 14214 and the fifth input subunit 14215 can be referred to above. Figure 3 The step S104 in the corresponding embodiment will not be described in detail here.

[0244] See also Fig.10 ,The video frame alignment model also includes A feature upsampling layers, the first feature convolution layer and a feature fusion layer;

[0245] The fourth generating module 15 may include: a third processing unit 151 , a fourth processing unit 152 , a fifth processing unit 153 and a sixth processing unit 154 .

[0246] The third processing unit 151 is used to perform feature upsampling processing on the second to-be-processed video feature through A feature upsampling layers to generate a third to-be-processed video feature, and input the third to-be-processed video feature to the first feature convolution layer; the feature resolution corresponding to the third to-be-processed video feature is equal to the feature resolution corresponding to the first to-be-processed video feature;

[0247] The fourth processing unit 152 is used to perform feature convolution processing on the third video feature to be processed through the first feature convolution layer to generate a second video feature to be fused; the number of feature channels corresponding to the second video feature to be fused is equal to the number of feature channels corresponding to the first video feature to be fused;

[0248] The fifth processing unit 153 is used to input the first video feature to be fused and the second video feature to be fused into a feature fusion layer, and perform feature fusion processing on the first video feature to be fused and the second video feature to be fused in the feature fusion layer to generate a normalized video feature for the second video frame;

[0249] The sixth processing unit 154 is used to perform feature denormalization processing on the normalized video features to generate aligned video frames.

[0250] The specific functional implementation of the third processing unit 151, the fourth processing unit 152, the fifth processing unit 153 and the sixth processing unit 154 can be found in the above Figure 3 The step S105 in the corresponding embodiment will not be described in detail here.

[0251] See also Fig.10 , A feature upsampling layers include feature upsampling layer K m And the feature upsampling layer K m+1 , m is a positive integer and m is less than A;

[0252] The third processing unit 151 may include: a third processing subunit 1511 , a fourth processing subunit 1512 , a first determining subunit 1513 , and a second determining subunit 1514 .

[0253] The third processing subunit 1511 is used to sample the feature layer K m In the process, the second input feature is subjected to feature upsampling to generate the second transition video feature L m ; If the feature upsampling layer K m is the first feature upsampling layer among the A feature upsampling layers, then the second input feature is the second video feature to be processed; if the feature upsampling layer K m If it is not the first feature upsampling layer among the A feature upsampling layers, then the second input feature is, feature upsampling layer K m The second transition video feature output by the previous feature upsampling layer;

[0254] The fourth processing subunit 1512 is used to convert the second transition video feature L m As the feature upsampling layer K m+1 The second input feature of m+1In the second transition video feature L m Perform feature upsampling to generate the second transition video feature L m+1 ;

[0255] The first determining subunit 1513 is used to determine if the feature upsampling layer K m+1 If it is not the last feature upsampling layer among the A feature upsampling layers, the second transition video feature L m+1 Determined as, the feature upsampling layer K m+1 The second input feature of the next feature upsampling layer;

[0256] The second determining subunit 1514 is used to determine if the feature upsampling layer K m+1 is the last feature upsampling layer in the A feature upsampling layers, then the second transition video feature L m+1 Determined as the third video feature to be processed.

[0257] The specific functional implementation of the third processing subunit 1511, the fourth processing subunit 1512, the first determining subunit 1513 and the second determining subunit 1514 can refer to the above Figure 3 The step S105 in the corresponding embodiment will not be described in detail here.

[0258] See also Fig.10 , feature upsampling layer K m including a second convolution component and a feature upsampling component;

[0259] The third processing subunit 1511 may include: a sixth input subunit 15111 and a seventh input subunit 15112 .

[0260] The sixth input subunit 15111 is used to input the second input feature into the second convolution component, and perform feature convolution processing on the second input feature in the second convolution component to generate a video feature to be upsampled;

[0261] The seventh input subunit 15112 is used to input the video features to be upsampled into the feature upsampling component; the feature upsampling component carries the upsampling size;

[0262] The seventh input subunit 15112 is further configured to perform feature upsampling processing on the to-be-upsampled video feature in the feature upsampling component according to the upsampling size, and generate a second transition video feature L m ; The second transition video feature L m The corresponding feature resolution is equal to the product of the feature resolution corresponding to the second input feature and the upsampling size.

[0263] The specific functional implementation of the sixth input subunit 15111 and the seventh input subunit 15112 can refer to the above Figure 3 The step S105 in the corresponding embodiment will not be described in detail here.

[0264] See also Fig.10 , the second convolution component includes a second convolution component N o , and the second convolutional component N o The associated activation component, the second convolution component N o+1 , and the second convolutional component N o+1 The associated fusion component and the second convolution component N o+1 The associated activation component; o is a positive integer, and o is less than the total number of the second convolution components;

[0265] The sixth input subunit 15111 is specifically configured to input the second input feature into the second convolution component N o ; The second convolutional component N o Includes P convolution channels, where P is a positive integer;

[0266] The sixth input subunit 15111 is further specifically configured to pass the second convolution component N o The P convolution channels in the image are used to perform feature convolution processing on the second input feature to generate the second video feature Q to be activated. o ; The second video feature Q to be activated o The corresponding number of feature channels is equal to P;

[0267] The sixth input subunit 15111 is further configured to input the second video feature Q to be activated. o Input to the second convolution component N o The associated activation component is connected to the second convolution component N o The associated activation component is for the second video feature Q to be activated o Perform feature nonlinear processing to generate the second convolution video feature R o ;

[0268] The sixth input subunit 15111 is further specifically configured to convert the second convolutional video feature R o Input to the second convolutional component N o+1 ; The second convolutional component N o+1 Includes P convolution channels;

[0269] The sixth input subunit 15111 is further specifically configured to pass the second convolution component N o+1 The P convolution channels in the second convolution video feature R o Perform feature convolution processing to generate the second video feature Q to be activated o+1; The second video feature Q to be activated o+1 The corresponding number of feature channels is equal to P;

[0270] The sixth input subunit 15111 is further specifically configured to input the second video feature Q to be activated o+1 And the second convolutional video feature R o Input to the second convolution component N o+1 The associated fusion component, in conjunction with the second convolutional component N o+1 In the associated fusion component, the second video feature Q to be activated o+1 And the second convolutional video feature R o Perform feature fusion processing to generate the second video feature Q to be activated o+2 ;

[0271] The sixth input subunit 15111 is further specifically configured to input the second video feature Q to be activated o+2 Input to the second convolution component N o+1 The associated activation component is connected to the second convolution component N o+1 The associated activation component is for the second video feature Q to be activated o+2 Perform feature nonlinear processing to generate video features to be upsampled.

[0272] The specific functional implementation of the sixth input subunit 15111 can refer to the above Figure 3 The step S105 in the corresponding embodiment will not be described in detail here.

[0273] See also Fig.10 , the first feature convolution layer includes a feature convolution component, a feature activation component and a feature normalization component;

[0274] The fourth processing unit 152 may include: a first generating subunit 1521 , a second generating subunit 1522 , and a third generating subunit 1523 .

[0275] The first generating subunit 1521 is used to input the third video feature to be processed into the feature convolution component, and in the feature convolution component, perform feature convolution processing on the third video feature to be processed to generate a fourth video feature to be processed; the number of feature channels corresponding to the fourth video feature to be processed is equal to the number of feature channels corresponding to the first video feature to be fused;

[0276] The second generating subunit 1522 is used to input the fourth to-be-processed video feature into the feature activation component, and in the feature activation component, perform feature nonlinear processing on the fourth to-be-processed video feature to generate a fifth to-be-processed video feature;

[0277] The third generating subunit 1523 is used to input the fifth video feature to be processed into the feature normalization component, and perform a feature normalization operation on the fifth video feature to be processed in the feature normalization component to generate a second video feature to be fused.

[0278] The specific functional implementation of the first generation subunit 1521, the second generation subunit 1522 and the third generation subunit 1523 can be found in the above Figure 3 The step S105 in the corresponding embodiment will not be described in detail here.

[0279] See also Fig.10 ,The video frame alignment model also includes a feature splicing layer;

[0280] The second generating module 13 may include: a first generating unit 131 and a second generating unit 132 .

[0281] The first generating unit 131 is used to perform normalization processing on the first video frame to generate a first normalized video frame, and perform normalization processing on the second video frame to generate a second normalized video frame; the number of video frame channels corresponding to the first normalized video frame is equal to the number of video frame channels corresponding to the second normalized video frame;

[0282] The first generating unit 131 is further configured to input the first initial video feature corresponding to the first normalized video frame and the second initial video feature corresponding to the second normalized video frame into the feature splicing layer;

[0283] The second generating unit 132 is used to perform feature stitching processing on the first initial video feature corresponding to the first normalized video frame and the second initial video feature corresponding to the second normalized video frame in the feature stitching layer to generate a first video feature to be processed; the number of feature channels corresponding to the first video feature to be processed is equal to twice the number of video frame channels corresponding to the first normalized video frame.

[0284] The specific functional implementation of the first generation unit 131 and the second generation unit 132 can be found in the above Figure 3 The step S103 in the corresponding embodiment will not be described in detail here.

[0285] See also Fig.10 ,The video frame alignment model also includes a second feature convolution layer and a third feature convolution layer;

[0286] The first generating module 12 may include a third generating unit 121 , a fourth generating unit 122 and a fifth generating unit 123 .

[0287] The third generating unit 121 is used to perform normalization processing on the first video frame to generate a first normalized video frame;

[0288] The fourth generating unit 122 is used to input the first initial video feature corresponding to the first normalized video frame into the second feature convolution layer, and in the second feature convolution layer, perform feature dimension increase processing on the first initial video feature corresponding to the first normalized video frame to generate an increased dimension video feature; the number of feature channels corresponding to the increased dimension video feature is greater than the number of video frame channels corresponding to the first normalized video frame;

[0289] The fifth generating unit 123 is used to input the increased-dimensional video features into the third feature convolution layer, and perform feature dimensionality reduction processing on the increased-dimensional video features in the third feature convolution layer to generate the first video features to be fused; the number of feature channels corresponding to the first video features to be fused is equal to the number of video frame channels corresponding to the first normalized video frame.

[0290] The specific functional implementation of the third generation unit 121, the fourth generation unit 122 and the fifth generation unit 123 can be found in the above Figure 3 The step S102 in the corresponding embodiment will not be described in detail here.

[0291] See also Fig.10 The data processing device 1 may further include: a second acquisition module 16 and a fifth generation module 17 .

[0292] A second acquisition module 16 is used to acquire an image super-resolution model for enhancing the clarity of video frames;

[0293] The fifth generation module 17 is used to input the second video frame and the aligned video frame into the image super-resolution model. In the image super-resolution model, based on the aligned video frame, the second video frame is subjected to resolution enhancement processing to generate an enhanced resolution video frame corresponding to the second video frame.

[0294] The specific functional implementation of the second acquisition module 16 and the fifth generation module 17 can be found in the above Figure 8 Steps S206 and S207 in the corresponding embodiment will not be described in detail here.

[0295] Please see again Fig.10 , in the data processing device 1, the inter-frame difference between the aligned video frame and the first video frame is smaller than the inter-frame difference between the second video frame and the first video frame.

[0296] In an embodiment of the present application, a first video frame and a second video frame in the same video scene are obtained to determine that the moving objects in the two video frames remain consistent; a feature convolution process is performed on a first initial video feature corresponding to the first video frame to extract hidden features of the first video frame and generate a first video feature to be fused; further, a feature splicing process is performed on the first initial video feature corresponding to the first video frame and a second initial video feature corresponding to the second video frame to generate a first video feature to be processed including the first initial video feature and the second initial video feature; a feature downsampling process is performed on the first video feature to be processed to reduce the motion difference between the first initial video feature and the second initial video feature and generate a second video feature to be processed; further, an aligned video frame for the second video frame is generated based on the second video feature to be processed and the first video feature to be fused. From the above, it can be seen that the present application can reduce the inter-frame motion difference between the first video frame and the second video frame by performing feature downsampling processing on the first to-be-processed video features including the first initial video features and the second initial video features, thereby achieving high-accuracy recognition of aligned optical flow data. Therefore, the present application can improve the recognition accuracy of aligned optical flow data, and further generate high-accuracy aligned video frames.

[0297] For further information, see Fig.11 , Fig.11 : is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. The above-mentioned data processing device may be a computer program (including program code) running in a computer device, for example, the data processing device is an application software; the device may be used to execute the corresponding steps in the method provided in an embodiment of the present application. Fig.11 As shown, the data processing device 2 may include: a first acquisition module 21 , a first generation module 22 , a second generation module 23 , a third generation module 24 , a fourth generation module 25 , a second acquisition module 26 and a fifth generation module 27 .

[0298] A first acquisition module 21 is used to acquire a first sample video frame and a second sample video frame in a sample video; the sample video scene corresponding to the first sample video frame is the same as the sample video scene corresponding to the second sample video frame;

[0299] A first generating module 22 is used to input the first sample video frame and the second sample video frame into the video frame alignment initial model respectively, and perform feature convolution processing on the first initial sample video feature corresponding to the first sample video frame in the video frame alignment initial model to generate a first sample video feature to be fused corresponding to the first sample video frame;

[0300] The second generating module 23 is used to perform feature splicing processing on the first initial sample video feature corresponding to the first sample video frame and the second initial sample video feature corresponding to the second sample video frame to generate a first sample video feature to be processed;

[0301] The third generating module 24 is used to perform feature downsampling processing on the first sample video feature to be processed to generate a second sample video feature to be processed;

[0302] A fourth generating module 25, configured to generate an alignment estimation video frame for the second sample video frame according to the second sample video feature to be processed and the first sample video feature to be fused;

[0303] A second acquisition module 26 is used to obtain a mean square error value between the second sample video frame and the aligned estimated video frame, and determine the mean square error value as a loss value of the initial model of the video frame alignment;

[0304] The fifth generation module 27 is used to adjust the model parameters in the initial model of the video frame alignment according to the loss value to generate a video frame alignment model; the video frame alignment model is used to perform inter-frame alignment processing on the video frames in the video.

[0305] The specific functional implementation of the first acquisition module 21, the first generation module 22, the second generation module 23, the third generation module 24, the fourth generation module 25, the second acquisition module 26 and the fifth generation module 27 can be referred to above. Fig. 9 Steps S301 to S307 in the corresponding embodiment are not described in detail here.

[0306] In summary, the embodiment of the present application proposes a fast video frame alignment algorithm that achieves considerable performance with low model complexity. The specific implementation is as follows: the first sample video feature to be processed, including the first initial sample video feature and the second initial sample video feature, is subjected to feature downsampling processing, which can reduce the inter-frame motion difference between the first sample video frame and the second sample video frame, and achieve high-accuracy recognition of aligned estimated optical flow data. Therefore, the use of this application can improve the recognition accuracy of aligned estimated optical flow data, and then generate highly accurate aligned estimated video frames. Through the error between the highly accurate aligned estimated video frame and the second sample video frame, the initial model of video frame alignment can be accurately adjusted, and the video frame alignment model can be efficiently generated.

[0307] For further information, see Fig.12 , Fig.12 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.12As shown, the computer device 1000 may include: at least one processor 1001, such as a CPU, at least one network interface 1004, a user interface 1003, a memory 1005, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display) and a keyboard (Keyboard), and the network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 1005 may optionally also be at least one storage device located away from the aforementioned processor 1001. As shown in FIG. Fig.12 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a device control application.

[0308] exist Fig.12 In the computer device 1000 shown, the network interface 1004 can provide a network communication function; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0309] Acquire a first video frame and a second video frame in a video; the video scene corresponding to the first video frame is the same as the video scene corresponding to the second video frame;

[0310] Performing feature convolution processing on a first initial video feature corresponding to the first video frame to generate a first to-be-fused video feature corresponding to the first video frame;

[0311] Performing feature splicing processing on a first initial video feature corresponding to the first video frame and a second initial video feature corresponding to the second video frame to generate a first video feature to be processed;

[0312] Performing feature downsampling processing on the first video feature to be processed to generate a second video feature to be processed;

[0313] An aligned video frame for the second video frame is generated according to the second video feature to be processed and the first video feature to be fused.

[0314] It should be understood that the computer device 1000 described in the embodiment of the present application can execute the above Figure 3 , Figure 8 as well as Fig. 9 The description of the data processing method in the corresponding embodiment can also be performed as described above. Fig.10The description of the data processing device 1 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.

[0315] For further information, see Fig.13 , Fig.13 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.13 As shown, the above-mentioned computer device 2000 may include: a processor 2001, a network interface 2004 and a memory 2005. In addition, the above-mentioned computer device 2000 may also include: a user interface 2003, and at least one communication bus 2002. Among them, the communication bus 2002 is used to realize the connection and communication between these components. Among them, the user interface 2003 may include a display screen (Display), a keyboard (Keyboard), and the optional user interface 2003 may also include a standard wired interface and a wireless interface. The network interface 2004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 2005 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 2005 may optionally also be at least one storage device located away from the aforementioned processor 2001. As Fig.13 As shown, the memory 2005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0316] exist Fig.13 In the computer device 2000 shown, the network interface 2004 can provide a network communication function; the user interface 2003 is mainly used to provide an input interface for the user; and the processor 2001 can be used to call the device control application stored in the memory 2005 to achieve:

[0317] Acquire a first sample video frame and a second sample video frame in the sample video; the sample video scene corresponding to the first sample video frame is the same as the sample video scene corresponding to the second sample video frame;

[0318] The first sample video frame and the second sample video frame are respectively input into the video frame alignment initial model, and in the video frame alignment initial model, a feature convolution process is performed on the first initial sample video feature corresponding to the first sample video frame to generate a first sample video feature to be fused corresponding to the first sample video frame;

[0319] Performing feature splicing processing on a first initial sample video feature corresponding to the first sample video frame and a second initial sample video feature corresponding to the second sample video frame to generate a first sample video feature to be processed;

[0320] Performing feature downsampling processing on the first sample video feature to be processed to generate a second sample video feature to be processed;

[0321] Generate an aligned estimated video frame for the second sample video frame according to the second sample video feature to be processed and the first sample video feature to be fused;

[0322] Obtaining a mean square error value between the second sample video frame and the aligned estimated video frame, and determining the mean square error value as a loss value of the initial model for video frame alignment;

[0323] The model parameters in the initial model of the video frame alignment are adjusted according to the loss value to generate a video frame alignment model; the video frame alignment model is used to perform inter-frame alignment processing on video frames in the video.

[0324] It should be understood that the computer device 2000 described in the embodiment of the present application can execute the above Figure 3 , Figure 8 as well as Fig. 9 The description of the data processing method in the corresponding embodiment can also be performed as described above. Fig.11 The description of the data processing device 2 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.

[0325] The present application also provides a computer-readable storage medium storing a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, Figure 3 , Figure 8 as well as Fig. 9 For details on the data processing methods provided in each step, please refer to the above Figure 3 , Figure 8 as well as Fig. 9 The implementation methods provided by each step will not be described in detail here. In addition, the description of the beneficial effects of adopting the same method will not be described in detail here either.

[0326] The computer-readable storage medium may be the data processing device provided in any of the aforementioned embodiments or the internal storage unit of the computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the computer device. Further, the computer-readable storage medium may also include both the internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0327] The embodiment of the present application also provides a computer program product or a computer program, the computer program product or the computer program includes a computer instruction, the computer instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device can execute the above Figure 3 , Figure 8 as well as Fig. 9 The description of the data processing method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated here.

[0328] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but optionally includes steps or modules that are not listed, or optionally includes other step units inherent to these processes, methods, devices, products, or equipment.

[0329] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0330] The method and related apparatus provided by the embodiment of the present application are described with reference to the method flow chart and / or structural diagram provided by the embodiment of the present application. Specifically, each process and / or box in the method flow chart and / or structural diagram, as well as the combination of the processes and / or boxes in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A flow or multiple flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0331] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A data processing method, characterized in that: include: Acquire a first video frame and a second video frame in a video; the video scene corresponding to the first video frame is the same as the video scene corresponding to the second video frame; Performing feature convolution processing on a first initial video feature corresponding to the first video frame to generate a first to-be-fused video feature corresponding to the first video frame; Performing feature splicing processing on a first initial video feature corresponding to the first video frame and a second initial video feature corresponding to the second video frame to generate a first video feature to be processed; Performing feature downsampling processing on the first to-be-processed video feature to generate a second to-be-processed video feature; An aligned video frame for the second video frame is generated according to the second video feature to be processed and the first video feature to be fused.

2. The method according to claim 1, characterized in that The performing feature downsampling processing on the first to-be-processed video feature to generate a second to-be-processed video feature includes: Obtain a video frame alignment model; the video frame alignment model includes A feature downsampling layers, the A feature downsampling layers include a feature downsampling layer B c And feature downsampling layer B c+1 , A is a positive integer, c is a positive integer and c is less than A; In the feature downsampling layer B c In the process, the first input feature is downsampled to generate the first transition video feature D c ; If the feature downsampling layer B c is the first feature downsampling layer in the A feature downsampling layers, then the first input feature is the first to-be-processed video feature; if the feature downsampling layer B c is not the first feature downsampling layer among the A feature downsampling layers, then the first input feature is, the feature downsampling layer B c The first transition video feature output by the previous feature downsampling layer; The first transition video feature D c As the feature downsampling layer B c+1 The first input feature, in the feature downsampling layer B c+1 In the example, the first transition video feature D c Perform feature downsampling to generate the first transition video feature D c+1 ; If the feature downsampling layer B c+1 is not the last feature downsampling layer among the A feature downsampling layers, then the first transition video feature D c+1 Determined as, the feature downsampling layer B c+1 The first input feature of the next feature downsampling layer; If the feature downsampling layer B c+1 is the last feature downsampling layer in the A feature downsampling layers, then the first transition video feature D c+1 Determined as the second video feature to be processed.

3. The method according to claim 2, characterized in that The feature downsampling layer B c Includes a first convolution component and a feature pooling component; The feature downsampling layer B c In the process, the first input feature is downsampled to generate the first transition video feature D c ,include: Inputting the first input feature into the first convolution component, and performing feature convolution processing on the first input feature in the first convolution component to generate a video feature to be pooled; Inputting the video features to be pooled into the feature pooling component; the feature pooling component carries the pooling size; According to the pooling size, the video feature to be pooled is subjected to feature downsampling in the feature pooling component to generate the first transition video feature D c ; The feature resolution corresponding to the first input feature is equal to the pooling size and the first transition video feature D c The product of the corresponding feature resolution.

4. The method according to claim 3, characterized in that The first convolution component includes a first convolution component G h , and the first convolution component G h The associated activation component, the first convolution component G h+1 , and the first convolution component G h+1 The associated fusion component and the first convolution component G h+1 The associated activation component; h is a positive integer, and h is less than the total number of the first convolution components; The step of inputting the first input feature into the first convolution component, and performing feature convolution processing on the first input feature in the first convolution component to generate a video feature to be pooled, includes: Input the first input feature to the first convolution component G h ; The first convolution component G h Includes E convolution channels, where E is a positive integer; Through the first convolution component G h The E convolution channels in the first input feature are used to perform feature convolution processing to generate a first video feature I to be activated. h ; The first video feature to be activated I h The corresponding number of feature channels is equal to E; The first video feature to be activated I h Input to the first convolution component G h The activation component associated with it is connected to the first convolution component G h The associated activation component activates the first video feature I to be activated h Perform feature nonlinear processing to generate the first convolution video feature J h ; The first convolutional video feature J h Input to the first convolutional component G h+1 ; The first convolution component G h+1 Includes E convolution channels; Through the first convolution component G h+1 The E convolution channels in the first convolution video feature J h Perform feature convolution processing to generate the first video feature I to be activated h+1 ; The first video feature to be activated I h+1 The corresponding number of feature channels is equal to E; The first video feature to be activated I h+1 And the first convolutional video feature J h Input to the first convolution component G h+1 The associated fusion component, in conjunction with the first convolutional component G h+1 In the associated fusion component, the first video feature I to be activated h+1 And the first convolutional video feature J h Perform feature fusion processing to generate the first video feature I to be activated h+2 ; The first video feature to be activated I h+2 Input to the first convolution component G h+1 The activation component associated with it is connected to the first convolution component G h+1 The associated activation component activates the first video feature I to be activated h+2 Perform feature nonlinear processing to generate the video features to be pooled.

5. The method according to claim 2, characterized in that: The video frame alignment model also includes A feature upsampling layers, a first feature convolution layer and a feature fusion layer; The step of generating an aligned video frame for the second video frame according to the second video feature to be processed and the first video feature to be fused includes: Performing feature upsampling processing on the second to-be-processed video feature through the A feature upsampling layers to generate a third to-be-processed video feature, and inputting the third to-be-processed video feature into the first feature convolution layer; the feature resolution corresponding to the third to-be-processed video feature is equal to the feature resolution corresponding to the first to-be-processed video feature; Through the first feature convolution layer, feature convolution processing is performed on the third video feature to be processed to generate a second video feature to be fused; the number of feature channels corresponding to the second video feature to be fused is equal to the number of feature channels corresponding to the first video feature to be fused; Inputting the first video feature to be fused and the second video feature to be fused into the feature fusion layer, and performing feature fusion processing on the first video feature to be fused and the second video feature to be fused in the feature fusion layer to generate a normalized video feature for the second video frame; Perform feature denormalization processing on the normalized video features to generate the aligned video frame.

6. The method according to claim 5, characterized in that The A feature upsampling layers include a feature upsampling layer K m And the feature upsampling layer K m+1 , m is a positive integer and m is less than A; The step of performing feature upsampling processing on the second to-be-processed video feature through the A feature upsampling layers to generate a third to-be-processed video feature includes: Sampling layer K on the features m In the process, the second input feature is subjected to feature upsampling to generate the second transition video feature L m ; If the feature upsampling layer K m is the first feature upsampling layer in the A feature upsampling layers, then the second input feature is the second video feature to be processed; if the feature upsampling layer K m is not the first feature upsampling layer among the A feature upsampling layers, then the second input feature is, m The second transition video feature output by the previous feature upsampling layer; The second transition video feature L m As the feature upsampling layer K m+1 The second input feature is sampled at layer K on the feature m+1 , the second transition video feature L m Perform feature upsampling to generate the second transition video feature L m+1 ; If the feature upsampling layer K m+1 is not the last feature upsampling layer in the A feature upsampling layers, then the second transition video feature L m+1 Determined as, the feature upsampling layer K m+1 The second input feature of the next feature upsampling layer; If the feature upsampling layer K m+1 is the last feature upsampling layer in the A feature upsampling layers, then the second transition video feature L m+1 Determined as the third video feature to be processed.

7. The method according to claim 6, characterized in that The feature upsampling layer K m including a second convolution component and a feature upsampling component; The K layer is used to sample the features m In the process, the second input feature is subjected to feature upsampling to generate the second transition video feature L m ,include: Inputting the second input feature into the second convolution component, and performing feature convolution processing on the second input feature in the second convolution component to generate a video feature to be upsampled; Inputting the video features to be upsampled into the feature upsampling component; the feature upsampling component carries the upsampling size; According to the upsampling size, the feature upsampling processing is performed on the to-be-upsampled video feature in the feature upsampling component to generate the second transition video feature L m The second transition video feature L m The corresponding feature resolution is equal to the product of the feature resolution corresponding to the second input feature and the upsampling size.

8. The method according to claim 7, characterized in that The second convolution component includes a second convolution component N o , and the second convolution component N o The associated activation component, the second convolution component N o+1 , and the second convolution component N o+1 The associated fusion component and the second convolution component N o+1 The associated activation component; o is a positive integer, and o is less than the total number of the second convolution components; The step of inputting the second input feature into the second convolution component, and performing feature convolution processing on the second input feature in the second convolution component to generate a video feature to be upsampled, includes: Input the second input feature to the second convolution component N o ; The second convolution component N o Includes P convolution channels, where P is a positive integer; Through the second convolution component N o The P convolution channels in the image are used to perform feature convolution processing on the second input feature to generate a second video feature Q to be activated. o The second video feature Q to be activated o The corresponding number of feature channels is equal to P; The second to-be-activated video feature Q o Input to the second convolution component N o The activation component associated with the second convolution component N o The associated activation component activates the second video feature Q to be activated o Perform feature nonlinear processing to generate the second convolution video feature R o ; The second convolutional video feature R o Input to the second convolutional component N o+1 ; The second convolution component N o+1 Includes P convolution channels; Through the second convolution component N o+1 The P convolution channels in the second convolution video feature R o Perform feature convolution processing to generate the second video feature Q to be activated o+1 The second video feature Q to be activated o+1 The corresponding number of feature channels is equal to P; The second to-be-activated video feature Q o+1 And the second convolutional video feature R o Input to the second convolution component N o+1 The associated fusion component, in conjunction with the second convolutional component N o+1 In the associated fusion component, the second to-be-activated video feature Q o+1 And the second convolutional video feature R o Perform feature fusion processing to generate the second video feature Q to be activated o+2 ; The second to-be-activated video feature Q o+2 Input to the second convolution component N o+1 The activation component associated with the second convolution component N o+1 The associated activation component activates the second video feature Q to be activated o+2 Perform feature nonlinear processing to generate the video features to be upsampled.

9. The method according to claim 5, characterized in that The first feature convolution layer includes a feature convolution component, a feature activation component and a feature normalization component; The step of performing feature convolution processing on the third video feature to be processed through the first feature convolution layer to generate a second video feature to be fused includes: Inputting the third video feature to be processed into the feature convolution component, performing feature convolution processing on the third video feature to be processed in the feature convolution component to generate a fourth video feature to be processed; the number of feature channels corresponding to the fourth video feature to be processed is equal to the number of feature channels corresponding to the first video feature to be fused; Inputting the fourth to-be-processed video feature into the feature activation component, and in the feature activation component, performing feature nonlinear processing on the fourth to-be-processed video feature to generate a fifth to-be-processed video feature; The fifth video feature to be processed is input into the feature normalization component, and in the feature normalization component, a feature normalization operation is performed on the fifth video feature to be processed to generate the second video feature to be fused.

10. The method according to claim 2, characterized in that The video frame alignment model also includes a feature splicing layer; The step of performing feature splicing processing on the first initial video feature corresponding to the first video frame and the second initial video feature corresponding to the second video frame to generate a first video feature to be processed includes: Normalizing the first video frame to generate a first normalized video frame, and normalizing the second video frame to generate a second normalized video frame; the number of video frame channels corresponding to the first normalized video frame is equal to the number of video frame channels corresponding to the second normalized video frame; Inputting the first initial video feature corresponding to the first normalized video frame and the second initial video feature corresponding to the second normalized video frame into the feature splicing layer; In the feature stitching layer, feature stitching processing is performed on the first initial video feature corresponding to the first normalized video frame and the second initial video feature corresponding to the second normalized video frame to generate the first video feature to be processed; the number of feature channels corresponding to the first video feature to be processed is equal to twice the number of video frame channels corresponding to the first normalized video frame.

11. The method according to claim 2, characterized in that The video frame alignment model also includes a second feature convolution layer and a third feature convolution layer; The performing feature convolution processing on the first initial video feature corresponding to the first video frame to generate the first to-be-fused video feature corresponding to the first video frame includes: Normalizing the first video frame to generate a first normalized video frame; Inputting the first initial video feature corresponding to the first normalized video frame into the second feature convolution layer, and performing feature dimension increase processing on the first initial video feature corresponding to the first normalized video frame in the second feature convolution layer to generate an increased-dimensional video feature; the number of feature channels corresponding to the increased-dimensional video feature is greater than the number of video frame channels corresponding to the first normalized video frame; The increased-dimensional video features are input into the third feature convolution layer, and in the third feature convolution layer, feature dimensionality reduction processing is performed on the increased-dimensional video features to generate the first video features to be fused; the number of feature channels corresponding to the first video features to be fused is equal to the number of video frame channels corresponding to the first normalized video frame.

12. The method according to any one of claims 1 to 11, characterized in that: The method further comprises: Obtain an image super-resolution model for enhancing video frame clarity; The second video frame and the aligned video frame are input into the image super-resolution model. In the image super-resolution model, the second video frame is subjected to resolution enhancement processing based on the aligned video frame to generate an enhanced resolution video frame corresponding to the second video frame.

13. The method according to any one of claims 1 to 11, characterized in that: An inter-frame difference between the aligned video frame and the first video frame is smaller than an inter-frame difference between the second video frame and the first video frame.

14. A data processing method, characterized in that: include: Acquire a first sample video frame and a second sample video frame in a sample video; the sample video scene corresponding to the first sample video frame is the same as the sample video scene corresponding to the second sample video frame; Inputting the first sample video frame and the second sample video frame into a video frame alignment initial model respectively, performing feature convolution processing on a first initial sample video feature corresponding to the first sample video frame in the video frame alignment initial model, and generating a first sample video feature to be fused corresponding to the first sample video frame; Performing feature splicing processing on a first initial sample video feature corresponding to the first sample video frame and a second initial sample video feature corresponding to the second sample video frame to generate a first sample video feature to be processed; Performing feature downsampling processing on the first sample video feature to be processed to generate a second sample video feature to be processed; Generate an aligned estimated video frame for the second sample video frame according to the second sample video feature to be processed and the first sample video feature to be fused; Obtaining a mean square error value between the second sample video frame and the aligned estimated video frame, and determining the mean square error value as a loss value of the initial model for video frame alignment; The model parameters in the initial model of the video frame alignment are adjusted according to the loss value to generate a video frame alignment model; the video frame alignment model is used to perform inter-frame alignment processing on video frames in the video.

15. A computer device, characterized in that: include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide a data communication function, the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method described in any one of claims 1 to 14.

Images