Video processing method and device, computer device and storage medium

Abstract

This application discloses a video processing method, apparatus, computer device, and storage medium, applicable to scenarios such as computer vision, cloud technology, intelligent transportation, and assisted driving. The method includes: acquiring target video data to be processed; calling an encoder of a video processing model to process the target video data to obtain feature representation information of the target video data; the encoder is pre-trained based on multiple frames of images included in the sample video data and a preset occlusion strategy; the preset occlusion strategy includes adjacent images having the same occlusion rules and an occlusion ratio greater than or equal to a preset ratio; and determining the processing result of the target video data based on the video processing model and the feature representation information of the target video data. By using the embodiment of this application and the encoder trained based on the preset occlusion strategy, the feature representation information of the video data can be extracted accurately and efficiently, thereby accurately determining the processing result of the video data.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a video processing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] Currently, self-supervised representation learning methods for images / videos mainly fall into two categories: generative-based self-supervised pre-training frameworks and contrastive-based self-supervised pre-training methods. Generative-based self-supervised pre-training frameworks typically first use an encoder to map the input image to a high-dimensional space, then use a decoder to regenerate the original image from the features in the high-dimensional space. A better encoder is learned by matching the input and generated images. However, this method is limited to research on image data pre-training methods and lacks exploration of video data pre-training methods. Contrastive-based self-supervised pre-training frameworks mainly learn the self-supervised task by comparing feature distances in the high-dimensional space. Most of these frameworks require two parallel branches to perform different data augmentations, which further increases the computational cost of pre-training.

[0003] In existing (supervised / self-supervised) pre-training methods for Video Transformer models, the Transformer model heavily relies on large-scale image datasets of millions or tens of millions of images for pre-training, followed by fine-tuning on video data. This approach is computationally expensive, resulting in low efficiency. Summary of the Invention

[0004] This application provides a video processing method, apparatus, computer device, and storage medium that can accurately and efficiently extract feature representation information from video data, thereby accurately determining the processing result of the video data.

[0005] In a first aspect, embodiments of this application provide a video processing method, the method comprising:

[0006] Acquire the target video data to be processed;

[0007] The encoder of the video processing model is invoked to process the target video data to obtain the feature representation information of the target video data; the encoder is pre-trained based on multiple frames of images included in the sample video data and a preset occlusion strategy; the preset occlusion strategy includes that the occlusion rules of adjacent images are the same and the occlusion ratio is greater than or equal to a preset ratio.

[0008] Based on the video processing model and the feature representation information of the target video data, the processing result of the target video data is determined.

[0009] Secondly, embodiments of this application provide a video processing apparatus, the apparatus comprising:

[0010] The acquisition module is used to acquire the target video data to be processed;

[0011] The processing module is used to call the encoder of the video processing model to process the target video data and obtain the feature representation information of the target video data; the encoder is pre-trained based on multiple frames of images included in the sample video data and a preset occlusion strategy; the preset occlusion strategy includes that the occlusion rules of adjacent images are the same and the occlusion ratio is greater than or equal to a preset ratio.

[0012] The determination module is used to determine the processing result of the target video data based on the video processing model and the feature representation information of the target video data.

[0013] Thirdly, embodiments of this application provide a computer device, which includes a processor, a communication interface, and a memory. The processor, communication interface, and memory are interconnected. The memory stores a computer program, and the processor is used to call the computer program to execute the video processing method provided in embodiments of this application.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the video processing method provided in embodiments of this application.

[0015] Fifthly, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the video processing method provided in embodiments of this application.

[0016] In this embodiment, a computer device acquires target video data to be processed; it calls the encoder of a video processing model to process the target video data, obtaining feature representation information of the target video data; the encoder is pre-trained based on multiple frames of images included in the sample video data and a preset occlusion strategy; the preset occlusion strategy includes adjacent images having the same occlusion rules and an occlusion ratio greater than or equal to a preset ratio; based on the video processing model and the feature representation information of the target video data, the processing result of the target video data is determined. By using the encoder trained based on the preset occlusion strategy in this embodiment, the feature representation information of the video data can be extracted accurately and efficiently, thereby accurately determining the processing result of the video data. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of a video processing scheme provided in an embodiment of this application;

[0019] Figure 2 This is a flowchart illustrating a video processing method provided in an embodiment of this application;

[0020] Figure 3 This is a flowchart illustrating a training method for a video representation learning model provided in an embodiment of this application;

[0021] Figure 4 This is a schematic diagram of the training framework of a video representation learning model provided in an embodiment of this application;

[0022] Figure 5 This is a schematic diagram of the structure of a video representation learning model provided in an embodiment of this application;

[0023] Figure 6a This is a schematic diagram of the structure of another video representation learning model provided in the embodiments of this application;

[0024] Figure 6b This is a schematic diagram of the encoder structure in the video representation learning model provided in this application embodiment;

[0025] Figure 6c This is a schematic diagram of the structure of the MLP module in the encoder provided in the embodiments of this application;

[0026] Figure 7 This is a schematic diagram of a video processing device provided in an embodiment of this application;

[0027] Figure 8 This is a schematic diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0029] To facilitate understanding of the embodiments disclosed in this application, some concepts involved in the embodiments of this application will be explained first, including but not limited to the following.

[0030] 1. Encoder

[0031] An encoder is a network that transforms raw input data into high-level features, primarily composed of multiple Transformer layers. The Transformer itself refers to a deep self-attention transformation network, whose extracted features exhibit good spatiotemporal attention properties.

[0032] 2. Representation learning

[0033] Representation learning refers to a set of techniques for learning features, transforming raw data into a form that can be effectively developed by machine learning.

[0034] 3. Decoder

[0035] A decoder is a network that transforms high-level features into the original input data, and it mainly consists of multiple layers of Transformers.

[0036] To facilitate understanding of the embodiments of this application, the video processing method of this application is described below.

[0037] To enhance the realism of virtual humans, this application provides a video processing solution. Please refer to... Figure 1 , Figure 1 This is a schematic diagram of a video processing scheme provided in an embodiment of this application. The following is a description of the process. Figure 1 The general implementation process of the video processing scheme proposed in the embodiments of this application is described below. Figure 1As shown, solid lines connect the training process of the model, and dashed lines connect the process of using the model. Computer device 101 first acquires sample data 102, which includes multiple sample video data. Next, computer device 101 uses the multiple sample video data in sample data 102 and a preset masking strategy to train the initial encoder 103a and decoder 103b of the self-supervised learning video representation learning model 103, obtaining the trained encoder 1031 and decoder 1032. The trained encoder 1031 is then used as the encoder of the video processing model. The preset masking strategy includes that adjacent images have the same masking rules, and the masking ratio is greater than or equal to a preset ratio (which can be understood as an extremely high masking ratio). Then, the computer device acquires the target video data 104 to be processed and calls the encoder 1031 of the video processing model 105 to process the target video data, obtaining the feature representation information of the target video data. Finally, based on the video processing model 105 and the feature representation information of the target video data, the processing result of the target video data is determined.

[0038] Practice has shown that the video processing scheme proposed in this application has the following beneficial effects: ① By using an encoder trained based on a preset masking strategy, the feature representation information of video data can be extracted accurately and efficiently, thereby accurately determining the processing result of the video data. ② It can be widely applied in various different scenarios.

[0039] It should be noted that in specific implementations, the above scheme can be executed by computer devices, which can be terminals or servers. The terminals mentioned here can include, but are not limited to, smartphones, tablets, laptops, desktop computers, smartwatches, smart TVs, and smart in-vehicle terminals. Various client applications (APPs) can run on these terminals, such as video playback clients, social media clients, browser clients, news feed clients, educational clients, and so on. The servers mentioned here can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, etc. Furthermore, the computer device mentioned in the embodiments of this application can be located outside or inside the blockchain network, and there is no limitation on this. The so-called blockchain network is a network composed of a peer-to-peer network (P2P network) and a blockchain. The blockchain refers to a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm. In essence, it is a decentralized database, which is a series of data blocks (or blocks) linked together using cryptographic methods.

[0040] The video processing method provided in this application embodiment can be implemented based on Artificial Intelligence (AI) technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making functions. Artificial intelligence technology is a comprehensive discipline involving a wide range of fields, including both hardware and software technologies. Basic AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, and mechatronics. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0041] The video processing method provided in this application primarily relates to Computer Vision (CV) technology within AI. Computer vision is a science that studies how to enable machines to "see." More specifically, it refers to machine vision that uses cameras and computers to replace human eyes for target recognition, tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision researches related theories and technologies, attempting to establish artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and other technologies. It should be noted that this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0042] Based on the above solution, this application proposes a video processing method. Please refer to [link to relevant documentation]. Figure 2 , Figure 2 This is a schematic flowchart of a video processing method provided in an embodiment of this application. The method can be executed by a computer device, such as... Figure 2 As shown, the video processing method may include the following steps S201-S203:

[0043] S201. Obtain the target video data to be processed.

[0044] Optionally, the computer device can obtain the target video data to be processed from a local database, or obtain the captured video data as the target data to be processed from the shooting device, or obtain the target video data to be processed from a publicly available dataset, etc., without limitation here.

[0045] S202. Call the encoder of the video processing model to process the target video data and obtain the feature representation information of the target video data; the encoder is pre-trained based on the multi-frame images included in the sample video data and the preset masking strategy; the preset masking strategy includes that the masking rules of adjacent images are the same and the masking ratio is greater than or equal to the preset ratio.

[0046] The same occlusion rules for adjacent images means that the positions of the preserved image blocks and the occluded image blocks are consistent for adjacent images.

[0047] In one alternative implementation, the computer device may also acquire training samples, which include multiple sample video data; pre-train the initialized encoder and decoder of the video representation learning model using the multiple sample video data and a preset masking strategy to obtain the trained encoder and decoder; and use the trained encoder as the encoder of the video processing model.

[0048] Optionally, the video representation learning model can be an autoencoder structure based on self-supervised learning, including an encoder and a decoder. Self-supervised learning primarily utilizes an auxiliary task (pretext) to mine its own supervisory information from large-scale unsupervised data. This constructed supervisory information is used to train the network, thereby learning representations valuable for downstream tasks. An autoencoder is an encoder-decoder network structure used in self-supervised learning, mainly composed of an encoder and a decoder. Its function is to perform self-supervised representation learning on the input information by using the input information as the learning target.

[0049] S203. Based on the video processing model and the feature representation information of the target video data, determine the processing result of the target video data.

[0050] In one alternative implementation, the computer device may invoke a classification network of a video processing model to process the feature representation information of the target video data to obtain a classification result of the target video data, which includes one or more of video theme classification and video action classification.

[0051] In one optional implementation, when the classification network of the video processing model is used to classify the themes of video data (e.g., travel, food, etc.), the computer device can process the feature representation information of the target video data through the classification network to obtain the theme category to which the target video data belongs; when the classification network of the video processing model is used to classify the actions of video data (e.g., dance actions, sports actions, etc.), the computer device can process the feature representation information of the target video data through the classification network to obtain the action category to which the target video data belongs.

[0052] In this embodiment, a computer device acquires target video data to be processed; it calls the encoder of a video processing model to process the target video data, obtaining feature representation information of the target video data; the encoder is pre-trained based on multiple frames of images included in the sample video data and a preset masking strategy; based on the video processing model and the feature representation information of the target video data, the processing result of the target video data is determined. By using this embodiment and the encoder trained based on a preset masking strategy, the feature representation information of the video data can be extracted accurately and efficiently, thereby accurately determining the processing result of the video data.

[0053] Please see Figure 3 , Figure 3 This is a schematic diagram of the training process of a video representation learning model provided in an embodiment of this application. This process can be executed by a computer device. Figure 3 As shown, the training process may include the following steps S301-S306:

[0054] S301. Obtain training samples.

[0055] The training samples include multiple sample video data.

[0056] In one alternative implementation, the computer device may obtain training samples from a publicly available dataset, such as the Kinetics-400 (or K400) dataset, the Something-Something-V2 (or SSV2) dataset, etc.

[0057] S302. Preprocess the video data of multiple samples included in the training samples.

[0058] In one optional implementation, the computer device preprocesses the training samples including multiple sample video data by: performing frame extraction processing on each sample video data in the multiple sample video data to obtain a preset number of original images; and performing sampling processing on the preset number of original images of each sample video data according to a preset sampling interval to obtain multiple original images including each sample video data.

[0059] Taking a sample video data as an example, the computer device can first use a random sampling method to process the sample video data into frames, and obtain a preset number of frames (e.g., 32 frames) of original images, where these 32 original images are continuous; then, according to a preset sampling interval (assuming the interval is 2), the 32 original images are temporally downsampled to obtain 16 original images included in the sample video data. Figure 4The 401 section refers to the multiple original images obtained after the computer device preprocesses the video data. Optionally, the computer device can also directly extract 16 original images from the sample video data. Optionally, the computer device can input these 16 original images into the video representation learning model for model training.

[0060] S303. Using a preset masking strategy, mask the multiple original images included in each sample video data in multiple sample video data to obtain the preserved image information of each sample video data.

[0061] The preset occlusion strategy includes adjacent images having the same occlusion rules and an occlusion ratio greater than or equal to a preset ratio.

[0062] Preserving image information refers to retaining the image information within an image block. Preserving an image block involves dividing an original frame into several image blocks, randomly occluding these blocks based on an occlusion ratio, and then selecting the unoccluded image blocks as the preserved image blocks.

[0063] In one alternative implementation, before the computer device performs masking processing on the multiple original images included in each sample video data using a preset masking strategy, it may also perform video block embedding processing on the multiple original images included in each sample video data.

[0064] Taking a sample video dataset as an example, assuming the dataset consists of 16 original images, and the size of the video segment composed of these 16 original images is 16×224×224 (T×H×W), the computer device will use video block embedding to convert a pixel block of size 2×16×16 into a token. After this conversion, a token set of size 8×14×14 will be obtained. Here, T represents the number of channels, H represents the height of the original image, and W represents the width of the original image.

[0065] In one optional implementation, the computer device uses a preset masking strategy to mask multiple frames of original images included in each sample video data in multiple sample video data to obtain retained image information for each sample video data, including: determining a target original image from the multiple frames of original images, wherein the target original image is one of the frames of original images; determining the retained image information of the target original image based on the masking ratio; and determining the retained image information of other frames of original images in the multiple frames of original images based on the retained image information of the target original image.

[0066] Optionally, the computer device can determine the target original image from multiple original images randomly or by specifying it. Taking 16 original images as an example, the computer device can randomly select one image from these 16 original images as the target original image (assuming it is the first frame), or it can specify the fifth original image as the target original image.

[0067] Optionally, due to the extremely high redundancy of video data, the computer device can use a masking ratio greater than or equal to a preset ratio when performing masking (or masking) processing, masking most of the input video (assuming 90%-95%) and retaining only a very small portion of the information (assuming 5%-10%) as input to the encoder. This not only greatly reduces the computational cost of the training process but also allows the encoder to learn more valuable (or more spatiotemporally focused) high-level feature information from the video data.

[0068] In one optional implementation, the computer device determines the retained image information of other original images in a multi-frame set based on the retained image information of the target original image, including: obtaining the position information of each image block in the target original image from the retained image information of the target original image; determining the target image block at the corresponding position in each other original image in the multi-frame set based on the position information of each image block in the target original image; and determining the retained image information of each other original image based on the target image block at the corresponding position in each other original image.

[0069] For example, suppose there are 16 original images, each divided into 100 image blocks, and the target original image is the 3rd original image. The computer device can first perform random occlusion processing on the 100 image blocks included in the 3rd original image based on the occlusion ratio (assuming it is 95%), and determine that the number of retained image blocks in the 3rd original image is 100*(1-95%) = 5. Next, the position information (or position coordinates) of each of these 5 retained image blocks in the 3rd original image is obtained. Then, based on the position information of each retained image block in the 3rd original image, the 5 image blocks at the corresponding positions in other original images are determined. Finally, based on the 5 image blocks at the corresponding positions in each of the other original images, the retained image information of each of the other original images is determined. Because video data has a strong temporal correlation between consecutive frames, this ensures the consistency of the preserved or masked image blocks for adjacent frame images (or tokens in the same time dimension). This effectively avoids the leakage of masked image block information between adjacent frames, and thus avoids the situation where the encoder only needs to learn a simple copy and paste operation to recover the masked image block information when information is leaked.

[0070] Alternatively, the above method can also be called a tube masking method. Tube masking refers to the fact that for the same video data, the masking strategy is the same for each image frame. The same masking strategy means that for adjacent frames, the positions of the preserved or masked image blocks are the same.

[0071] like Figure 4 As shown, Figure 4 Part 402 in the diagram corresponds to step S303. The four image blocks marked with shaded areas in 402 represent the retained image blocks, while the remaining white image blocks represent the occluded image blocks. The four image blocks connected by solid lines indicate that the retained image blocks are in the same position in each of the four image frames, while the four image blocks connected by dashed lines indicate that the occluded image blocks are in the same position in each of the four image frames. The Token w / o[M] in 402 represents the retained image information of the sample video data.

[0072] S304. The video representation learning model is invoked, including the initialized encoder, to process the preserved image information of each sample video data to obtain the feature representation information of each sample video data.

[0073] In one optional implementation, the computer device invokes an initialized encoder included in the video representation learning model to process the preserved image information of each sample video data to obtain feature representation information of each sample video data. This includes: invoking an initialized encoder included in the video representation learning model to process the preserved image information of each sample video data to obtain feature representation information of the preserved image information of each sample video data; generating filled feature representation information of each sample video data through random initialization; and determining the feature representation information of each sample video data based on the feature representation information of the preserved image information and the filled feature representation information of each sample video data.

[0074] Taking a sample video data as an example, assuming that the preserved image information of the sample video data includes the information of 5 image blocks in each of the 16 original images, the computer device can first call the initialized encoder to process the information of the 5 image blocks in each of the 16 original images to obtain the feature representation information of the 5 image blocks in each of the 16 original images; then, the padding feature information of the 16 original images is generated through random initialization; finally, based on the feature representation information of the 5 image blocks in each of the 16 original images and the padding feature information of the 16 original images, the feature representation information of the sample video data is determined.

[0075] In one alternative implementation, the computer device may employ a joint space-time self-attention layer to extract features from a subset of retained image information. The joint self-attention layer is a special structure in the Transformer network that includes multiple dot product operations; this layer is one of the main reasons why the Transformer network can output higher-quality features.

[0076] like Figure 4 As shown, Figure 4 Part 403 corresponds to step S304. Here, white represents the feature representation information of the preserved image information obtained after processing the preserved image information of the sample video data using the encoder, black represents the filled feature representation information of the sample video data, and the combination of white and black represents the feature representation information of the sample video data.

[0077] S305. Call the video representation learning model, including the initialized decoder, to process the feature representation information of each sample video data to obtain a multi-frame reference image for each sample video data.

[0078] In one alternative implementation, after obtaining the feature representation information of each sample data, the computer device may call the initialized decoder included in the video representation learning model to decode the feature representation information of each sample video data, thereby obtaining a multi-frame reference image of each sample video data.

[0079] In one alternative implementation, the computer device may employ a lightweight decoder to decode the feature representation information (or high-level feature) extracted from the encoder to obtain multiple reference images; this process may also be referred to as reconstructing the original video.

[0080] In one alternative implementation, the video representation learning model may include an initialized decoder that can be obtained by a computer device by setting the number of channels of the decoder of the Transformer model to half the original number of channels.

[0081] like Figure 4 As shown, Figure 4 The 404 section corresponds to step S305. The shaded portion represents the reference feature representation information obtained after processing the feature representation information of the sample video data using the decoder. The Targetvideo clip in 404 represents the multi-frame reference image.

[0082] S306. Based on the differences between multiple reference images and multiple original images, the model parameters of the video representation learning model are adjusted to obtain the trained encoder and decoder.

[0083] In one optional implementation, when the computer device adjusts the model parameters of the video representation learning model based on the differences between the multi-frame reference images and the multi-frame original images, it can calculate a loss function between the multi-frame reference images and the multi-frame original images; and adjust the model parameters of the video representation learning model based on the loss function. Optionally, the computer device can use the mean-square error loss between the multi-frame reference images and the multi-frame original images as the loss function.

[0084] In one alternative implementation, after obtaining the trained encoder and decoder, the computer device can further fine-tune the model parameters based on all the input sample video data. Optionally, during the fine-tuning phase, the computer device can also employ a higher occlusion ratio to reduce computational costs.

[0085] As mentioned above, the video representation learning model trained through steps S301 to S306 is a proxy task for video self-supervision and training based on the Masked-Video-Modeling algorithm framework. Therefore, this video representation learning model can also be called a video self-supervised representation learning model based on a masked autoencoder (or a video masked autoencoder).

[0086] It should be noted that the video representation learning model provided in this application embodiment is trained based on a self-supervised learning method. Therefore, this video representation learning model does not require pre-training using image data or a large-scale video dataset. It can be widely used in application scenarios where video training data lacks labeled information, especially in scenarios where training data or labeled information is extremely difficult to obtain. For example, it can be applied to scenarios such as internet short video classification and video action detection. It can still perform good pre-training even when data and labeled information are difficult to obtain, thus promoting the development of the video understanding field. Optionally, if a large-scale video dataset is available, this video representation training model can also be pre-trained on such a large dataset.

[0087] In this embodiment, a computer device acquires training samples; preprocesses multiple sample video data included in the training samples; uses a preset masking strategy to mask multiple frames of original images included in each sample video data to obtain preserved image information for each sample video data. The preset masking strategy includes that adjacent images have the same masking rules and that the masking ratio is greater than or equal to a preset ratio; calls an initialized encoder included in the video representation learning model to process the preserved image information of each sample video data to obtain feature representation information for each sample video data; calls an initialized decoder included in the video representation learning model to process the feature representation information of each sample video data to obtain multiple frames of reference images for each sample video data; and adjusts the model parameters of the video representation learning model based on the differences between the multiple frames of reference images and the multiple frames of original images to obtain the trained encoder and decoder. Using this embodiment, since the encoder processes the preserved image information of each sample video data obtained using the preset masking strategy, the high masking ratio not only greatly reduces the computational cost of the training process but also allows the encoder to learn more valuable high-dimensional features, thereby accurately and efficiently extracting the feature representation information of the video data. On the other hand, since adjacent images share the same occlusion rules, information about the masked image patches can be effectively avoided from leaking between adjacent frames. This prevents the encoder from simply learning a copy-paste operation to recover the masked image patch information when information is leaked, thereby improving the accuracy of the feature representation information of the video data. Therefore, transferring the trained encoder as a backbone network to downstream tasks can achieve excellent performance in downstream tasks (such as video classification, action detection, etc.).

[0088] Furthermore, since the encoder in the video representation learning model processes the preserved image information of each sample video data obtained using a preset masking strategy, and the decoder processes the feature representation information of each sample video data, and the decoder is lightweight, the video representation learning model is an asymmetric encoder-decoder architecture, which further reduces the computational cost of model training.

[0089] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of a video representation learning model provided in an embodiment of this application. Please refer to... Figures 6a to 6c , Figure 6a This is a schematic diagram of another video representation learning model based on a masking autoencoder provided in this application embodiment; Figure 6b This is a schematic diagram of the encoder structure in the video representation learning model provided in this application embodiment; Figure 6c This is a schematic diagram of the structure of the MLP module in the encoder provided in this application embodiment. The following is in conjunction with... Figure 5 and Figures 6a to 6b The training process of the video representation learning model provided in the embodiments of this application is described.

[0090] Step 1: Preprocess the sample video data (corresponding to...) Figure 5 The row containing the data, Figure 6a (601 in the dataset). Specifically, sample video data from dataset K400 was temporally downsampled at intervals of 4, and sample video data from dataset SSV2 was temporally downsampled at intervals of 2, resulting in video segments of size 3×16×224×224. Here, 3 represents the three dimensions of the RGB image, 16 represents the number of channels, and 224×224 represents the image's height × width.

[0091] Step 2: Perform video block embedding processing on the preprocessed video data of size 3×16×224×224 (corresponding to...) Figure 5 The line containing the cube Figure 6a (Referring to section 602). The original 16 frames of 224×224 pixels (i.e., a 16×224×224 video clip) undergo video block embedding processing. This transforms 2×16×16 pixel blocks into tokens (or vectors). After this processing, a token set of 8×14×14 pixels is obtained. Flattening the 14×14 image blocks along both dimensions yields 196 tokens. Each image block is then mapped to a one-dimensional vector using linear mapping, resulting in a vector of length 768. In other words, this processing yields a video clip of size 8×768×196.

[0092] Step 3: Perform masking processing on the video segment with a size of 8×768×196 according to the preset masking strategy (corresponding to...). Figure 5 The line containing the mask, Figure 6a (603 in the example). A video segment of size 8×768×196 is masked according to a preset masking strategy. Assuming the masking ratio in the preset masking strategy is ρ, the masking process will result in a preserved image information of size 8×768×196(1-ρ).

[0093] Step 4: Input the retained image information of size 8×768×196(1-ρ) into the encoder for encoding (corresponding to...) Figure 5 The line containing the encoder in the code. Figure 6a (604 in the middle). For example Figure 6bAs shown, encoder 604 includes a layer normalization module 609, a multi-head attention (MHA) module 610, and a multilayer perceptron (MLP) module 611. The MHA module 610 has 768 input nodes. Figure 6c As shown, the MLP module 610 consists of a fully connected layer (Liner) 612, a GELU activation function 613, and Dropout (614). Its first fully connected layer (i.e., Figure 6c The 612) in the formula quadruples the number of input nodes, resulting in 768 × 4 = 3072. Here, 12 represents the number of stacked transformer layers. The final result is 8 × 768 × 196 (1-ρ) feature representations that retain information.

[0094] Step 5: Use the encoder's MLP module to obtain supplementary feature representation information (corresponding to...) Figure 5 The line containing projector1, Figure 6a (605 in the middle). The input node of the MLP module is 384, so based on the feature information of the retained information and the supplementary feature information, a feature representation information of size 8×384×196 can be obtained.

[0095] Step Six: Input the 8×384×196 feature representations into the decoder for decoding (corresponding to...) Figure 5 The line containing the decoder, Figure 6a (606 in the example). The MHA module has 384 input nodes. The first fully connected layer of the MLP module quadruples the number of input nodes, i.e., 384×4=1536. The number of stacked transformer layers is 4, thus obtaining 8×384×196 reference feature information.

[0096] Step 7: Process the 8×384×196 reference feature information to obtain the decoded reference video segment (corresponding to...). Figure 5 The line containing projector2, Figure 6a (607 in the text). The size of its reference video clip is 8×1536×196.

[0097] Step 8: Reconstruct the reference video clip with a size of 8×1536×196 (corresponding to...) Figure 5 The line containing reshape, Figure 6a (608 in the example). Dividing the 1536 tokens into image blocks of size 3×2×16×16 yields a reference video clip of size 3×16×224×224.

[0098] In this embodiment, since the reference video segment obtained in step eight is the same size as the video segment obtained in step one, the accuracy of the video representation learning model provided in this application is verified, thereby verifying the accuracy of the feature representation information of the video data obtained by the encoder in the video representation learning model. Furthermore, practical experience shows that the video representation learning model provided in this embodiment can be successfully pre-trained on video data of any order of magnitude (e.g., 3,000 to 200,000) and achieves excellent results.

[0099] It should be noted that when the embodiments of this application are applied to specific products or technologies, the target video data to be processed, sample video data, etc. involved in the embodiments of this application are obtained only after obtaining the user's permission or consent; and the collection, use and processing of the video data to be processed, sample video data, etc., must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0100] Based on the description of the relevant embodiments of the video processing method above, this application also proposes a video processing apparatus, which can be a computer program (including program code) running on a computer device. The video processing apparatus can execute... Figure 2 The video processing method shown; please refer to [link / reference]. Figure 7 , Figure 7 This is a schematic diagram of a video processing apparatus provided in an embodiment of this application. The video processing apparatus may include the following modules:

[0101] The acquisition module 701 is used to acquire the target video data to be processed;

[0102] The processing module 702 is used to call the encoder of the video processing model to process the target video data and obtain the feature representation information of the target video data. The encoder is pre-trained based on the multi-frame images included in the sample video data and the preset occlusion strategy. The preset occlusion strategy includes that the occlusion rules of adjacent images are the same and the occlusion ratio is greater than or equal to the preset ratio.

[0103] The determination module 703 is used to determine the processing result of the target video data based on the video processing model and the feature representation information of the target video data.

[0104] In one alternative embodiment, the video processing apparatus further includes a training module 704.

[0105] In one optional implementation, the training module 704 is used to: acquire training samples, which include multiple sample video data; and pre-train the initialized encoder and decoder of the video representation learning model using the multiple sample video data and a preset masking strategy to obtain the trained encoder and decoder.

[0106] In an alternative implementation, the processing module 702 is further configured to use the trained encoder as the encoder of the video processing model.

[0107] In an optional implementation, when the training module 704 pre-trains the initialized encoder and decoder of the video representation learning model using multiple sample video data and a preset masking strategy to obtain the trained encoder and decoder, it specifically performs the following:

[0108] Using a preset masking strategy, masking is performed on multiple original images of each sample video data in multiple sample video data to obtain the preserved image information of each sample video data.

[0109] The video representation learning model is invoked, including the initialized encoder, to process the preserved image information of each sample video data to obtain the feature representation information of each sample video data;

[0110] The video representation learning model is invoked, including the initialized decoder, to process the feature representation information of each sample video data, and obtain multi-frame reference images for each sample video data.

[0111] Based on the differences between multiple reference images and multiple original images, the model parameters of the video representation learning model are adjusted to obtain the trained encoder and decoder.

[0112] In an optional implementation, when the training module 704 performs masking processing on the multiple frames of original images included in each sample video data in multiple sample video data using a preset masking strategy to obtain the retained image information of each sample video data, it is specifically used for:

[0113] The target original image is determined from one of the multiple original images;

[0114] Determine the retained image information of the original target image based on the occlusion ratio;

[0115] Based on the retained image information of the target original image, determine the retained image information of other original images in the multi-frame original image.

[0116] In an optional implementation, the training module 704, when determining the retained image information of other frames of original images among multiple frames of original images based on the retained image information of the target original image, has the following functions:

[0117] Obtain the position information of each image patch in the target original image from the retained image information;

[0118] Based on the position information of each image patch in the target original image, the target image patch at the corresponding position in each other original image in multiple frames of original images is determined.

[0119] Based on the target image blocks at the corresponding positions of the other original images, determine the retained image information of the other original images.

[0120] In an optional implementation, when the training module 704 processes the retained image information of each sample video data by invoking the initialized encoder included in the video representation learning model to obtain the feature representation information of each sample video data, it specifically performs the following:

[0121] The video representation learning model is invoked, including the initialized encoder, to process the preserved image information of each sample video data, thereby obtaining the feature representation information of the preserved image information of each sample video data;

[0122] Filled feature representation information for each sample video data is generated through random initialization;

[0123] Based on the feature representation information that preserves the image information and the padding feature representation information for each sample video data, the feature representation information for each sample video data is determined.

[0124] In an optional implementation, the training module 704 is further configured to perform frame extraction processing on each sample video data in the multiple sample video data to obtain a preset number of original images.

[0125] Each sample video data is processed by sampling a preset number of original images at a preset sampling interval to obtain multiple original images included in each sample video data.

[0126] In an optional implementation, when determining the processing result of the target video data based on the video processing model and the feature representation information of the target video data, the determining module 703 is specifically used for:

[0127] The classification network of the video processing model is invoked to process the feature representation information of the target video data, and the classification result of the target video data is obtained. The classification result includes one or more of video topic classification and video action classification.

[0128] According to one embodiment of this application, Figure 2 The method shown and Figure 3 Each step involved in the training process shown can be derived from... Figure 7 The video processing device shown is executed by various modules within it. For example, Figure 2 Step S201 shown can be performed by Figure 7 The acquisition module 701 shown is executed, and step S202 can be performed by... Figure 7 The processing module 702 shown is executed, and step S203 can be performed by... Figure 7 The determination module 703 shown is executed. For example, Figure 3 Steps S301-S306 shown can all be performed by Figure 7 The training module 704 shown in the figure performs, etc.

[0129] According to another embodiment of this application, Figure 7 The various modules in the video processing apparatus shown can be individually or entirely combined into one or more other modules, or some of the modules can be further divided into multiple functionally smaller modules. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above modules are based on logical function division. In practical applications, the function of one module can be implemented by multiple modules, or the function of multiple modules can be implemented by one module. In other embodiments of this application, the video processing apparatus also includes other modules. In practical applications, these functions can also be implemented with the assistance of other modules, and can be implemented collaboratively by multiple modules.

[0130] According to another embodiment of this application, the following can be executed by running on a general-purpose computer device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). Figure 2 or Figure 5 The computer program (including program code) for each step involved in the corresponding method shown, to construct such... Figure 7 The video processing apparatus shown herein, and the video processing method for implementing the embodiments of this application, are described. A computer program may be recorded on, for example, a computer-readable storage medium, loaded onto the aforementioned computer device via the computer-readable storage medium, and executed therein.

[0131] It is understood that the specific implementation of each unit in the video processing apparatus provided in this application embodiment and the beneficial effects that can be achieved can be referred to the description of the foregoing video processing method embodiment, and will not be repeated here.

[0132] Based on the description of the above method and apparatus embodiments, this application also provides a computer device. Please refer to... Figure 8 The computer device includes at least a processor 801, a memory 802, and a communication interface 803. The processor 801, memory 802, and communication interface 803 can be connected via a bus 804 or other means; this embodiment of the application takes connection via bus 804 as an example.

[0133] The processor 801 (or CPU, Central Processing Unit) is the computing and control core of the computer device. It can parse various instructions and process various data within the computer device. For example, the CPU can parse power-on / off commands sent by the user and control the computer device to perform power-on / off operations; it can also transmit various interactive data between internal structures of the computer device. The communication interface 803 may optionally include standard wired interfaces or wireless interfaces (such as Wi-Fi, mobile communication interfaces, etc.), controlled by the processor 801 for sending and receiving data. The memory 802 is the storage device in the computer device, used to store computer programs and data. It is understood that the memory 802 here can include the computer device's built-in memory, or it can include extended memory supported by the computer device. The memory 802 provides storage space for the computer device's operating system, which may include, but is not limited to, Windows, Linux, Android, iOS, etc., and this application does not limit this. In an optional implementation, the processor 801 of this application embodiment can perform the following operations by running a computer program stored in the memory 802:

[0134] Acquire the target video data to be processed;

[0135] The encoder of the video processing model is invoked to process the target video data and obtain the feature representation information of the target video data. The encoder is pre-trained based on multiple frames of images included in the sample video data and a preset occlusion strategy. The preset occlusion strategy includes that the occlusion rules of adjacent images are the same and the occlusion ratio is greater than or equal to a preset ratio.

[0136] Based on the video processing model and the feature representation information of the target video data, the processing result of the target video data is determined.

[0137] In an alternative implementation, the processor 801 is further configured to:

[0138] Obtain training samples, which include multiple sample video data;

[0139] The video representation learning model, including the initial encoder and decoder, is pre-trained using multiple sample video data and a preset masking strategy to obtain the trained encoder and decoder.

[0140] The trained encoder is used as the encoder for the video processing model.

[0141] In an optional implementation, when the processor 801 pre-trains the initialized encoder and decoder of the video representation learning model using multiple sample video data and a preset masking strategy to obtain the trained encoder and decoder, it specifically performs the following:

[0142] Using a preset masking strategy, masking is performed on multiple original images of each sample video data in multiple sample video data to obtain the preserved image information of each sample video data.

[0143] The video representation learning model is invoked, including the initialized encoder, to process the preserved image information of each sample video data to obtain the feature representation information of each sample video data;

[0144] The video representation learning model is invoked, including the initialized decoder, to process the feature representation information of each sample video data, and obtain multi-frame reference images for each sample video data.

[0145] Based on the differences between multiple reference images and multiple original images, the model parameters of the video representation learning model are adjusted to obtain the trained encoder and decoder.

[0146] In an optional implementation, when the processor 801 performs masking processing on the multiple frames of original images included in each sample video data in multiple sample video data using a preset masking strategy to obtain the retained image information of each sample video data, it specifically performs the following:

[0147] The target original image is determined from one of the multiple original images;

[0148] Determine the retained image information of the original target image based on the occlusion ratio;

[0149] Based on the retained image information of the target original image, determine the retained image information of other original images in the multi-frame original image.

[0150] In one alternative implementation, the processor 801, when determining the retained image information of other frames of original images among multiple frames of original images based on the retained image information of the target original image, has the following functions:

[0151] Obtain the position information of each image patch in the target original image from the retained image information;

[0152] Based on the position information of each image patch in the target original image, the target image patch at the corresponding position in each other original image in multiple frames of original images is determined.

[0153] Based on the target image blocks at the corresponding positions of the other original images, determine the retained image information of the other original images.

[0154] In one optional implementation, the processor 801 adjusts the point cloud of the virtual eyeball according to a first adjustment parameter, aligning the point cloud of the virtual eyeball with the point cloud of the eye socket in the first virtual video. The first adjustment parameter includes one or more of rotation, translation, and scaling. When invoking the initialized encoder included in the video representation learning model to process the preserved image information of each sample video data to obtain feature representation information for each sample video data, the processor 801 specifically performs the following:

[0155] The video representation learning model is invoked, including the initialized encoder, to process the preserved image information of each sample video data, thereby obtaining the feature representation information of the preserved image information of each sample video data;

[0156] Filled feature representation information for each sample video data is generated through random initialization;

[0157] Based on the feature representation information that preserves the image information and the padding feature representation information for each sample video data, the feature representation information for each sample video data is determined.

[0158] In an alternative implementation, the processor 801 is further configured to:

[0159] Frame extraction is performed on each sample video data from multiple sample video data to obtain a preset number of original images;

[0160] Each sample video data is processed by sampling a preset number of original images at a preset sampling interval to obtain multiple original images included in each sample video data.

[0161] In an optional implementation, when determining the processing result of the target video data based on the video processing model and the feature representation information of the target video data, the processor 801 specifically performs the following:

[0162] The classification network of the video processing model is invoked to process the feature representation information of the target video data, and the classification result of the target video data is obtained. The classification result includes one or more of video topic classification and video action classification.

[0163] In specific implementations, the processor 801, memory 802, and communication interface 803 described in the embodiments of this application can execute the implementation of the computer device described in the video processing method provided in the embodiments of this application, or the implementation of the video processing apparatus provided in the embodiments of this application, which will not be repeated here.

[0164] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to execute the video processing method described in any of the possible implementations above. Specific implementations can be found in the preceding description and will not be repeated here.

[0165] This application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the video processing method described in any of the possible implementations above. Specific implementations can be found in the foregoing description and will not be repeated here.

[0166] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0167] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0168] The above-disclosed embodiments are only some of the embodiments of this application, and should not be construed as limiting the scope of this application. Therefore, any equivalent changes made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A video processing method, characterized in that, The method includes: Acquire the target video data to be processed; The encoder of the video processing model is invoked to process the target video data to obtain the feature representation information of the target video data; Based on the video processing model and the feature representation information of the target video data, the processing result of the target video data is determined; The training process of the encoder includes: Acquire training samples, which include multiple sample video data; A preset masking strategy is used to mask multiple frames of original images included in each sample video data in the plurality of sample video data to obtain the retained image information of each sample video data; wherein, the preset masking strategy includes that the masking rules of adjacent images are the same and the masking ratio is greater than or equal to a preset ratio, and the masking rules of adjacent images are the same, which means that the positions of the retained image blocks and the masked image blocks are consistent for adjacent images. The video representation learning model includes an initialized encoder that processes the preserved image information of each sample video data to obtain the feature representation information of each sample video data. The video representation learning model includes an initialized decoder that processes the feature representation information of each sample video data to obtain a multi-frame reference image of each sample video data. Based on the differences between the multi-frame reference images and the multi-frame original images, the model parameters of the video representation learning model are adjusted to obtain the trained encoder and decoder, and the trained encoder is used as the encoder of the video processing model.

2. The method according to claim 1, characterized in that, The step of using a preset masking strategy to mask multiple frames of original images included in each of the multiple sample video data to obtain the preserved image information of each sample video data includes: The target original image is determined from the multiple original images, wherein the target original image is one of the multiple original images; The retained image information of the original target image is determined based on the occlusion ratio; Based on the retained image information of the target original image, the retained image information of other original images in the multi-frame original images is determined.

3. The method according to claim 2, characterized in that, The step of determining the retained image information of other frames of the original images in the multi-frame original images based on the retained image information of the target original image includes: Obtain the position information of each image block in the target original image from the retained image information of the target original image; Based on the position information of each image block in the target original image, the target image block at the corresponding position in each other original image in the multi-frame original image is determined; Based on the target image blocks at the corresponding positions of the other original images in each frame, the retained image information of the other original images in each frame is determined.

4. The method according to any one of claims 1 to 3, characterized in that, The video representation learning model includes an initialized encoder that processes the preserved image information of each sample video data to obtain feature representation information for each sample video data, including: The video representation learning model includes an initialized encoder that processes the preserved image information of each sample video data to obtain the feature representation information of the preserved image information of each sample video data. The filler feature representation information for each sample video data is generated through random initialization; Based on the feature representation information of the preserved image information and the padding feature representation information of each sample video data, the feature representation information of each sample video data is determined.

5. The method according to claim 1, characterized in that, The method further includes: Frame extraction is performed on each of the multiple sample video data to obtain a preset number of original images; Each sample video data is sampled at a preset sampling interval to obtain a preset number of original images, which are then processed to obtain multiple original images included in each sample video data.

6. The method according to claim 1, characterized in that, The step of determining the processing result of the target video data based on the video processing model and the feature representation information of the target video data includes: The classification network of the video processing model is invoked to process the feature representation information of the target video data to obtain the classification result of the target video data. The classification result includes one or more of video theme classification and video action classification.

7. A video processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire the target video data to be processed; The processing module is used to call the encoder of the video processing model to process the target video data and obtain the feature representation information of the target video data; The determining module is used to determine the processing result of the target video data based on the video processing model and the feature representation information of the target video data; The training process of the encoder includes: Acquire training samples, which include multiple sample video data; A preset masking strategy is used to mask multiple frames of original images included in each sample video data in the plurality of sample video data to obtain the retained image information of each sample video data; wherein, the preset masking strategy includes that the masking rules of adjacent images are the same and the masking ratio is greater than or equal to a preset ratio, and the masking rules of adjacent images are the same, which means that the positions of the retained image blocks and the masked image blocks are consistent for adjacent images. The video representation learning model includes an initialized encoder that processes the preserved image information of each sample video data to obtain the feature representation information of each sample video data. The video representation learning model includes an initialized decoder that processes the feature representation information of each sample video data to obtain a multi-frame reference image of each sample video data. Based on the differences between the multi-frame reference images and the multi-frame original images, the model parameters of the video representation learning model are adjusted to obtain the trained encoder and decoder, and the trained encoder is used as the encoder of the video processing model.

8. A computer device, characterized in that, The computer device includes a memory, a communication interface, and a processor, wherein the memory, the communication interface, and the processor are interconnected; the memory stores a computer program, and the processor calls the computer program stored in the memory to implement the video processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the video processing method according to any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions, which, when executed by a processor, implement the video processing method as described in any one of claims 1 to 6.

Images