Video processing method, device, computer equipment, medium and program product

Abstract

The embodiment of the present application discloses a video processing method, device, computer equipment, medium and program product. The method includes: obtaining positive sample video features corresponding to sample video data, and obtaining predicted video data corresponding to the sample video data; the predicted video data is video data obtained by reconstructing the sample video data; obtaining negative sample video features corresponding to historical sample video data, and pre-training a pre-trained video model based on the positive sample video features, the negative sample video features, the sample video data and the predicted video data to obtain a target video processing model; obtaining a video to be processed, and calling the target video processing model to process the video to be processed to obtain the video features of the video to be processed. The accuracy of the video features extracted from the video can be improved, thereby improving the accuracy of video processing.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a video processing method, apparatus, computer equipment, medium and program product. Background Art

[0002] With the rapid development of computer technology and multimedia technology, multimedia data has also developed rapidly. Users can watch a variety of videos on the Internet. In order to enable users to obtain videos of interest from a large number of videos, video processing such as video search, video classification, and video recognition is usually performed. In these video processing, video representation (or video features) of the video can usually be used to achieve video processing. Therefore, how to ensure the accuracy of video features has become a hot research issue in current computer vision technology. Summary of the invention

[0003] The embodiments of the present application provide a video processing method, apparatus, computer equipment, medium and program product, which can improve the accuracy of video features extracted from a video, thereby improving the accuracy of video processing.

[0004] The first aspect of the embodiment of the present application discloses a video processing method, the method comprising:

[0005] Acquire positive sample video features corresponding to sample video data, and acquire predicted video data corresponding to the sample video data; the predicted video data is video data obtained by reconstructing the sample video data;

[0006] Obtaining negative sample video features corresponding to historical sample video data, and pre-training a pre-trained video model based on the positive sample video features, the negative sample video features, the sample video data, and the predicted video data to obtain a target video processing model;

[0007] The video to be processed is obtained, and the target video processing model is called to perform video processing on the video to be processed to obtain video features of the video to be processed.

[0008] A second aspect of an embodiment of the present application discloses a video processing device, the device comprising:

[0009] An acquisition unit, used to acquire positive sample video features corresponding to sample video data, and acquire predicted video data corresponding to the sample video data; the predicted video data is video data obtained by reconstructing the sample video data;

[0010] A pre-training unit, used to obtain negative sample video features corresponding to historical sample video data, and pre-train a pre-trained video model based on the positive sample video features, the negative sample video features, the sample video data and the predicted video data to obtain a target video processing model;

[0011] The processing unit is used to obtain the video to be processed, and call the target video processing model to perform video processing on the video to be processed to obtain the video features of the video to be processed.

[0012] A third aspect of an embodiment of the present application discloses a computer device, comprising a processor and a memory, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method of the first aspect above.

[0013] A fourth aspect of an embodiment of the present application discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method of the first aspect.

[0014] The fifth aspect of the embodiments of the present application discloses a computer program product or a computer program, wherein the computer program product or the computer program includes computer instructions, and when the computer instructions are executed by a processor, the method of the first aspect is implemented.

[0015] In an embodiment of the present application, the positive sample video features corresponding to the sample video data can be obtained, and the predicted video data corresponding to the sample video data can be obtained; further, the negative sample video features corresponding to the historical sample video data can be obtained, so as to pre-train the pre-trained video model based on the positive sample video features, the negative sample video features, the sample video data and the predicted video data to obtain the target video processing model. Thus, the target video processing model can be called to perform video processing on the video to be processed to obtain the video features of the video to be processed. By implementing the above method, the pre-trained video model can be pre-trained based on data of multiple dimensions to improve the stability of the model training and improve the model performance, so as to improve the model's processing ability for the video, and then when the target video processing model obtained by pre-training is used to perform video processing tasks such as video recognition, video classification, video retrieval, etc. on the video, the accuracy of the video features extracted from the video can also be improved, thereby improving the accuracy of the video processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0017] Figure 1 is a schematic diagram of the architecture of a video processing system provided in an embodiment of the present application;

[0018] Figure 2 It is a flowchart of a video processing method provided by an embodiment of the present application;

[0019] Figure 3 It is a structural diagram of a pre-trained video model provided in an embodiment of the present application;

[0020] Figure 4 is a flowchart of another video processing method provided in an embodiment of the present application;

[0021] Figure 5a is a schematic diagram of the structure of another pre-trained video model provided in an embodiment of the present application;

[0022] Figure 5b It is a schematic diagram of the structure of a backbone network provided in an embodiment of the present application;

[0023] Figure 5c It is a structural diagram of pre-training of a pre-trained video model provided in an embodiment of the present application;

[0024] Figure 5d It is a structural schematic diagram of another pre-training video model provided in an embodiment of the present application for pre-training;

[0025] Figure 5e It is a structural diagram of pre-training of another pre-trained video model provided in an embodiment of the present application;

[0026] Figure 6 is a structural schematic diagram of a video processing device provided in an embodiment of the present application;

[0027] Figure 7 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0030] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0031] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0032] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further processing graphics so that the computer processing becomes an image that is more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0033] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0034] Natural language model: Natural language is relative to machine language. The natural language model converts a large amount of human language into machine language through statistical models, and then uses it for cognition, understanding and generation. Specific uses include machine translation and automatic question answering.

[0035] Pre-training: The language model is trained by a large amount of unlabeled language text to obtain a set of model parameters. The model is initialized with this set of parameters, and then fine-tuned based on the existing language model according to the specific task. The pre-training method has been proven to have better results in the classification and labeling tasks of natural language processing.

[0036] Based on the computer vision technology and machine learning technology mentioned in the above-mentioned artificial intelligence technology, the embodiment of the present application proposes a video processing solution to realize the pre-training of the pre-trained video model using sample video data without labeled information, so as to provide an adaptive pre-trained model for downstream tasks based on the video. For example, the downstream task can be a video processing task such as multi-label recognition, video classification, and video retrieval of the video. Specifically, the positive sample video features corresponding to the sample video data and the predicted video data corresponding to the sample video data can be obtained, and the negative sample video features corresponding to the historical sample video data can be obtained, so as to pre-train the pre-trained video model based on the positive sample video features, the negative sample video features, the sample video data, and the predicted video data to obtain the target video processing model. Furthermore, the target video processing model can be called to perform video processing. For example, the video to be processed can be obtained, and the target video processing model can be called to perform video processing on the video to be processed to obtain the video features of the video to be processed.

[0037] Optionally, the target video processing model can also be used to fine-tune downstream tasks so that the fine-tuned target video processing model can process video processing for downstream tasks. For example, the target video processing model can be fine-tuned using training video data with annotation information. Exemplarily, the training video data and the annotation information corresponding to the training video data can be obtained to train the target video processing model based on the training video data and the corresponding annotation information to obtain a trained target video processing model, which is the fine-tuned target video processing model.

[0038] In summary, pre-training of pre-trained video models based on data in multiple dimensions can improve the stability of model training to improve model performance, thereby improving the model's video processing capabilities. When the target video processing model obtained by pre-training is used to perform video processing tasks such as video recognition, video classification, and video retrieval, the accuracy of video features extracted from the video can be improved, thereby improving the accuracy of video processing.

[0039] In a specific implementation, the execution subject of the above-mentioned video processing solution can be a computer device, which includes but is not limited to a terminal or a server. In other words, the computer device can be a server or a terminal, or a system composed of a server and a terminal. Among them, the terminal mentioned above can be an electronic device, including but not limited to mobile phones, tablet computers, desktop computers, laptop computers, PDAs, vehicle-mounted devices, intelligent voice interaction devices, augmented reality / virtual reality (AR / VR) devices, helmet displays, wearable devices, smart speakers, smart home appliances, aircraft, digital cameras, cameras and other mobile Internet devices (Mobile Internet Device, MID) with network access capabilities. Among them, the server mentioned above can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, vehicle-road collaboration, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0040] In one implementation, when the computer device is a server, the embodiment of the present application provides a video processing system, such as Figure 1 As shown, the video processing system includes at least one terminal and at least one server; the terminal can obtain sample video data, and upload the obtained sample video data to the server (i.e., a computer device), so that the computer device can obtain the sample video data, and perform video processing based on the sample video data, such as further obtaining positive sample video features and predicted video data corresponding to the sample video data, as well as negative sample video features corresponding to historical sample video data, so that the server can pre-train the pre-trained video model based on these features.

[0041] Based on the video processing solution provided above, the embodiment of the present application provides a video processing method, which can be executed by the computer device mentioned above. Figure 2 The video processing method includes but is not limited to the following steps:

[0042] S201, obtaining positive sample video features corresponding to sample video data, and obtaining predicted video data corresponding to the sample video data.

[0043] Among them, the sample video data may refer to video data without labeled information, that is, the sample video data without labeled information can be used to implement pre-training of the pre-trained video model. For example, the labeled information can be understood differently according to different downstream tasks. Exemplarily, in a multi-label recognition task, the labeled information may refer to the label result corresponding to the sample video data (such as variety show videos or sports videos, etc.); exemplarily, in the video quality classification, the labeled information may refer to the quality classification result of the sample video data (such as high-quality video or low-quality video).

[0044] The sample video data may refer to a video, such as a video of various types such as games, variety shows, movies, etc. It should be noted that the number of sample video data used to pre-train the pre-trained video model is relatively large, and this application mainly uses a sample video data as an example for explanation.

[0045] In one implementation, sample video data may be obtained first, and then corresponding positive sample video features and predicted video data may be obtained based on the sample video data. In the application, the video features obtained based on the sample video data may be referred to as positive sample video features, and the predicted video data may refer to new video data generated by reconstructing the sample video data. The following describes the acquisition of positive sample video features and predicted video data.

[0046] In one implementation, first, the sample video data may be preprocessed to obtain the first target sample video data and the second target sample video data of the sample video data, wherein the preprocessing may include one or more of frame extraction processing, block processing, data enhancement processing, etc., which are not specifically limited in the present application. The implementation method of the preprocessing may be specifically described below. After obtaining the first target sample video data and the second target sample video data, the first target sample video data and the second target sample video data may be used to obtain the positive sample video features corresponding to the sample video data. Among them, the positive sample video features may include the first positive sample video features and the second positive sample video features. Optionally, the first target sample video data may be encoded to obtain the first positive sample video features of the sample video data; and the second target sample video data may be encoded to obtain the second positive sample video features of the sample video data.

[0047] In one implementation, when the positive sample video features of the sample video data are obtained, the predicted video data of the sample video data can also be obtained. The predicted video data can be the video data generated by the reconstruction processing based on the sample video data, wherein the reconstruction processing can be implemented based on the codec, that is, the sample video data can be used to perform the codec processing to achieve the reconstruction of the sample video data. In a specific implementation, the first target sample video data can be encoded to obtain the third positive sample video features of the sample video data; then, the third positive sample video features are decoded to obtain the predicted video data of the sample video data.

[0048] In one implementation, the positive sample video features (the first positive sample video features and the second positive sample video features) and the predicted video data can be obtained by calling a pre-trained video model. Optionally, after obtaining the first target sample video data and the second target sample video data based on the sample video data, the first target sample video data and the second target sample video data can be input into the pre-trained video model to obtain the first positive sample video features, the second positive sample video features and the predicted video data.

[0049] In one implementation, the pre-trained video model may include a first encoding module, a second encoding module, and a decoding module. The first encoding module may encode the first target positive sample video data to obtain the first positive sample video feature and the third positive sample video feature of the sample video data; the second encoding module may encode the second target positive sample video data to obtain the second positive sample video feature of the sample video data; and the decoding module may decode the third positive sample video feature to obtain the predicted video data of the sample video data.

[0050] In summary, the model structure of the pre-trained video model can be seen as follows Figure 3 As shown. Figure 3 As shown, the pre-trained language model may include a first encoding module, a second encoding module and a decoding module. Then, after the computer device obtains the sample video data and pre-processes the sample video data to obtain the corresponding first target sample video data and second target sample video data, the first target sample video data can be input into the first encoding module to obtain the first positive sample video feature and the third positive sample video feature, and at the same time, the second target sample video data can be input into the second encoding module to obtain the second positive sample video feature; then, the third positive sample video feature can be input into the decoding module to obtain the predicted video data.

[0051] The following specifically describes the preprocessing of sample video data (eg, the preprocessing may include one or more of frame extraction processing, block processing, data enhancement processing, etc.).

[0052] In one implementation, the number of video frames in a video is large. In order to reduce the processing complexity of the pre-trained video model, the sample video data can also be subjected to frame extraction processing to reduce the amount of data. That is, the pre-processing can include frame extraction processing, and the sample video data can be subjected to frame extraction processing, and the video data obtained after the frame extraction processing can be used as the input of the first encoding module and the second encoding module. For example, the video data obtained by the frame extraction processing can be used as the first target sample video data and the second target sample video data.

[0053] Optionally, the frame extraction process performed on the sample video data may be a random frame extraction method, a segmented frame extraction method, etc. In one possible implementation, for the random frame extraction method, a preset number of video frames may be extracted from the sample video data, and the preset number may be preset, such as a value of 10, 15, etc. In another possible implementation, for the segmented frame extraction method, the sample video data may be segmented to obtain a plurality of segmented video data, for example, the sample video data may be evenly divided to obtain a plurality of segmented video data. After obtaining a plurality of segmented video data, a specified number of video frames may be extracted from each segmented video data. Among them, when extracting a specified number of video frames from each segmented video data, a random frame extraction method may be used for extraction; the number of segmented video data and the specified number may be preset, for example, the number of segmented video data may be a value of 5, 10, etc., and the specified data may be a value of 1, 2, etc.

[0054] In one implementation, the preprocessing may include block processing. For example, the sample video data may be subjected to block processing in three dimensions, and the size of the block may have a certain influence on the performance of self-supervision, wherein the three dimensions may refer to time length, width and height, for example, the size of the block may be: 2*4*4. Exemplarily, assuming that the sample video data r∈C*T*H*W, wherein C is the number of channels (such as 1, 3, etc.), T is the time length (such as T∈{4,8,16,32}, or may also be understood as the number of video frames), and H and W are the width and height of the image, respectively. The sample video data may be subjected to block processing (such as a block size of 2*4*4), and then after the block processing, the first target sample video data r1∈C*32*(T / 2)*(H / 4)*(W / 4) may be obtained, and the first target sample video data r1 may be composed of a large number of video blocks of size 2*4*4.

[0055] In one implementation, preprocessing may include data enhancement processing, such as data enhancement processing methods such as color change, brightness change, saturation change, contrast change, random cropping, inversion, translation transformation, rotation transformation, affine transformation, noise addition, blurring, etc. Then the first target sample video data and the second target sample video data may be enhanced data corresponding to the sample video data, and the sample video data may be subjected to data enhancement processing to obtain the first target sample video data and the second target sample video data. Among them, the first target sample video data and the second target sample video data may be different enhanced data for the sample video data, and the sample video data may be subjected to different data enhancement processing, so as to ensure that the inputs of the first encoding module and the second encoding module in the pre-trained video model are different data obtained for the sample video data, so as to strengthen the difference between the first positive sample video feature and the second positive sample video feature obtained subsequently, thereby improving the robustness of model learning. In a specific implementation, the sample video data may be subjected to a first data enhancement processing to obtain the first target sample video data; and the sample video data may be subjected to a second data enhancement processing to obtain the second target sample video data.

[0056] Among them, the first data enhancement processing and the second data enhancement processing are different data enhancement processing, for example, one of the first data enhancement processing and the second data enhancement processing is a strong enhancement processing, and the other is a weak enhancement processing. For another example, the first data enhancement processing can be any one or more of the data enhancement processing methods mentioned above, and the second data enhancement processing can also be any one or more of the data enhancement processing methods mentioned above, and the first data enhancement processing and the second data enhancement processing use different data enhancement processing methods. Exemplarily, the first data enhancement processing can be a color change, and the second data enhancement processing can be a brightness change; Exemplarily, the first data enhancement processing can be a color change and inversion, and the second data enhancement processing can be a brightness change and inversion.

[0057] In addition to the above-described preprocessing including any one of frame extraction, block processing and data enhancement processing, preprocessing may also include any combination of these processes, for example, preprocessing may include frame extraction and data enhancement processing. Optionally, before performing data enhancement processing, the sample video data may be subjected to frame extraction, that is, video frames for subsequent data enhancement processing are extracted from the sample video data to obtain initial sample video data, which is synthesized based on one or more video frames. After obtaining the initial sample video data, the initial sample video data is subjected to data enhancement processing to obtain first target sample video data and second target sample video data.

[0058] In one implementation, preprocessing may also include frame extraction, block processing, and data enhancement processing. In the actual pre-training process, the pre-processing of the sample video data by the pre-trained video model during the pre-training process may generally be a combination of the above three processes. Optionally, the sample video data may first be subjected to frame extraction to obtain initial sample video data, and the initial sample video data may be synthesized based on one or more video frames; after obtaining the initial sample video data, the initial sample video data may be subjected to block processing to obtain sample video block data; further, the sample video block data may be subjected to a first data enhancement processing to obtain a first target sample video data, and the sample video block data may be subjected to a second data enhancement processing to obtain a second target sample video data. The specific implementation of the frame extraction, block processing, and data enhancement processing (the first data enhancement processing and the second data enhancement processing) may refer to the above description and will not be repeated here.

[0059] S202, obtaining negative sample video features corresponding to historical sample video data, and pre-training a pre-trained video model based on the positive sample video features, the negative sample video features, the sample video data and the predicted video data to obtain a target video processing model.

[0060] The target video processing model may include a first encoding module, which may be used to extract video features in the video.

[0061] In one implementation, negative sample video features corresponding to historical sample video data may be further obtained, and the historical sample video data may refer to video data different from the sample video data. In the present application, video features that are not obtained using sample video data may be referred to as negative sample video features, and video features obtained based on sample video data may be referred to as positive sample video features. As mentioned above, the video features corresponding to the historical sample video data may be referred to as negative sample video features. The negative sample video features corresponding to the historical sample video data may be video features obtained by processing the historical sample video data using a pre-training processing model during historical pre-training. These negative sample video features may be stored in a preset location so that they can be directly obtained when needed later. For example, the negative sample video features corresponding to the historical sample video data may be stored in a queue.

[0062] In one implementation, the loss value of the pre-trained video model can be determined based on the positive sample video features, the negative sample video features, the sample video data, and the predicted video data, so as to use the loss value to pre-train the pre-trained video model and obtain the corresponding target video processing model. Optionally, the loss value may include a first loss value and a second loss value, so as to implement the pre-training of the pre-trained video model based on multiple loss values. For example, the first loss value of the pre-trained video model can be determined based on the positive sample video features and the negative sample video features, wherein the positive sample video features include the first positive sample video features and the second positive sample video features, that is, the first loss value of the pre-trained video model can be determined based on the first positive sample video features, the second positive sample video features, and the negative sample video features. And the second loss value of the pre-trained video model can also be determined based on the sample video data and the predicted video data. Considering that the first target sample video data corresponding to the sample video data is input into the first encoding module, the second loss value of the pre-trained video model can be determined based on the first target sample video data corresponding to the sample video data and the predicted video data.

[0063] After obtaining the first loss value and the second loss value, the pre-trained video model can be pre-trained based on the first loss value and the second loss value to obtain a target video processing model. Optionally, a target loss value of the pre-trained video model can be determined based on the first loss value and the second loss value, so as to pre-train the pre-trained video model based on the target loss value to obtain the target video processing model. For example, the target loss value can be the sum of the first loss value and the second loss value.

[0064] In one implementation, contrast loss and reconstruction loss may be introduced to determine the target loss value of the pre-trained video model, wherein contrast loss may be used to calculate the first loss value, and reconstruction loss may be used to calculate the second loss value. For example, contrast loss may be used to calculate the first loss value of the pre-trained video model based on the first positive sample video feature, the second positive sample video feature, and the negative sample video feature; and reconstruction loss may be used to calculate the second loss value of the pre-trained video model based on the sample video data and the predicted video data.

[0065] From the above, it can be seen that the present application can adopt two self-supervised pre-training methods to realize the pre-training of the pre-trained video model. These two self-supervised pre-training methods are: self-supervised pre-training based on contrastive learning and self-supervised pre-training based on generative learning. Among them, the self-supervised pre-training based on contrastive learning involves the first encoding module and the second encoding module in the pre-trained video model, and the self-supervised pre-training based on generative learning involves the first encoding module and the decoding module in the pre-trained video model.

[0066] In self-supervised pre-training based on contrastive learning, in the absence of labeled information, positive sample pairs can be constructed based on the same sample, and different samples can be used as negative sample pairs, so that video representation (or information in the video) can be better learned from the semantic level. Among them, the positive sample pair can be constructed by the output of the first encoding module and the output of the second encoding module, that is, the first positive sample video feature and the second positive sample video feature can constitute a positive sample pair; and the negative sample pair can be constructed by the second positive sample video feature and the negative sample video feature.

[0067] In self-supervised pre-training based on generative learning, the input can be reconstructed so that the input and output are as similar as possible, that is, the first target sample video data and the predicted video data are as similar as possible, so as to better learn video representation from the pixel (or image) level.

[0068] From the above, it can be seen that in this application, the pre-training of the pre-trained video model can be achieved based on the self-supervised pre-training method of generative learning and contrastive learning, which can better learn the video information from the semantic and pixel levels, thereby helping to improve video understanding and the generalization ability of the model.

[0069] As can be seen from the above description, the above steps S201-S202 can be a process of obtaining a target video processing model from a pre-training process of a pre-trained video model. Optionally, in other embodiments, the following step S203 can also be included after step S203, which is mainly an actual application scenario of the target video processing model.

[0070] S203, obtaining the video to be processed, and calling the target video processing model to perform video processing on the video to be processed, to obtain the video features of the video to be processed.

[0071] In one implementation, the target video processing model may be used to implement video processing on a video. For example, a video to be processed may be obtained, and the target video processing model may be used to perform video processing on the video to be processed to obtain video features corresponding to the video to be processed.

[0072] Optionally, in actual application scenarios, the target video processing model can be applied to specific video processing tasks, such as video quality classification, video multi-label recognition, video retrieval, etc. In a possible implementation, the target video processing model may include other modules in addition to the first encoding module mentioned above, so that the first encoding module and the other modules constitute a complete video processing model that can be used in specific application scenarios. For example, in the classification of video quality, the other module may include a classification module, so that after obtaining the video features of the video to be processed based on the first encoding module in the target video model, the classification module in the target video model can also be called to classify the video features to obtain the classification results of the video to be processed. For another example, in the multi-label recognition of the video, the other module may include a recognition module, so that after obtaining the video features of the video to be processed based on the first encoding module in the target video model, the recognition module can also be called to recognize the video features to obtain the label recognition results of the video to be processed.

[0073] In one implementation, the training video data and the corresponding annotation information can be obtained to fine-tune the target video processing model to obtain a fine-tuned target video processing model, which can be applied to actual video processing tasks. For example, in video classification, the video to be processed can be obtained, and the target video processing model can be called to perform video processing on the video to be processed to obtain the video features of the video to be processed; after obtaining the video features, the video features can be further used for classification processing to obtain the classification results of the video to be processed.

[0074] In an embodiment of the present application, the positive sample video features corresponding to the sample video data can be obtained, and the predicted video data corresponding to the sample video data can be obtained; further, the negative sample video features corresponding to the historical sample video data can be obtained, so as to pre-train the pre-trained video model based on the positive sample video features, the negative sample video features, the sample video data and the predicted video data to obtain the target video processing model. Thus, the target video processing model can be called to perform video processing on the video to be processed, and the video features of the video to be processed can be obtained. By implementing the above method, the model can be pre-trained using a self-supervised pre-training method based on generative learning and contrastive learning, which can improve the stability of the model training to improve the model performance, thereby improving the model's processing ability for the video; the self-supervised pre-training based on contrastive learning pays more attention to the features at the semantic level, while the self-supervised pre-training based on generative learning pays more attention to the features at the pixel level. The self-supervised pre-training framework based on generative learning and contrastive learning can better learn the information of the video from the semantic and pixel levels, which helps to improve the video understanding and the generalization ability of the model. Furthermore, when the pre-trained target video processing model is used to perform video processing tasks such as video recognition, video classification, and video retrieval, the accuracy of video features extracted from the video can be improved, and the accuracy of video processing can also be improved.

[0075] Based on the above description, the embodiment of the present application further proposes another video processing method; in the embodiment of the present application, the video processing method is mainly described by taking a computer device executing the video processing method as an example. Figure 4 As shown, the video processing method includes but is not limited to the following steps:

[0076] S401, obtaining positive sample video features corresponding to sample video data, and obtaining predicted video data corresponding to the sample video data.

[0077] Among them, sample video data may refer to video data without labeled information, and the video data may refer to any video, such as variety shows, sports videos, etc.; positive sample video data may refer to video features obtained based on sample video data, and correspondingly, the negative sample video data mentioned below may refer to video features not obtained based on the sample video data; predicted video data may refer to video data generated by reconstruction processing based on sample video data.

[0078] In one implementation, sample video data may be first obtained, that is, corresponding positive sample video features and predicted video data may be obtained based on the sample video data. Optionally, the sample video data may be preprocessed to obtain first target sample video data and second target sample video data of the sample video data. The implementation of the preprocessing may refer to the relevant description in the above step S201, which will not be repeated here.

[0079] After obtaining the first target sample video data and the second target sample video data, the first target sample video data and the second target sample video data can be further used to obtain positive sample video features corresponding to the sample video data, wherein the positive sample video features can include the first positive sample video features and the second positive sample video features. Optionally, the first target sample video data can be encoded to obtain the first positive sample video features of the sample video data; and the second target sample video data can be encoded to obtain the second positive sample video features of the sample video data.

[0080] In one implementation, after obtaining the first target sample video data, the predicted video data of the sample video data can also be obtained based on the first target sample video data, such as the first target sample video data can be encoded and decoded to obtain the predicted video data. Optionally, the first target sample video data can be encoded to obtain a third positive sample video feature of the sample video data. The third sample video feature may be the same as or different from the first sample video feature mentioned above, and may have different results based on different pre-trained video model architectures. For a specific understanding, please refer to the following related description. After obtaining the third positive sample video feature, the third positive sample video feature can be decoded to obtain the predicted video data of the sample video data.

[0081] In one implementation, the positive sample video features (the first positive sample video features and the second positive sample video features) and the predicted video data can be obtained by calling a pre-trained video model. Figure 3 As shown, the pre-trained video model may include a first encoding module, a second encoding module, and a decoding module. The first encoding module may encode the first target positive sample video data to obtain the first positive sample video feature and the third positive sample video feature of the sample video data; the second encoding module may encode the second target positive sample video data to obtain the second positive sample video feature of the sample video data; and the decoding module may decode the third positive sample video feature to obtain the predicted video data of the sample video data.

[0082] In one implementation, in the process of obtaining the first positive sample video feature, the first target sample video data may be masked to enhance the encoding capability. The specific implementation method of using the first target positive sample video data for encoding to obtain the first positive sample video feature may be: first, the first target sample video data may be masked to obtain the masked video feature of the sample video data, wherein the masking may be understood as covering part of the video data in the first target sample video data, that is, the pixel value corresponding to the masked part may be changed to 0, and the pixel value corresponding to the uncovered part may remain unchanged; after obtaining the masked video feature, the masked video feature may be further feature extracted to obtain the first positive sample video feature of the sample video data.

[0083] In one implementation, the third positive sample video feature mentioned above can also be determined based on the mask video feature. In this case, the specific implementation method of obtaining the predicted video data of the sample video data can be: first, the mask video feature of the sample video data can be obtained, and the mask video feature can be obtained by masking the first target sample video data; after obtaining the mask video feature, the mask video feature can be extracted to obtain the third positive sample video feature of the sample video data; further, the sample video data can be predicted based on the third positive sample video feature, that is, the decoding process of the third positive sample video feature mentioned above can be performed to obtain the predicted video data of the sample video data.

[0084] In summary, the first encoding module in the pre-trained video model may include a mask module and a feature extraction module, wherein the mask module may be used to perform mask processing on the first target sample video data to obtain mask video features, and the feature extraction module may be used to perform feature extraction on the mask video features to obtain first positive sample visual features and third positive sample video features. For the second encoding module, the second encoding module may also include a feature extraction module to perform feature extraction (or encoding processing) on ​​the second target sample video data to obtain second positive sample video features. Among them, the feature extraction module in the first encoding module and the feature extraction module in the second encoding module may have the same module structure or different module structures, and their specific contents may refer to the following related descriptions. In order to distinguish the feature extraction modules in the two encoding modules, the feature extraction module in the first encoding module may be referred to as the first feature extraction module, and the feature extraction module in the second encoding module may be referred to as the second feature extraction module. For example, the structural diagram of the pre-trained video model mentioned here can be seen as follows. Figure 5a shown.

[0085] The following are Figure 5aEach module in the pre-trained video model shown is explained in detail.

[0086] In one implementation, the main function of the mask module can be to mask a certain proportion of pixel information of the input video data. In the application, the mask processing is performed on the first target sample video data. Optionally, the mask processing method may include: mask block and mask frame.

[0087] The masking method for the mask block is to mask the block of the first target sample video data. Optionally, if the first target sample video data is composed of a large number of video blocks, the video block can be masked on the first target sample video data, such as selecting a target mask block from all the video blocks included in the first target sample video data. The target mask frame can refer to a video block that needs to be masked. The number of the target videos can be one or more, and the number can be preset. The target mask block can be randomly selected or selected in a preset manner, which is not specifically limited in this application. After determining the target video block, the target video block can be masked.

[0088] For example, a block mask can be performed on the first target sample video data in three dimensions, and the size of the block will have a certain impact on the performance of self-supervision, where the three dimensions can refer to time length, width and height, such as the size of the block can be: 2*4*4. Exemplarily, assume that the sample video data r∈C*T*H*W, where C is the number of channels (such as 1, 3, etc.), T is the time length (such as T∈{4,8,16,32}, or can also be understood as the number of video frames), and H and W are the width and height of the image, respectively. In preprocessing, the sample video data can be divided into blocks (such as the block size is: 2*4*4), and the first target sample video data r1∈C*32*(T / 2)*(H / 4)*(W / 4) obtained after the block processing, the first target sample video data r1 can be composed of a large number of video blocks of size 2*4*4. Among them, the target mask block can be randomly selected from the video blocks included in the first target sample video data r1, and the pixels corresponding to the target mask block are multiplied by 0, and the non-target mask block is multiplied by 1, that is, the pixels corresponding to the target mask block are changed to 0, and the pixels corresponding to the non-target mask block are kept unchanged, thereby realizing the processing of the mask block.

[0089] The masking method for the masked frame is to perform frame masking (masking) on ​​the first target sample video data. The target video frame can be determined from the video frames included in the first target sample video data. The target video frame can refer to the video frame that needs to be masked. The number of the target video frames can be one or more, and the number can be preset; after determining the target video frame, the target video frame can be masked. Optionally, the target video frame can be randomly extracted from the included video frames; or the target video frame can be determined from the included video frames in a preset manner. For example, the preset manner can be an interval determination manner, that is, every other video frame is used as the target video frame. Exemplarily, assuming that the video frames included in the first target sample video data are video frame 1, video frame 2, video frame 3, video frame 4, ..., the target video frame can be video frame 2, video frame 4, ...

[0090] In one implementation, the first feature extraction module in the first encoding module and the second feature extraction module in the second encoding module can both be used to extract video features. The first feature extraction module and the second feature extraction module in the second encoding module can have the same structure or different structures.

[0091] Optionally, the first feature extraction module and the second feature extraction module may both include a backbone network (Backbone), which is a basic network used by the pre-trained video model, and the Backbone may specifically refer to a network used to extract video features in video frames. For example, the Backbone may adopt traditional CNN (Convolutional Neural Networks), Transformer and other networks. Since Transformer has better processing effects in the visual field, the Backbone in this application may be a 3D Swin Transformer. 3D Swin Transformer can introduce a sliding window mechanism so that the model can learn information across windows; at the same time, 3D Swin Transformer has a hierarchical design (ie, a downsampling layer), so that the model can process super-resolution images, save computational effort, and pay attention to global and local information.

[0092] For example, the structure of 3D Swin Transformer can be as follows Figure 5b As shown. Figure 5b As can be seen, 3D SwinTransformer adopts a hierarchical design, such as Figure 5bThe image shown in the figure contains 4 stages (understood as stages or layers). For example, for a video data (such as a certain Videos can be represented as T*H*W*3, and this data can be used to indicate the size of Videos), first, a video data (T*H*W*3) can be input into a block processing module for block processing to obtain block data. For example, the block processing module can be called a Patch Partition module. If the block size can be 2*4*4, the block data can be represented as (T / 2)*(H / 4)*(W / 4)*3*32. Among them, 3 is the number of channels (C). Then, the block data is linearly embedded through a linear embedding module, which can be used to reduce channels. For example, the linear embedding module can be called a Linear Embedding module. Furthermore, feature maps of different sizes can be constructed through four stages. Except for Stage 1, which first passes through the Linear Embedding module, the other three stages all use downsampling modules to reduce the resolution of the input feature map. For example, the downsampling module can be called a Patch Merging module. Then, the video feature fusion processing is performed through the video feature processing module, such as the Vioeos Swin Transformer Block. Among them, Figure 5b The data indicated by each Stage may be the size of the feature map output by the corresponding Stage. For example, the size of the feature map output by Stage 2 is (T / 2)*(H / 8)*(W / 8)*2C.

[0093] Among them, the structure of Backbone can also be changed according to the requirements of downstream tasks. For example, other deep learning networks such as Resnet and Vit can be used as Backbone.

[0094] The pre-training process of the pre-trained video model mentioned here can be as follows Figure 5c As shown. Figure 5cAs shown, after the sample video data is preprocessed by the preprocessing module, the first target sample video data and the second target sample video data can be obtained. Among them, the first target sample video data can be input into the mask module in the first encoding module to obtain the mask video feature, and then the mask video feature can be input into the Backbone to obtain the first positive sample video feature and the third positive sample video feature, where the first positive sample video feature and the third positive sample video feature can be the same video feature. It should be noted that Backbone only encodes visible video data (i.e., video data that has not been masked in the mask module), and the input of the subsequent decoding module is the complete video data. The second target sample video data can be input into the Backbone corresponding to the second encoding module to obtain the second positive sample video feature. The third positive sample video feature can be further input into the decoding module to obtain the predicted video data.

[0095] Optionally, the first encoding module and the second encoding module may also include a Projector module; that is, the first encoding module may include a mask module, a Backbone module, and a Projector module, and the second encoding module may include a Backbone module and a Projector module. Figure 5d As shown. The Projector module can be used to reduce the dimension of the features output by Backbone to reduce the processing complexity of the model. The Projector modules in the first encoding module and the second encoding module can have the same structure. For example, the Projector module can be an MLP (Multilayer Perceptron), such as a single-layer MLP, a two-layer MLP, etc., and batch normalization and activation function are not used.

[0096] Optionally, in order to achieve asymmetry between the first encoding module and the second encoding module to increase the difference between the first positive sample video feature and the second positive sample video feature, the first encoding module may further include a Predictor module; that is, the first encoding module may include a mask module, a Backbone, a Projector module and a Predictor module, and the second encoding module may include a Backbone and a Projector module. The pre-training process of the pre-trained video model mentioned here can be as follows: Figure 5e As shown in the figure, the Predictor module can also be an MLP, such as a single-layer MLP, a two-layer MLP, etc., and batch normalization and activation function are not used.

[0097] Optionally, the Predictor module and the Projector module may have the same structure or different structures. For example, the Predictor module and the Projector module may both be two-layer MLPs; or, for another example, the Predictor module may be a two-layer MLP and the Projector module may both be a single-layer MLP.

[0098] In one implementation, the purpose of the decoding module is to perform a reconstruction task at the pixel level, that is, to obtain the predicted video data, and the input of the decoding module can be a third positive sample video feature obtained by using the first encoding module. Specifically, the video feature output by the Backbone in the first encoding module can be used as the third positive sample video feature, and the decoding module can be called to decode the third positive sample video feature to reconstruct the original video data, that is, to obtain the predicted video data. Since the first encoding module is the one that actually learns the potential feature representation in the video data, the decoding module can have fewer channels and a shallower network depth than the first encoding module. Among them, the decoding module can be any form of network structure, as long as its output is consistent with the input of the first encoding module, so that the second loss value can be calculated later; at the same time, it can complete the pixel reconstruction task. For example, the decoding module can be a simple structure such as a two-layer MLP or a two-layer Tansformer. Exemplarily, the backbone network in the first encoding module can be a multi-layer stacked Transformer, and the decoding module can only need fewer layers or 1 layer of Transformer.

[0099] Optionally, the present application can use original pixel-level regression to complete the reconstruction, the pixel value is continuous in the original space, and the masked pixel value can usually be restored by regression prediction. It should be noted that the video features learned through the visual framework are generally down-sampled. For example, when the third positive sample video features are obtained using the 3D Swin Transformer, the architecture of the 3D Swin Transformer involves down-sampling processing. In order to be able to perform regression of full pixel values, when designing the decoding module, upsampling can be combined, or the features (such as the third positive sample video features) can be linearly mapped, or non-linear mapping and other processing can be performed, so that the obtained predicted video data can be consistent with the size of the original video (i.e., the first target sample video data). For example, taking the backbone network in the first encoding module as 3D SwinTransformer, if the size of the sample video data is C*T*H*W, after the sample video data is processed in blocks (such as the block size is 2*4*4), the size of the obtained video data is C*32*(T / 2)*(H / 4)*(W / 4), then the final size of the first target sample video data is C*32*(T / 2)*(H / 4)*(W / 4), that is, the input size of the model becomes C*32*(T / 2)*(H / 4)*(W / 4); and after being processed by the 3D Swin Transformer and the decoding module, the output size of the decoding module can also be C*32*(T / 2)*(H / 4)*(W / 4).

[0100] S402, obtaining negative sample video features corresponding to historical sample video data, and pre-training a pre-trained video model based on the positive sample video features, the negative sample video features, the sample video data and the predicted video data to obtain a target video processing model.

[0101] Among them, historical sample video data may refer to video data different from sample video data, and negative sample video features may refer to video features that are not obtained using sample video data, that is, video features corresponding to historical sample video data can be called negative sample video features.

[0102] In one implementation, the negative sample video features corresponding to the historical sample video data may be video features obtained by processing the historical sample video data using a pre-training processing model during the historical pre-training process. These negative sample video features may be stored in a preset location so that they can be directly obtained when needed later. For example, the negative sample video features corresponding to the historical sample video data may be stored in a queue.

[0103] Optionally, the size (i.e., quantity) of the negative sample video features may be the size of batch-size, and batch-size may refer to the size of sample video data in a batch training. Due to the problem of GPU resources, the sufficiency of negative sample video features may not be guaranteed, so it is usually possible to consider using a queue to store a large number of negative sample video features. For example, in the process of pre-training the pre-trained video model, the second positive sample video features obtained by the second encoding module can be saved in the queue by using a memory bank. These second positive sample video features can be understood as the second positive sample video features obtained in the non-current iteration process (or understood as the training of the sample video data of the non-current batch), but the second positive sample video features obtained in the historical iteration process. Then, each time the first loss value is calculated subsequently, or when the negative sample video features are to be obtained, the second positive sample video features can be selected from the queue as the negative sample video features, such as the second positive sample video features of batch-size can be selected, so that the negative sample video features obtained from the queue can be kept similar to the video features encoded by the second encoding module. Optionally, the second positive sample video features in the queue can also be continuously updated. For example, after waiting for the batch to be trained each time, the old second positive sample video features in the queue will be popped out, and then the second positive sample video features obtained in the current batch will be added to the queue to further maintain the consistency of the negative sample video features obtained from the queue and the video features encoded by the second encoding module.

[0104] In one implementation, after the negative sample video features are obtained, the loss value of the pre-trained video model can be determined based on the positive sample video features, the negative sample video features, the sample video data, and the predicted video data, so as to use the loss value to pre-train the pre-trained video model and obtain the corresponding target video processing model. Optionally, the loss value may include a first loss value and a second loss value. For example, the first loss value of the pre-trained video model can be determined based on the positive sample video features (the first positive sample video features and the second positive sample video features) and the negative sample video features; and the second loss value of the pre-trained video model can also be determined based on the sample video data and the predicted video data. Considering that the input of the first encoding module is the first target sample video data corresponding to the sample video data, the second loss value of the pre-trained video model can be determined based on the first target sample video data and the predicted video data.

[0105] After obtaining the first loss value and the second loss value, the pre-trained video model can be pre-trained based on the first loss value and the second loss value to obtain a target video processing model. Optionally, a target loss value of the pre-trained video model can be determined based on the first loss value and the second loss value, so as to pre-train the pre-trained video model based on the target loss value to obtain the target video processing model. For example, the target loss value can be the sum of the first loss value and the second loss value.

[0106] In one implementation, contrast loss and reconstruction loss may be introduced to determine the target loss value of the pre-trained video model, wherein contrast loss may be used to calculate the first loss value, and reconstruction loss may be used to calculate the second loss value. For example, contrast loss may be used to calculate the first loss value of the pre-trained video model based on the first positive sample video feature, the second positive sample video feature, and the negative sample video feature; and reconstruction loss may be used to calculate the second loss value of the pre-trained video model based on the sample video data and the predicted video data.

[0107] In one implementation, the mathematical formula of the contrast loss can be shown as follows:

[0108]

[0109] Among them, q represents the first positive sample video feature, k + represents the second positive sample video feature, k i represents the negative sample video features, that is, the negative sample video features sampled from the queue, and K is the total number of negative sample video features.

[0110] In one implementation, the mathematical formula of the reconstruction loss can be shown as follows:

[0111] L recontruction·loss =||yx||2 (2)

[0112] Wherein, y represents the predicted video data, and x represents the first target sample video data.

[0113] Based on the above description, it can be known that in this application, the pre-training method of the pre-trained video model can be understood as a combination of a self-supervised pre-training method based on contrastive learning and a self-supervised pre-training method based on generative learning. Among them, the self-supervised pre-training method based on contrastive learning involves the calculation of the first loss value, and the pre-training process involves the related processing of the first encoding module and the second encoding module in the pre-trained video model; the self-supervised pre-training method based on generative learning involves the calculation of the second loss value, and the pre-training process involves the related processing of the first encoding module and the decoding module in the pre-trained video model.

[0114] (1) In the self-supervised pre-training method based on contrastive learning, positive and negative sample pairs can be defined to maximize the distance between positive sample pairs and minimize the distance between negative sample pairs in the representation space, thereby achieving the effect of mutual attraction between similar samples and mutual repulsion between different samples. In the absence of labeled information, the self-supervised pre-training method based on contrastive learning can construct positive sample pairs based on the same sample, such as constructing positive sample pairs by performing different data augmentations (such as strong data augmentation or weak data augmentation) on the same sample, and constructing negative sample pairs based on different samples, thereby learning video representation. Among them, the positive sample pair can be constructed by the output of the first encoding module and the output of the second encoding module, that is, the first positive sample video feature and the second positive sample video feature can constitute the positive sample pair; and the negative sample pair can be constructed by the second positive sample video feature and the negative sample video feature.

[0115] At the same time, the present application can mask a certain proportion of the video level information, and then map the global and local to the representation space, and supervise through contrast loss to minimize the distance between positive sample pairs and maximize the distance between negative sample pairs. Based on the above description, it can be seen that the contrast loss in the present application can also be called local-global contrast loss.

[0116] (2) In the self-supervised pre-training method based on generative learning, the main thing is to reconstruct the input so that the input and output are as similar as possible. Generative learning can define a codec structure (such as the first encoding module and decoding module in this application), mask out some frames or blocks in the video, and then reconstruct the original video data through the first encoding module and decoding module, so that the pre-trained video model can predict the masked video data. In this way, the video data itself can be used as supervision without the need for complex manual annotation.

[0117] At the same time, the present application can reconstruct local information through a codec structure (such as the first encoding module and decoding module in the present application), and then supervise through pixel-level reconstruction loss to achieve pre-training of the model by minimizing pixel-level reconstruction loss.

[0118] The present application can perform self-supervision simultaneously through two pre-training methods to improve the stability of model training, further improve model performance, and thus improve the model's ability to model videos.

[0119] In one implementation, there is a difference in the parameter update method of the first encoding module and the second encoding module, wherein the first encoding module can use the back propagation method of stochastic gradient descent (SGD) to update each parameter in the first encoding module, and the second encoding module can use the momentum update method to update each parameter in the second encoding module.

[0120] S403: Obtain training video data and annotation information corresponding to the training video data.

[0121] In one implementation, after the pre-trained video model is pre-trained and the corresponding target video processing model is obtained, the downstream task is fine-tuned. For example, the downstream task can be a video processing task such as multi-label recognition of videos, video classification, video retrieval, low-quality and high-quality video recognition. In order to complete the fine-tuning of the downstream task, the target video processing model can be fine-tuned using the training video data with labeled information to obtain the fine-tuned target video processing model. The fine-tuned target video processing model can then process the downstream task.

[0122] In one implementation, training video data and the annotation information corresponding to the training video data can be obtained, so that the target video processing model can be fine-tuned based on the training video data and the corresponding annotation information. Among them, for different downstream tasks, the corresponding training video data and the corresponding annotation information may be different. For example, if the downstream task is multi-label recognition of videos, the training video data can be a video, wherein the video has a corresponding label, and the label is the annotation information of the training video data, such as a video label can be one of a game video, a variety show video, a sports video, and a film and television video. For another example, if the downstream task is the recognition of low-quality and high-quality videos, the training video data can include a video, wherein the video has a corresponding label that can be one of low-quality and high-quality.

[0123] It should be noted that the amount of training video data used to fine-tune the target video processing model can be large, and this application is mainly explained using one training video data as an example.

[0124] S404: Train the target video processing model based on the training video data and the annotation information corresponding to the training video data to obtain a trained target video processing model.

[0125] In one implementation, the target video processing model can be obtained based on the pre-trained first encoding module. For example, the target video processing model can be composed of the backbone network in the pre-trained first encoding module. The backbone network can serve as a feature extraction module. Optionally, in the fine-tuning of the target video processing model, the target video processing model may include other modules in addition to the backbone network to form a complete video processing model that can be used to process downstream tasks. For example, in a video classification task, the target video processing model may also include a classification module. For another example, in a video recognition task, the target video processing model may also include a recognition module.

[0126] In one implementation, the target video model can be trained using training video data to obtain predicted annotation information of the training video data, and the target video processing model can be trained based on the annotation information and the predicted annotation information to obtain a trained target video model. Optionally, a model loss function corresponding to the target video processing model can be obtained, and the model loss value of the target video processing model can be calculated using the model loss function and the annotation information and the predicted annotation information; and after obtaining the model loss value, the target video processing model can be trained based on the model loss value to obtain a trained target video processing model. For example, the model parameters of the target video processing model can be optimized in the direction of reducing the model loss value. Among them, the model loss function can be a cross entropy loss function or other loss function, which is not specifically limited in this application.

[0127] From the above, it can be seen that the present application can provide an adaptive pre-trained model (i.e., a pre-trained video model) for downstream tasks. The pre-trained model has a strong generalization ability. On the one hand, it can improve the model performance, and on the other hand, it can also reduce the number of iterations and training data in the fine-tuning of downstream tasks. For example, in the current major video applications, a large amount of video data (such as news applications) can be poured in every day. The multi-label recognition task of the downstream task needs to be identified and recommended for all videos. When a large amount of unlabeled data pours into the application, it is very labor-intensive and financially intensive to label a noise-free and sizable data set. The pre-trained video model in the present application can use a small amount of labeled data to enable the model to present better results on the newly added data, while continuously supporting newly added labels. By utilizing the pre-trained video model, the downstream tasks can converge faster while improving the downstream recognition performance, and it is also conducive to faster adaptation to the newly added labels, so that a small amount of newly added labeled data can be used to conveniently support the output of the newly added labels.

[0128] From the above description, it can be known that the above steps S401-S404 can be a training process for a pre-trained video model, wherein the training process can specifically include the process of pre-training the pre-trained video model involved in steps S401-S402 to obtain a target video processing model, and the process of fine-tuning the target video processing model involved in steps S402-S404 to obtain a fine-tuned target video processing model. Optionally, in other embodiments, the following step S405 can also be included after step S404, and this step is mainly the actual application scenario of the trained target video processing model.

[0129] S405, obtaining a video to be processed, and calling a trained target video processing model to perform video processing on the video to be processed, to obtain video features of the video to be processed.

[0130] In one implementation, after obtaining the trained target video processing model, the trained target video processing model can be applied in an actual video processing scenario. For example, if the downstream task corresponding to fine-tuning is multi-label recognition, the trained target video processing model can be applied in multi-label recognition. Optionally, a video to be processed can be obtained, and the trained target video processing model can be called to perform video processing on the video to be processed to obtain the video features of the video to be processed. Further, multi-label recognition of the video to be processed can be implemented based on the video features. In a possible implementation, the target video processing model can include a recognition module, and the recognition module can be called to perform recognition processing on the video features to obtain the label of the video to be processed.

[0131] In one implementation, in order to prove the model effect (such as multi-label recognition effect) of the target video processing model obtained by the video processing method provided in the present application, the target video processing model can also be tested. For example, the multi-label recognition effect of the target video processing model obtained in the present application can be compared with other video processing models, such as a comparative analysis of the multi-label recognition effect. Among them, in the comparative analysis, there is a difference in the training method of the target video processing model in the present application and the compared video processing model. The target video processing model may refer to the one obtained by the pre-training method involved in the present application and combined with the fine-tuning method, and the compared video processing model may refer to the model obtained without pre-training. The data shown in Table 1 may be the multi-label recognition effect of the video processing model obtained in different training methods.

[0132] Table 1:

[0133] Model Precision Recall Not using pre-trained models 70.57％ 37.8％ Models in this application 70.29％ 42.0％

[0134] Among them, the model effect of the model can be characterized by precision and recall. A higher precision value indicates a better model effect, and a higher recall value indicates a better model effect. From the data in Table 1, it can be seen that when the precision is similar, the target video processing model in this application has a higher recall rate, and the target video processing model in this application has a better model effect than the video processing model obtained without pre-training.

[0135] To better understand the video processing method provided in the embodiment of the present application, Figure 5e The pre-training process of the pre-trained video model shown in FIG. Figure 5e As shown, the process shown in the figure can be a pre-training process that combines a self-supervised pre-training method based on contrastive learning and a self-supervised pre-training method based on generative learning proposed in this application. Figure 5e It can be seen that the design of the video-based self-supervised pre-training framework is mainly divided into the design of two modules, namely the design of video reconstruction and video feature comparison learning. Among them, the pre-trained video model can be divided into three main modules: the first encoding module (which can be called Encoder), the second encoding module (which can be called Momentum encoder) and the decoding module (which can be called Decoder).

[0136] like Figure 5e In the framework shown, for a sample video data, the sample video data can be preprocessed first, such as preprocessing can include frame extraction, block processing and data enhancement data. Specifically, T frames can be randomly extracted from the sample video data through a multi-scale temporal input module to obtain initial sample video data, such as the initial sample video data can be x∈C*T*H*W, where C is the number of channels (such as 1, 3, etc.), T is the time length (such as T∈{4,8,16,32}, or can also be understood as the number of video frames), and H and W are the width and height of the image respectively. After obtaining the initial sample video data x, the initial sample video data x can also be processed by block processing to obtain sample video block data. For example, the block size can be: 2*4*4, then x∈C*T*H*W can be processed by block processing to obtain sample video block data x1∈C*32*(T / 2)*(H / 4)*(W / 4).

[0137] Furthermore, data enhancement processing may be performed on the divided video data (ie, sample video divided data) to obtain the first target sample video data (x q ) and the second target sample video data (x k). Then, x q and x k Through the Encoder and Momentum encoder, they are mapped to the high-dimensional space to obtain the first positive sample video feature (q) and the second positive sample video feature (k + ). At the same time, a larger memory bank can be used to store negative sample video features. For example, the negative sample video features can be stored in a queue, so that when the negative sample video features need to be obtained to calculate the first loss value, they can be directly obtained from the queue.

[0138] Thus, the first positive sample video feature (q) and the second positive sample video feature (k + ) and the negative sample video features in the queue are used to calculate the contrast loss, that is, the first loss value. For example, the above formula (1) can be used to calculate the contrast loss.

[0139] In order to achieve pixel-level self-supervision, the features of the Backbone output in the Encoder can be used to reconstruct the original video pixels (i.e., predicted video data) through the Decoder, and then the reconstruction loss (or L2 loss) can be used for self-supervision, that is, the second loss value can be calculated. For example, the above formula (2) can be used to calculate the reconstruction loss.

[0140] The following is a detailed introduction to Encoder, Momentum encoder, and Decoder.

[0141] For Encoder, Encoder is the main part of the self-supervised pre-training framework. The pre-trained parameters of this part can be used as the initialization parameters of downstream tasks. Figure 5e As shown in the figure, the Encoder can include four modules: Mask block, Backbone, Projector module, and Predictor module. The functions and detailed implementation methods of each module will be introduced in turn below:

[0142] (1) Mask block: A certain proportion of pixel information of the input video data can be masked. For example, mask block and mask frame can be used for masking. The specific implementation can refer to the above description.

[0143] (2) Backbone, this module is the basic network used by the pre-trained video model, which can usually use CNN network and Transformer network. Due to the excellence of Transformer network in the field of vision, this application can use 3DSwin Transformer as the basic network for self-supervised pre-training.

[0144] (3) The Projector module and the Predictor module can both be two-layer MLPs, and batch normalization and activation functions are not used. The Predictor module can maintain the asymmetric structure between the Encoder and the Momentum encoder.

[0145] For Momentum encoder, such as Figure 5e As shown in the figure, the Momentum encoder can include two modules: Backbone and Projector. The Backbone can have the same structure as the Backbone in the Encoder network, such as 3D Swin Transformer; the Projector module can also have the same structure as the Projector module in the Encoder network, such as two layers of MLP.

[0146] There are differences in the parameter updating methods of Momentum encoder and Encoder. Encoder uses the back propagation update method of Stochastic Gradient Descent (SGD) to update parameters, while Momentum encoder uses the momentum update method to update parameters.

[0147] For the decoder, the input of the decoder is the video features extracted from the encoder (such as the third positive sample video feature mentioned above). The purpose of the encoder is to perform pixel-level reconstruction tasks. Since the encoder is the one that actually learns the latent feature representation, the decoder has fewer channels and a shallower network depth than the encoder. The decoder can be any form of network as long as its output is consistent with the encoder input and can complete the pixel reconstruction task.

[0148] In one implementation, the self-supervised pre-training method of the model framework of the pre-trained video model of the present application can be replaced by other self-supervised methods, for example, it can be MOCO, MOCOv2, BYOL and other self-supervised methods.

[0149] By implementing the above method, the model can be pre-trained using a self-supervised pre-training method based on generative learning and contrastive learning, which can improve the stability of model training to improve model performance, thereby improving the model's ability to process videos; self-supervised pre-training based on contrastive learning focuses more on semantic-level features, while self-supervised pre-training based on generative learning focuses more on pixel-level features. The self-supervised pre-training framework based on generative learning and contrastive learning can better learn video information from the semantic and pixel levels, which helps to improve video understanding and the generalization ability of the model. Furthermore, when the target video processing model obtained by pre-training is used to perform video processing tasks such as video recognition, video classification, and video retrieval, the accuracy of video processing can also be improved.

[0150] See also Figure 6 , is a schematic diagram of the structure of a video processing device provided in an embodiment of the present application. The video processing device described in this embodiment includes:

[0151] The acquisition unit 601 is used to acquire positive sample video features corresponding to sample video data, and acquire predicted video data corresponding to the sample video data; the predicted video data is video data obtained by reconstructing the sample video data;

[0152] A pre-training unit 602 is used to obtain negative sample video features corresponding to historical sample video data, and pre-train a pre-trained video model based on the positive sample video features, the negative sample video features, the sample video data and the predicted video data to obtain a target video processing model;

[0153] The processing unit 603 is used to obtain the video to be processed, and call the target video processing model to perform video processing on the video to be processed to obtain the video features of the video to be processed.

[0154] In one implementation, the pre-training unit 602 is specifically configured to:

[0155] Determining a first loss value of the pre-trained video model based on the positive sample video feature and the negative sample video feature;

[0156] Determining a second loss value of the pre-trained video model based on the sample video data and the predicted video data;

[0157] Determining a target loss value of the pre-trained video model based on the first loss value and the second loss value;

[0158] The pre-trained video model is pre-trained based on the target loss value to obtain a target video processing model.

[0159] In one implementation, the positive sample video feature includes a first positive sample video feature and a second positive sample video feature; the acquisition unit 601 is specifically configured to:

[0160] Preprocessing the sample video data to obtain first target sample video data and second target sample video data of the sample video data;

[0161] Performing encoding processing on the first target sample video data to obtain a first positive sample video feature of the sample video data;

[0162] The second target sample video data is encoded to obtain a second positive sample video feature of the sample video data.

[0163] In one implementation, the acquisition unit 601 is specifically configured to:

[0164] Performing mask processing on the first target sample video data to obtain mask video features of the sample video data;

[0165] Feature extraction is performed on the mask video feature to obtain a first positive sample video feature of the sample video data.

[0166] In one implementation, the acquisition unit 601 is specifically configured to:

[0167] Performing frame extraction processing on the sample video data to obtain initial sample video data, wherein the initial sample video data is synthesized based on one or more video frames;

[0168] Performing block processing on the initial sample video data to obtain sample video block data;

[0169] A first data enhancement process is performed on the sample video block data to obtain first target sample video data, and a second data enhancement process is performed on the sample video block data to obtain second target sample video data.

[0170] In one implementation, the acquisition unit 601 is specifically configured to:

[0171] Obtaining mask video features of the sample video data;

[0172] Performing feature extraction on the mask video feature to obtain a third positive sample video feature of the sample video data;

[0173] The sample video data is predicted based on the third positive sample video feature to obtain predicted video data of the sample video data.

[0174] In one implementation, the apparatus further includes a training unit 604, specifically configured to:

[0175] Acquire training video data and annotation information corresponding to the training video data;

[0176] Training the target video processing model based on the training video data and the annotation information corresponding to the training video data to obtain a trained target video processing model;

[0177] The processing unit 603 is further used to call the trained target video processing model to perform video processing on the video to be processed to obtain video features of the video to be processed.

[0178] It is understood that the division of units in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. The functional units in the embodiments of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software functional units.

[0179] See also Figure 7 , Figure 7 701 is a schematic diagram of a computer device provided in an embodiment of the present application. The computer device described in this embodiment may be a terminal or a server, and the computer device includes: a processor 701 and a memory 702. Optionally, the computer device may also include a network interface 703. The processor 701, the memory 702 and the network interface 703 may exchange data.

[0180] The processor 701 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0181] The memory 702 may include a read-only memory and a random access memory, and provide program instructions and data to the processor 701. A portion of the memory 702 may also include a non-volatile random access memory. When the processor 701 calls the program instructions, it is used to execute:

[0182] Acquire positive sample video features corresponding to sample video data, and acquire predicted video data corresponding to the sample video data; the predicted video data is video data obtained by reconstructing the sample video data;

[0183] Obtaining negative sample video features corresponding to historical sample video data, and pre-training a pre-trained video model based on the positive sample video features, the negative sample video features, the sample video data, and the predicted video data to obtain a target video processing model;

[0184] The video to be processed is obtained, and the target video processing model is called to perform video processing on the video to be processed to obtain video features of the video to be processed.

[0185] In one implementation, the processor 701 is specifically configured to:

[0186] Determining a first loss value of the pre-trained video model based on the positive sample video feature and the negative sample video feature;

[0187] Determining a second loss value of the pre-trained video model based on the sample video data and the predicted video data;

[0188] Determining a target loss value of the pre-trained video model based on the first loss value and the second loss value;

[0189] The pre-trained video model is pre-trained based on the target loss value to obtain a target video processing model.

[0190] In one implementation, the positive sample video feature includes a first positive sample video feature and a second positive sample video feature; the processor 701 is specifically configured to:

[0191] Preprocessing the sample video data to obtain first target sample video data and second target sample video data of the sample video data;

[0192] Performing encoding processing on the first target sample video data to obtain a first positive sample video feature of the sample video data;

[0193] The second target sample video data is encoded to obtain a second positive sample video feature of the sample video data.

[0194] In one implementation, the processor 701 is specifically configured to:

[0195] Performing mask processing on the first target sample video data to obtain mask video features of the sample video data;

[0196] Feature extraction is performed on the mask video feature to obtain a first positive sample video feature of the sample video data.

[0197] In one implementation, the processor 701 is specifically configured to:

[0198] Performing frame extraction processing on the sample video data to obtain initial sample video data, wherein the initial sample video data is synthesized based on one or more video frames;

[0199] Performing block processing on the initial sample video data to obtain sample video block data;

[0200] A first data enhancement process is performed on the sample video block data to obtain first target sample video data, and a second data enhancement process is performed on the sample video block data to obtain second target sample video data.

[0201] In one implementation, the processor 701 is specifically configured to:

[0202] Obtaining mask video features of the sample video data;

[0203] Performing feature extraction on the mask video feature to obtain a third positive sample video feature of the sample video data;

[0204] The sample video data is predicted based on the third positive sample video feature to obtain predicted video data of the sample video data.

[0205] In one implementation, the processor 701 is further configured to:

[0206] Acquire training video data and annotation information corresponding to the training video data;

[0207] Training the target video processing model based on the training video data and the annotation information corresponding to the training video data to obtain a trained target video processing model;

[0208] The trained target video processing model is called to perform video processing on the video to be processed to obtain video features of the video to be processed.

[0209] The present application also provides a computer storage medium in which program instructions are stored. When the program is executed, it may include: Figure 2 or Figure 4 Part or all of the steps of the video processing method in the corresponding embodiment.

[0210] It should be noted that, for the above-mentioned various method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0211] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0212] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction, and when the computer instruction is executed by a processor, some or all of the steps in the above method can be implemented. For example, the computer instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device performs the steps performed in the embodiments of the above methods.

[0213] The above is a detailed introduction to a video processing method, device, computer equipment, medium and program product provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A video processing method, characterized in that: include: Obtaining positive sample video features corresponding to the sample video data, and obtaining predicted video data corresponding to the sample video data; The predicted video data is video data obtained by reconstructing the sample video data; Obtain negative sample video features corresponding to historical sample video data, and determine a first loss value of a pre-trained video model based on the positive sample video features and the negative sample video features, calculate the square value of the difference between the pixel values ​​at each pixel point in the predicted video data and the first target sample video data in the sample video data, calculate the average value of the square value of the difference between the pixel values ​​at each pixel point, use the average value as the second loss value of the pre-trained video model, determine the target loss value of the pre-trained video model based on the first loss value and the second loss value, pre-train the pre-trained video model based on the target loss value to obtain a target video processing model; the positive sample video features include a first positive sample video feature and a second positive sample video feature, the pre-trained video model includes a first encoding module, a second encoding module and a decoding module, the first encoding module and the second encoding module include a projection module, the first encoding module is used to perform mask processing on the first target sample video data to obtain mask video features of the sample video data, and perform feature extraction on the mask video features to obtain first positive sample video features and third positive sample video features of the sample video data; The second encoding module is used to encode the second target sample video data to obtain a second positive sample video feature of the sample video data by predicting the covered video data; The decoding module is used to decode the third positive sample video feature to obtain predicted video data of the sample video data; The first target sample video data and the second target sample video data are obtained by preprocessing the sample video data, the first positive sample video feature, the second positive sample video feature and the negative sample video feature are used for pre-training based on contrastive learning, and the sample video data and the predicted video data are used for pre-training based on generative learning; the projection module is used to perform dimensionality reduction processing on the first positive sample video feature, the second positive sample video feature and the third positive sample video feature; wherein the parameter update methods of the first encoding module and the second encoding module are different, the first encoding module uses the back propagation method of stochastic gradient descent to update the parameters, and the second encoding module uses the momentum update method to update the parameters; The video to be processed is obtained, and the target video processing model is called to perform video processing on the video to be processed to obtain video features of the video to be processed.

2. The method according to claim 1, characterized in that: The method further comprises: Performing frame extraction processing on the sample video data to obtain initial sample video data, wherein the initial sample video data is synthesized based on one or more video frames; Performing block processing on the initial sample video data to obtain sample video block data; A first data enhancement process is performed on the sample video block data to obtain first target sample video data, and a second data enhancement process is performed on the sample video block data to obtain second target sample video data.

3. The method according to claim 1, characterized in that Acquiring predicted video data corresponding to the sample video data, including: Obtaining mask video features of the sample video data; Performing feature extraction on the mask video feature to obtain a third positive sample video feature of the sample video data; The sample video data is predicted based on the third positive sample video feature to obtain predicted video data of the sample video data.

4. The method according to any one of claims 1 to 3, characterized in that: Also includes: Acquire training video data and annotation information corresponding to the training video data; Training the target video processing model based on the training video data and the annotation information corresponding to the training video data to obtain a trained target video processing model; The calling of the target video processing model to process the video to be processed to obtain the video features of the video to be processed includes: The trained target video processing model is called to perform video processing on the video to be processed to obtain video features of the video to be processed.

5. A video processing device, characterized in that: include: An acquisition unit, used to acquire positive sample video features corresponding to the sample video data, and acquire predicted video data corresponding to the sample video data; The predicted video data is video data obtained by reconstructing the sample video data; A pre-training unit, used to obtain negative sample video features corresponding to historical sample video data, and determine a first loss value of a pre-trained video model based on the positive sample video features and the negative sample video features, calculate the square value of the difference between the pixel values ​​at each pixel point in the predicted video data and the first target sample video data in the sample video data, calculate the average value of the square value of the difference between the pixel values ​​at each pixel point, use the average value as a second loss value of the pre-trained video model, determine a target loss value of the pre-trained video model based on the first loss value and the second loss value, and pre-train the pre-trained video model based on the target loss value to obtain a target video processing model; the positive sample video features include a first positive sample video feature and a second positive sample video feature, the pre-trained video model includes a first encoding module, a second encoding module and a decoding module, the first encoding module and the second encoding module include a projection module, the first encoding module is used to perform mask processing on the first target sample video data to obtain mask video features of the sample video data, and perform feature extraction on the mask video features to obtain first positive sample video features and third positive sample video features of the sample video data; The second encoding module is used to encode the second target sample video data to obtain a second positive sample video feature of the sample video data; The decoding module is used to decode the third positive sample video feature to obtain predicted video data of the sample video data by predicting the covered video data; The first target sample video data and the second target sample video data are obtained by preprocessing the sample video data; The first positive sample video feature, the second positive sample video feature and the negative sample video feature are used for pre-training based on contrastive learning, and the sample video data and the predicted video data are used for pre-training based on generative learning; the projection module is used to perform dimensionality reduction processing on the first positive sample video feature, the second positive sample video feature and the third positive sample video feature; wherein the parameter update methods of the first encoding module and the second encoding module are different, the first encoding module uses the back propagation method of stochastic gradient descent to update the parameters, and the second encoding module uses the momentum update method to update the parameters; The processing unit is used to obtain the video to be processed, and call the target video processing model to perform video processing on the video to be processed to obtain the video features of the video to be processed.

6. The device according to claim 5, characterized in that The acquisition unit is also used to perform frame extraction processing on the sample video data to obtain initial sample video data, where the initial sample video data is synthesized based on one or more video frames, perform block processing on the initial sample video data to obtain sample video block data, perform first data enhancement processing on the sample video block data to obtain first target sample video data, and perform second data enhancement processing on the sample video block data to obtain second target sample video data.

7. The device according to claim 5, characterized in that The acquisition unit is also used to acquire mask video features of the sample video data, perform feature extraction on the mask video features to obtain third positive sample video features of the sample video data, and perform prediction processing on the sample video data based on the third positive sample video features to obtain predicted video data of the sample video data.

8. The device according to any one of claims 5 to 7, characterized in that: The device also includes a training unit, which is used to obtain training video data and annotation information corresponding to the training video data, and train the target video processing model based on the training video data and the annotation information corresponding to the training video data to obtain the trained target video processing model; the processing unit is also used to call the trained target video processing model to perform video processing on the video to be processed to obtain the video features of the video to be processed.

9. A computer device, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 4.

11. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images