Video feature extraction model training method and device, equipment and medium

Abstract

The present application relates to artificial intelligence technology, and discloses a video feature extraction model training method, comprising: dividing a video sample into frames to obtain a first video frame set, performing video frame masking on the first video frame set to obtain a second video frame set, performing image block masking on image blocks of video frames that have not been subjected to video frame masking processing to obtain a third video frame set, performing video frame prediction training on a pre-constructed feature extraction model using the second video frame set and the third video frame set, calculating a loss value between a video frame prediction result and the first video frame set, adjusting parameters of the feature extraction model according to the loss value, and obtaining a trained video feature extraction model when the adjusted feature extraction model meets a preset model training condition. The present application also proposes a video feature extraction model training device, equipment and medium. The present application can improve the feature extraction capability of the video feature extraction model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a video feature extraction model training method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Video feature extraction is a fundamental technique in video processing. It extracts a series of features from the video to be processed to meet various video processing needs of users. For example, based on video feature extraction, different types of video processing needs such as video classification, video frame interpolation, video segmentation, and video description can be achieved.

[0003] The most common video feature extraction method currently is to use a large number of labeled video samples to train a deep learning-based neural network model in a supervised manner, and then use the trained model to extract features from the video to be processed.

[0004] This approach relies on high-quality labeled data, but supervised learning based on labeled data may lead to overfitting of the model, resulting in a weak actual feature extraction capability. Summary of the Invention

[0005] This invention provides a video feature extraction model training method, apparatus, electronic device, and computer-readable storage medium, the main purpose of which is to improve the feature extraction capability of the video feature extraction model.

[0006] To achieve the above objectives, the present invention provides a video feature extraction model training method, comprising:

[0007] Step A: Obtain video samples and perform frame segmentation on the video samples to obtain the first set of video frames;

[0008] Step B: Perform video frame masking processing on the first video frame set at a first preset ratio, and then collect the video frames after video frame masking processing into a second video frame set.

[0009] Step C: Perform image block occlusion processing on the image blocks of each video frame in the first video frame set that has not undergone video frame occlusion processing with a second preset ratio to obtain a third video frame set;

[0010] Step D: Use the second video frame set and the third video frame set to train the pre-constructed feature extraction model for video frame prediction, and obtain the video frame prediction results;

[0011] Step F: Calculate the loss value between the video frame prediction result and the first video frame set, adjust the parameters of the feature extraction model according to the loss value, and return to step D above. When the adjusted feature extraction model meets the preset model training conditions, exit the prediction training and obtain the trained video feature extraction model.

[0012] Optionally, the step of performing video frame masking processing on the first video frame set at a first preset ratio, and then aggregating the masked video frames into a second video frame set, includes:

[0013] Each video frame in the first video frame set is numbered sequentially;

[0014] Randomly select a number from the range of the numbers, representing the first preset proportion, as the target number to be obscured;

[0015] The image pixels in the video frame corresponding to each of the occluded target numbers are reset to the first preset pixel value;

[0016] The second video frame set is obtained by collecting all the video frames after pixel reset.

[0017] Optionally, the step of performing a second preset ratio of image block occlusion processing on image blocks of each video frame in the first video frame set that has not undergone video frame occlusion processing to obtain a third video frame set includes:

[0018] The image corresponding to each video frame in the first video frame set that has not undergone video frame masking is divided into an equal number of image blocks to form an image block matrix.

[0019] Each image patch matrix is ​​generated according to a preset random algorithm, and the occlusion range conforms to the second preset ratio.

[0020] The pixels within the occlusion range in each of the image block matrices are reset to the second preset pixel value;

[0021] The third video frame set is composed of video frames whose image block pixels have been reset.

[0022] Optionally, the step of training the pre-constructed feature extraction model for video frame prediction using the second video frame set and the third video frame set to obtain video frame prediction results includes:

[0023] Using the second set of video frames and the third set of video frames as input, the input is windowed and vectorized using the encoding layer of the pre-built feature extraction model to obtain the window vector matrix corresponding to each window;

[0024] The window attention mechanism of the pre-constructed feature extraction model is used to extract features from each window vector matrix to obtain the feature vector corresponding to each window vector matrix;

[0025] Using the window interaction mechanism of the pre-built feature extraction model, the feature vectors corresponding to each window vector are fused and calculated to obtain a global feature vector;

[0026] Using the decoding layer of the pre-built feature extraction model, a video frame prediction result corresponding to the input is generated based on the global feature vector.

[0027] Optionally, calculating the loss value between the video frame prediction result and the first video frame set includes:

[0028] The loss value between the video frame prediction result and the first video frame set is calculated using the following preset cross-entropy loss function:

[0029]

[0030] Where L(x,y) represents the loss value between the true result x and the predicted result y of the first video frame set, N represents the total number of video frames in the video samples, and l n x represents the difference between the nth video frame in the first video frame set and the nth video frame in the video frame prediction result. n y represents the vector value of the nth video frame in the first set of video frames. n represents the vector value of the nth video frame in the video frame prediction result, and t represents the number of iterations in the training process of the pre-constructed feature extraction model.

[0031] Optionally, the step of performing frame segmentation on the video samples to obtain a first set of video frames includes:

[0032] All frames corresponding to each video sample are averaged and grouped.

[0033] Randomly select an equal number of frames from each corresponding set of frames as the target frames;

[0034] All target frames are combined to form the first video frame set.

[0035] To address the above problems, the present invention also provides a video feature extraction model training device, the device comprising:

[0036] The sample frame segmentation processing module is used to acquire video samples, perform frame segmentation processing on the video samples, and obtain a first set of video frames.

[0037] The video frame masking processing module is used to perform video frame masking processing on the first video frame set at a first preset ratio, and to collect the video frames after video frame masking processing into a second video frame set.

[0038] The image masking processing module is used to perform image block masking processing on the image blocks of each video frame in the first video frame set that has not undergone video frame masking processing at a second preset ratio, so as to obtain a third video frame set.

[0039] The video frame prediction training module is used to train a pre-constructed feature extraction model for video frame prediction using the second video frame set and the third video frame set, obtain video frame prediction results, calculate the loss value between the video frame prediction results and the first video frame set, adjust the parameters of the feature extraction model according to the loss value, and exit the prediction training when the adjusted feature extraction model meets the preset model training conditions, thereby obtaining the trained video feature extraction model.

[0040] Optionally, the video frame prediction training module obtains the video frame prediction result through the following operations:

[0041] Using the second set of video frames and the third set of video frames as input, the input is windowed and vectorized using the encoding layer of the pre-built feature extraction model to obtain the window vector matrix corresponding to each window;

[0042] The window attention mechanism of the pre-constructed feature extraction model is used to extract features from each window vector matrix to obtain the feature vector corresponding to each window vector matrix;

[0043] Using the window interaction mechanism of the pre-built feature extraction model, the feature vectors corresponding to each window vector are fused and calculated to obtain a global feature vector;

[0044] Using the decoding layer of the pre-built feature extraction model, a video frame prediction result corresponding to the input is generated based on the global feature vector.

[0045] To address the above problems, the present invention also provides an electronic device, the electronic device comprising:

[0046] Memory, storing at least one computer program; and

[0047] The processor executes the program stored in the memory to implement the video feature extraction model training method described above.

[0048] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the video feature extraction model training method described above.

[0049] The embodiments of the present invention not only perform video frame occlusion processing on video frames in video samples at a first preset ratio, but also perform image block occlusion processing on image blocks in unoccluded video frames at a second preset ratio. This enables the pre-built feature extraction model to learn not only the contextual information between video frames, but also the contextual information within the same video frame during the training process of video frame prediction using the pre-built feature extraction model, thereby improving the feature extraction capability of the video feature extraction model. Attached Figure Description

[0050] Figure 1 This is a flowchart illustrating a video feature extraction model training method according to an embodiment of the present invention.

[0051] Figure 2 This is a detailed implementation flowchart of one step in the video feature extraction model training method provided in an embodiment of the present invention;

[0052] Figure 3 This is a detailed implementation flowchart of one step in the video feature extraction model training method provided in an embodiment of the present invention;

[0053] Figure 4 This is a detailed implementation flowchart of one step in the video feature extraction model training method provided in an embodiment of the present invention;

[0054] Figure 5 This is a functional block diagram of a video feature extraction model training device provided in an embodiment of the present invention;

[0055] Figure 6 This is a schematic diagram of the structure of an electronic device for implementing the video feature extraction model training method according to an embodiment of the present invention.

[0056] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0057] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0058] This application provides a method for training a video feature extraction model. The execution entity of the video feature extraction model training method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the video feature extraction model training method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0059] Reference Figure 1 The diagram shown is a flowchart illustrating a video feature extraction model training method according to an embodiment of the present invention. In this embodiment, the video feature extraction model training method includes:

[0060] Step A: Obtain video samples and perform frame segmentation on the video samples to obtain the first set of video frames;

[0061] In this embodiment of the invention, the video sample can be obtained from a designated video platform; preferably, the video sample is a short video.

[0062] In this embodiment of the invention, the video sample can be segmented into frames using a preset frame-segmentation script. The preset frame-segmentation script can be a video frame-segmentation program edited using languages ​​such as Python, Ruby, and MATLAB.

[0063] Understandably, a short video typically ranges in length from a few seconds to several minutes and contains hundreds to tens of thousands of video frames. In order to improve the processing efficiency of video samples and control the number of video frames, this solution can perform frame-segmentation processing on the video samples to obtain video frames, and further adopt a strategy of uniformly extracting an equal number of video frames to obtain the first set of video frames.

[0064] In detail, the step of performing frame segmentation on the video samples to obtain a first video frame set includes: average grouping all frames corresponding to each video sample; randomly selecting an equal number of frames from each corresponding frame set as target frames; and combining all target frames to form the first video frame set.

[0065] In this embodiment of the invention, the video samples can be averaged according to a preset grouping rule. The preset grouping rule can be averaged based on the number of video frames or averaged based on the duration of video frames.

[0066] In this embodiment of the invention, the equal number of frames can be 8 frames or 4 frames, etc. In practical applications, the number of target frames to be acquired can be determined by combining the number of frames in the video sample.

[0067] In another optional embodiment of the present invention, a small number of video samples with a longer duration can be obtained, and each video sample can be processed by frame segmentation to obtain multiple video frames corresponding to each video sample. All video frames of each video sample can be collected to obtain the first video frame set.

[0068] Step B: Perform video frame masking processing on the first video frame set at a first preset ratio, and then collect the video frames after video frame masking processing into a second video frame set.

[0069] In this embodiment of the invention, the video frame masking process can be based on a certain algorithm to select the video frame to be masked, or it can be based on a random method to select the video frame to be masked.

[0070] In this embodiment of the invention, the first preset ratio can be 40% or 30%, that is, 40% or 30% of the video frames in the first video frame set are masked. In practical applications, the proportion of video frames to be masked can be estimated based on the actual number of video frame sets.

[0071] For details, please refer to Figure 2 As shown, the step of performing video frame masking processing on the first video frame set at a first preset ratio, and then aggregating the masked video frames into a second video frame set, includes:

[0072] S21. Number each video frame in the first video frame set in sequence;

[0073] S22. Randomly select a number from the range of the numbers as the number of the first preset ratio as the number of the occlusion target;

[0074] S23. Reset the image pixels in the video frame corresponding to each of the masking target numbers to the first preset pixel value;

[0075] S24. Gather all the video frames after pixel reset to obtain the second video frame set.

[0076] In this embodiment of the invention, a preset random number generator can be used to randomly select target numbers to be masked from the range of the numbers.

[0077] In an optional embodiment of the present invention, the first preset pixel value can be 0 or 255 to achieve masking of the corresponding video frame, that is, based on the principle of image binarization processing, to achieve masking of the video frame.

[0078] It is understood that, in the embodiments of the present invention, the video frame masking process includes, but is not limited to, assigning values ​​to the pixels of the video frame image, and may also employ other methods, such as replacing the image in the original video frame with a uniform image or symbol element.

[0079] Step C: Perform image block occlusion processing on the image blocks of each video frame in the first video frame set that has not undergone video frame occlusion processing with a second preset ratio to obtain a third video frame set;

[0080] It is understood that the image corresponding to each video frame in the first video frame set can be divided into multiple image blocks, and the second preset ratio refers to the proportion of the image blocks that need to be occluded in each video frame to the total number of image blocks in the corresponding video frame, for example, 75%.

[0081] For details, please refer to Figure 3 As shown, the step of performing image block occlusion processing on image blocks of each video frame in the first video frame set that has not undergone video frame occlusion processing at a second preset ratio to obtain a third video frame set includes:

[0082] S31. Divide the image corresponding to each video frame in the first video frame set that has not undergone video frame masking into an equal number of image blocks to form an image block matrix.

[0083] S32. Generate an occlusion range for each image block matrix that conforms to the second preset ratio according to a preset random algorithm;

[0084] S33. Reset the pixels within the occlusion range in each of the image block matrices to the second preset pixel value;

[0085] S34. The video frames after the image block pixels have been reset are combined to form the third video frame set.

[0086] In this embodiment of the invention, the preset random algorithm can be a random algorithm that generates geometric shapes using JAVA or Python, as long as the proportion of the geometric shapes generated by the preset random algorithm covering the image block matrix is ​​consistent with the second preset proportion.

[0087] In an optional embodiment of the present invention, the second preset pixel value may be the same as or different from the first preset pixel value described above, or it may be set to 0 or 255.

[0088] It can be understood that, in this embodiment of the invention, similar to the video masking processing of the first video frame set with a first preset ratio described above, the image block masking processing of each video frame in the first video frame set that has not undergone video frame masking processing with a second preset ratio includes, but is not limited to, assigning values ​​to the pixels of the image blocks. Other methods may also be used, such as replacing the image blocks in the original video frame with uniform image or symbol elements.

[0089] In this embodiment of the invention, not only are a certain number of video frames in the first video frame set occluded as a whole, but also video frames in the first video frame set that have not undergone video frame occlusion processing are subjected to image block occlusion processing according to a certain proportion. This operation can make the occlusion objects more refined, which is beneficial for subsequent use of the model to extract relevant video features. It encourages the model to not only focus on learning the contextual information between multiple video frames, but also on learning the contextual information within the same video frame, which is beneficial for improving the model's learning ability for video feature extraction.

[0090] Step D: Use the second video frame set and the third video frame set to train the pre-constructed feature extraction model for video frame prediction, and obtain the video frame prediction results;

[0091] In this embodiment of the invention, the pre-built feature extraction model can be a model built based on Swin Transformer. Compared with traditional CNN (Convolutional Neural Networks) models, Swin Transformer can solve the problem of high computational cost when dealing with high resolution videos or images. In addition, Swin Transformer also has a significant advantage in understanding context.

[0092] For details, please refer to Figure 4 As shown, the step of training the pre-constructed feature extraction model for video frame prediction using the second video frame set and the third video frame set to obtain video frame prediction results includes:

[0093] S41. Using the second video frame set and the third video frame set as input, the input is windowed and vector transformed using the encoding layer of the pre-constructed feature extraction model to obtain the window vector matrix corresponding to each window.

[0094] S42. Using the window attention mechanism of the pre-constructed feature extraction model, feature extraction is performed on each of the window vector matrices to obtain the feature vector corresponding to each of the window vector matrices;

[0095] S43. Using the window interaction mechanism of the pre-built feature extraction model, the feature vectors corresponding to each window vector are fused and calculated to obtain the global feature vector;

[0096] S44. Using the decoding layer of the pre-constructed feature extraction model, generate video frame prediction results corresponding to the input based on the global feature vector.

[0097] In this embodiment of the invention, based on the computational advantages of the Swin Transformer in the video domain and its advantages in context understanding, the feature extraction model built on the Swin Transformer is used to predict video frame results, which helps to reduce the difficulty of training and improve the accuracy of training.

[0098] Step F: Calculate the loss value between the video frame prediction result and the first video frame set, adjust the parameters of the feature extraction model according to the loss value, and return to step D above. When the adjusted feature extraction model meets the preset model training conditions, exit the prediction training and obtain the trained video feature extraction model.

[0099] In this embodiment of the invention, a preset loss function can be used to calculate the loss value between the video frame prediction result and the first video frame set.

[0100] In this embodiment of the invention, the preset model training condition may be that the loss value is less than or equal to a preset loss value threshold or that the training of the pre-built feature extraction model tends to converge, or that the number of training iterations of the pre-built feature extraction model reaches a preset training iteration threshold.

[0101] In this embodiment of the invention, the loss value between the video frame prediction result and the first video frame set is calculated using the following preset cross-entropy loss function:

[0102]

[0103] Where L(x,y) represents the loss value between the true result x and the predicted result y of the first video frame set, N represents the total number of video frames in the video samples, and l n x represents the difference between the nth video frame in the first video frame set and the nth video frame in the video frame prediction result. n y represents the vector value of the nth video frame in the first set of video frames. n represents the vector value of the nth video frame in the video frame prediction result, and t represents the number of iterations in the training process of the pre-constructed feature extraction model.

[0104] In this embodiment of the invention, adding a fully connected layer and a softmax layer after the trained video feature extraction model can further perform video classification, or use the trained video feature extraction model to perform video frame interpolation and predict backend video content based on the frontend video.

[0105] The embodiments of the present invention not only perform video frame occlusion processing on video frames in video samples at a first preset ratio, but also perform image block occlusion processing on image blocks in unoccluded video frames at a second preset ratio. This enables the pre-built feature extraction model to learn not only the contextual information between video frames, but also the contextual information within the same video frame during the training process of video frame prediction using the pre-built feature extraction model, thereby improving the feature extraction capability of the video feature extraction model.

[0106] like Figure 5 The diagram shown is a functional block diagram of a video feature extraction model training device provided in an embodiment of the present invention.

[0107] The video feature extraction model training device 100 of the present invention can be installed in an electronic device. Depending on the functions implemented, the video feature extraction model training device 100 may include a sample frame processing module 101, a video frame occlusion processing module 102, an image occlusion processing module 103, and a video frame prediction training module 104. The module described in this invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and which are stored in the memory of the electronic device.

[0108] In this embodiment, the functions of each module / unit are as follows:

[0109] The sample frame segmentation processing module 101 is used to acquire video samples, perform frame segmentation processing on the video samples, and obtain a first video frame set.

[0110] The video frame masking processing module 102 is used to perform video frame masking processing on the first video frame set at a first preset ratio, and to collect the video frames after video frame masking processing into a second video frame set.

[0111] The image masking processing module 103 is used to perform a second preset ratio of image block masking processing on the image blocks of each video frame in the first video frame set that has not undergone video frame masking processing, so as to obtain a third video frame set.

[0112] The video frame prediction training module 104 is used to train a pre-constructed feature extraction model for video frame prediction using the second video frame set and the third video frame set, to obtain video frame prediction results, calculate the loss value between the video frame prediction results and the first video frame set, adjust the parameters of the feature extraction model according to the loss value, and exit the prediction training when the adjusted feature extraction model meets the preset model training conditions, thereby obtaining a trained video feature extraction model.

[0113] In detail, each module in the video feature extraction model training device 100 described in this embodiment of the invention adopts the same approach as described above. Figures 1 to 4 The training method for the video feature extraction model described herein uses the same techniques and can produce the same technical effects, so it will not be repeated here.

[0114] like Figure 6 The diagram shown is a schematic representation of an electronic device for implementing a video feature extraction model training method according to an embodiment of the present invention.

[0115] The electronic device 1 may include a processor 10, a memory 11 and a bus, and may also include a computer program stored in the memory 11 and capable of running on the processor 10, such as a video feature extraction model training program.

[0116] The memory 11 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device 1, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 11 can include both internal and external storage units of the electronic device 1. The memory 11 can be used not only to store application software and various types of data installed on the electronic device 1, such as the code for a video feature extraction model training program, but also to temporarily store data that has been output or will be output.

[0117] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules (such as video feature extraction model training programs) stored in the memory 11, and calls data stored in the memory 11 to perform various functions of the electronic device 1 and process data.

[0118] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0119] Figure 6 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 6 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0120] For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0121] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device 1 and other electronic devices.

[0122] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 1 and to display a visual user interface.

[0123] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0124] The video feature extraction model training program stored in the memory 11 of the electronic device 1 is a combination of multiple instructions, which, when run in the processor 10, can achieve the following:

[0125] Step A: Obtain video samples and perform frame segmentation on the video samples to obtain the first set of video frames;

[0126] Step B: Perform video frame masking processing on the first video frame set at a first preset ratio, and then collect the video frames after video frame masking processing into a second video frame set.

[0127] Step C: Perform image block occlusion processing on the image blocks of each video frame in the first video frame set that has not undergone video frame occlusion processing with a second preset ratio to obtain a third video frame set;

[0128] Step D: Use the second video frame set and the third video frame set to train the pre-constructed feature extraction model for video frame prediction, and obtain the video frame prediction results;

[0129] Step F: Calculate the loss value between the video frame prediction result and the first video frame set, adjust the parameters of the feature extraction model according to the loss value, and return to step D above. When the adjusted feature extraction model meets the preset model training conditions, exit the prediction training and obtain the trained video feature extraction model.

[0130] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0131] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following:

[0132] Step A: Obtain video samples and perform frame segmentation on the video samples to obtain the first set of video frames;

[0133] Step B: Perform video frame masking processing on the first video frame set at a first preset ratio, and then collect the video frames after video frame masking processing into a second video frame set.

[0134] Step C: Perform image block occlusion processing on the image blocks of each video frame in the first video frame set that has not undergone video frame occlusion processing with a second preset ratio to obtain a third video frame set;

[0135] Step D: Use the second video frame set and the third video frame set to train the pre-constructed feature extraction model for video frame prediction, and obtain the video frame prediction results;

[0136] Step F: Calculate the loss value between the video frame prediction result and the first video frame set, adjust the parameters of the feature extraction model according to the loss value, and return to step D above. When the adjusted feature extraction model meets the preset model training conditions, exit the prediction training and obtain the trained video feature extraction model.

[0137] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0138] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0139] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0140] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0141] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0142] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The term "second class" is used to indicate names and does not indicate any specific order.

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for training a video feature extraction model, characterized in that, The method includes: Step A: Obtain video samples and perform frame segmentation on the video samples to obtain the first set of video frames; Step B: Perform video frame masking processing on the first video frame set at a first preset ratio, and then collect the video frames after video frame masking processing into a second video frame set. Step C: Perform image block occlusion processing on the image blocks of each video frame in the first video frame set that has not undergone video frame occlusion processing with a second preset ratio to obtain a third video frame set; Step D: Use the second video frame set and the third video frame set to train the feature extraction model based on Swin Transformer for video frame prediction, and obtain the video frame prediction results. Step F: Calculate the loss value between the video frame prediction result and the first video frame set, adjust the parameters of the feature extraction model according to the loss value, and return to step D above. When the adjusted feature extraction model meets the preset model training conditions, exit the prediction training and obtain the trained video feature extraction model.

2. The video feature extraction model training method as described in claim 1, characterized in that, The step of performing video frame masking processing on the first video frame set at a first preset ratio, and then assembling the masked video frames into a second video frame set, includes: Each video frame in the first video frame set is numbered sequentially; Randomly select a number from the range of the numbers, representing the first preset proportion, as the target number to be obscured; The image pixels in the video frame corresponding to each of the occluded target numbers are reset to the first preset pixel value; The second video frame set is obtained by collecting all the video frames after pixel reset.

3. The video feature extraction model training method as described in claim 1, characterized in that, The step of performing image block masking processing on image blocks of each video frame in the first video frame set that has not undergone video frame masking processing at a second preset ratio to obtain a third video frame set includes: The image corresponding to each video frame in the first video frame set that has not undergone video frame masking is divided into an equal number of image blocks to form an image block matrix. Each image patch matrix is ​​generated according to a preset random algorithm, and the occlusion range conforms to the second preset ratio. The pixels within the occlusion range in each of the image block matrices are reset to the second preset pixel value; The third video frame set is composed of video frames whose image block pixels have been reset.

4. The video feature extraction model training method as described in claim 1, characterized in that, The step of training a feature extraction model based on the Win Transformer using the second and third video frame sets to predict video frames, and obtaining video frame prediction results, includes: Using the second set of video frames and the third set of video frames as input, the encoding layer of the feature extraction model is used to perform windowing and vector transformation on the input to obtain the window vector matrix corresponding to each window; The feature extraction model uses a window attention mechanism to extract features from each window vector matrix, resulting in a feature vector corresponding to each window vector matrix. Using the window interaction mechanism of the feature extraction model, the feature vectors corresponding to each window vector are fused and calculated to obtain a global feature vector; The decoding layer of the feature extraction model is used to generate video frame prediction results corresponding to the input based on the global feature vector.

5. The video feature extraction model training method as described in claim 1, characterized in that, The calculation of the loss value between the video frame prediction result and the first video frame set includes: The loss value between the video frame prediction result and the first video frame set is calculated using the following preset cross-entropy loss function: in, This represents the actual result of the first video frame set. With the video frame prediction results The loss value between This represents the total number of video frames in the video sample. Indicates the first video frame set The video frame and the prediction result of the video frame. The difference value between video frames Indicates the first video frame set Vector values ​​of each video frame. Indicating the first video frame in the prediction result Vector values ​​of each video frame. This indicates the number of iterations during the training process of the feature extraction model.

6. The video feature extraction model training method as described in claim 1, characterized in that, The step of performing frame segmentation on the video samples to obtain a first set of video frames includes: All frames corresponding to each video sample are averaged and grouped. Randomly select an equal number of frames from each corresponding set of frames as the target frames; All target frames are combined to form the first video frame set.

7. A video feature extraction model training device, characterized in that, The device includes: The sample frame segmentation processing module is used to acquire video samples, perform frame segmentation processing on the video samples, and obtain a first set of video frames. The video frame masking processing module is used to perform video frame masking processing on the first video frame set at a first preset ratio, and to collect the video frames after video frame masking processing into a second video frame set. The image masking processing module is used to perform image block masking processing on the image blocks of each video frame in the first video frame set that has not undergone video frame masking processing at a second preset ratio, so as to obtain a third video frame set. The video frame prediction training module is used to train the feature extraction model based on Swin Transformer to predict video frames using the second video frame set and the third video frame set, obtain video frame prediction results, calculate the loss value between the video frame prediction results and the first video frame set, adjust the parameters of the feature extraction model according to the loss value, and exit the prediction training when the adjusted feature extraction model meets the preset model training conditions, thus obtaining the trained video feature extraction model.

8. The video feature extraction model training device as described in claim 7, characterized in that, The video frame prediction training module obtains the video frame prediction results through the following operations: Using the second set of video frames and the third set of video frames as input, the encoding layer of the feature extraction model is used to perform windowing and vector transformation on the input to obtain the window vector matrix corresponding to each window; The feature extraction model uses a window attention mechanism to extract features from each window vector matrix, resulting in a feature vector corresponding to each window vector matrix. Using the window interaction mechanism of the feature extraction model, the feature vectors corresponding to each window vector are fused and calculated to obtain a global feature vector; The decoding layer of the feature extraction model is used to generate video frame prediction results corresponding to the input based on the global feature vector.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the video feature extraction model training method as described in any one of claims 1 to 6.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the video feature extraction model training method as described in any one of claims 1 to 6.

Images