Video classification methods, apparatus, computer equipment and storage media

Abstract

This application relates to a video classification method, apparatus, computer device, and storage medium. The method includes: extracting multiple video frames from a video sequence corresponding to the video to be classified, generating a sampling sequence. The sampling sequence includes frame vectors corresponding to the multiple video frames. Frame extraction reduces the amount of computational data, saves computational resources, and improves video classification efficiency. Based on the vector differences between pairs of adjacent video frames in the sampling sequence, each vector difference corresponds to a difference frame. The frame vectors corresponding to multiple difference frames form a difference sequence, which represents the action information in the video to be classified through the vector differences between adjacent video frames. The difference sequence is combined with the sampling sequence to generate a splicing sequence. Based on the first encoding result corresponding to the splicing sequence and the second encoding result corresponding to the difference sequence, the classification result of the video to be classified is determined. Combining the encoding results corresponding to the two sequences to determine the classification result of the video to be classified improves the accuracy of the video classification result.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a video classification method, apparatus, computer device, and storage medium. Background Technology

[0002] With the rapid development of mobile terminals and the quick advancements in terrestrial base stations, the mobile internet has developed rapidly. Consequently, internet social platforms, based on the mobile internet, have experienced rapid growth in recent years. In today's fast-paced internet environment, a vast amount of video is generated daily, and the types of videos are incredibly diverse. How social platforms can utilize and categorize these videos is an important issue that is currently under research and development. Only through reasonable video categorization can social platforms operate healthily and sustainably.

[0003] However, existing video classification methods extract a small portion of video frames, perform feature extraction on these frames, and then determine the classification result. But the features extracted from the video frames alone cannot reveal the corresponding action information in the video, so the classification results obtained based on the features extracted from a small portion of the video frames have low accuracy. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides a video classification method, apparatus, computer device, and storage medium.

[0005] Firstly, this application provides a video classification method, including:

[0006] Multiple video frames are extracted from the video sequence corresponding to the video to be classified to generate a sampling sequence, wherein the sampling sequence includes frame vectors corresponding to the multiple video frames;

[0007] A difference sequence is generated based on the vector difference between adjacent video frames in the sampling sequence, wherein each vector difference corresponds to a difference frame, and the difference sequence includes frame vectors corresponding to multiple difference frames;

[0008] The difference sequence is combined with the sampling sequence to generate a spliced ​​sequence;

[0009] Based on the first encoding result corresponding to the spliced ​​sequence and the second encoding result corresponding to the difference sequence, the classification result of the video to be classified is determined.

[0010] Secondly, this application provides a video classification device, comprising:

[0011] The frame extraction module is used to extract multiple video frames from the video sequence corresponding to the video to be classified and generate a sampling sequence, wherein the sampling sequence includes frame vectors corresponding to the multiple video frames;

[0012] The generation module is used to generate a difference sequence based on the vector difference between adjacent video frames in the sampling sequence, wherein each vector difference corresponds to a difference frame, and the difference sequence includes frame vectors corresponding to multiple difference frames;

[0013] The splicing module is used to combine the difference sequence and the sampling sequence to generate a spliced ​​sequence;

[0014] The classification module is used to determine the classification result of the video to be classified based on the first encoding result corresponding to the spliced ​​sequence and the second encoding result corresponding to the difference sequence.

[0015] Thirdly, this application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0016] Multiple video frames are extracted from the video sequence corresponding to the video to be classified to generate a sampling sequence, wherein the sampling sequence includes frame vectors corresponding to the multiple video frames;

[0017] A difference sequence is generated based on the vector difference between adjacent video frames in the sampling sequence, wherein each vector difference corresponds to a difference frame, and the difference sequence includes frame vectors corresponding to multiple difference frames;

[0018] The difference sequence is combined with the sampling sequence to generate a spliced ​​sequence;

[0019] Based on the first encoding result corresponding to the spliced ​​sequence and the second encoding result corresponding to the difference sequence, the classification result of the video to be classified is determined.

[0020] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0021] Multiple video frames are extracted from the video sequence corresponding to the video to be classified to generate a sampling sequence, wherein the sampling sequence includes frame vectors corresponding to the multiple video frames;

[0022] A difference sequence is generated based on the vector difference between adjacent video frames in the sampling sequence, wherein each vector difference corresponds to a difference frame, and the difference sequence includes frame vectors corresponding to multiple difference frames;

[0023] The difference sequence is combined with the sampling sequence to generate a spliced ​​sequence;

[0024] Based on the first encoding result corresponding to the spliced ​​sequence and the second encoding result corresponding to the difference sequence, the classification result of the video to be classified is determined.

[0025] This solution can be applied to optimize computer vision processing in the field of deep learning technology. Based on the video classification method described above, multiple video frames are extracted from the video sequence corresponding to the video to be classified, generating a sampling sequence. The sampling sequence includes frame vectors corresponding to multiple video frames. Frame extraction reduces the amount of data required for subsequent computation, thereby saving computational resources and improving video classification efficiency. Based on the vector difference between each pair of adjacent video frames in the sampling sequence, each vector difference corresponds to a difference frame. The frame vectors corresponding to multiple difference frames form a difference sequence, which represents the action information in the video to be classified through the vector difference between adjacent video frames. The difference sequence and the sampling sequence are combined to generate a splicing sequence. Based on the first encoding result corresponding to the splicing sequence and the second encoding result corresponding to the difference sequence, the classification result of the video to be classified is determined. Combining the encoding results corresponding to the two sequences to determine the classification result of the video to be classified improves the accuracy of the video classification result. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating a video classification method in one embodiment;

[0029] Figure 2 Here is a block diagram of the internal structure of the conversion model in one embodiment;

[0030] Figure 3 This is a block diagram of the internal structure of the encoder in one embodiment;

[0031] Figure 4 This is a flowchart illustrating a video classification method in one embodiment;

[0032] Figure 5 This is a structural block diagram of a video classification device in one embodiment;

[0033] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] In one embodiment, Figure 1 This is a flowchart illustrating a video classification method in one embodiment, with reference to... Figure 1 This paper provides a video classification method. This embodiment primarily illustrates the application of this method to a server, and the video classification method specifically includes the following steps:

[0036] Step S110: Extract multiple video frames from the video sequence corresponding to the video to be classified to generate a sampling sequence.

[0037] The sampling sequence includes frame vectors corresponding to multiple video frames.

[0038] Specifically, the video to be classified is the video that needs to be classified. The video sequence includes multiple video frames, each video frame corresponds to an image. Multiple video frames can be extracted from the video sequence by frame-by-frame extraction or sampling extraction. The extracted multiple video frames are convolved to convert the images corresponding to the video frames into corresponding frame vectors. The frame vectors corresponding to the extracted multiple video frames are used to form a sampling sequence.

[0039] Frame-by-frame extraction involves extracting each video frame from a video sequence according to its timestamp. This is equivalent to utilizing all video frames in the video sequence. Therefore, the advantage of frame-by-frame extraction is that it obtains more complete video information. However, since a large number of frames are obtained, it may increase the consumption of computing resources.

[0040] Sampling extraction involves sampling video frames from a video sequence at preset time intervals. The preset time interval is the difference between the timestamps of two extracted video frames. This preset time interval can be customized based on the number of sampled frames; a higher number of sampled frames results in a shorter preset time interval, and vice versa. For example, a preset time interval of one second means that one video frame is extracted every second. Using sampling extraction reduces the amount of data required for subsequent computation, thereby saving computational resources and improving video classification efficiency. In this embodiment, sampling extraction is chosen to process the video to be classified.

[0041] Step S120: Generate a difference sequence based on the vector difference between adjacent video frames in the sampling sequence.

[0042] Each vector difference corresponds to a difference frame, and the difference sequence includes frame vectors corresponding to multiple difference frames.

[0043] Specifically, the vector difference is obtained by subtracting the corresponding frame vectors of two adjacent video frames. For example, the sampling sequence includes frame vector 1 corresponding to video frame 1, frame vector 2 corresponding to video frame 2, frame vector 3 corresponding to video frame 3, ..., frame vector n corresponding to video frame n. The vector difference can be frame vector 2 - frame vector 1, frame vector 3 - frame vector 2, ..., frame vector n - frame vector (n-1), that is, subtract the frame vector with the smaller timestamp from the frame vector with the larger timestamp. Each vector difference corresponds to a difference frame. That is, the frame vector obtained by subtracting frame vector 1 from frame vector 2 corresponds to difference frame 1, the frame vector obtained by subtracting frame vector 2 from frame vector 3 corresponds to difference frame 2, and so on. Multiple difference frames and their corresponding frame vectors are combined to form a difference sequence.

[0044] Step S130: Combine the difference sequence with the sampling sequence to generate a spliced ​​sequence.

[0045] Specifically, the spliced ​​sequence includes the corresponding frame vectors of the video frames sampled from the video to be classified, as well as the frame vectors of the difference frames.

[0046] Step S140: Based on the first encoding result corresponding to the spliced ​​sequence and the second encoding result corresponding to the difference sequence, determine the classification result of the video to be classified.

[0047] Specifically, the first encoding result can be used to indicate the semantic representation of the spliced ​​sequence, and the second encoding result can be used to indicate the semantic representation of the difference sequence. Combining the semantic representations of the two sequences to determine the classification result of the video to be classified can improve the accuracy of the video classification result.

[0048] In one embodiment, determining the classification result of the video to be classified based on the first encoding result corresponding to the spliced ​​sequence and the second encoding result corresponding to the difference sequence includes: determining the first encoding result corresponding to the spliced ​​sequence and the second encoding result corresponding to the difference sequence respectively; and determining the classification result of the video to be classified based on the integrated result after integrating the first encoding result and the second encoding result.

[0049] Specifically, the frame vectors in the concatenated sequence are input as parameters to the first transformation model for encoding, and the first transformation model outputs the first encoded result corresponding to the concatenated sequence. The frame vectors in the difference sequence are input as parameters to the second transformation model for encoding, and the second transformation model outputs the second encoded result corresponding to the difference sequence. Both the first and second transformation models are implemented using Transformer network models; the first transformation model is denoted as Transformer1, and the second transformation model as Transformer2.

[0050] The first encoding result output by the first transformation model and the second encoding result output by the second transformation model are integrated to generate an integrated result. The integrated result is then used as an input parameter to the fully connected layer in the convolutional neural network to determine the classification result of the video to be classified.

[0051] In one embodiment, determining the first encoding result corresponding to the spliced ​​sequence includes: obtaining a position vector and a classification vector; and encoding the position vector, the classification vector, and the spliced ​​sequence to generate the first encoding result corresponding to the spliced ​​sequence.

[0052] Specifically, the classification vector is denoted as CLS Token. The classification vector is a randomly generated initial vector. The classification vector is just a vector, unlike the frame vectors in the concatenation sequence. Each frame vector is used to represent the semantic information corresponding to an image, while the classification vector does not represent any semantic information. Therefore, when encoding the classification vector and the frame vectors in the concatenation sequence, the classification vector without semantic information can fairly integrate the semantic information of different frame vectors.

[0053] Since the order of frame vectors will affect the action information corresponding to the video to be classified, the order between frame vectors is very important. When encoding the spliced ​​sequence, position vectors are added to add position information to each frame vector in the spliced ​​sequence, so as to determine the position of each frame vector in the spliced ​​sequence and the relative position between each frame vector.

[0054] In one embodiment, encoding the position vector, the classification vector, and the concatenated sequence to generate the first encoding result corresponding to the concatenated sequence includes: adding the classification vector to the beginning of the concatenated sequence to form a sequence to be encoded with multiple frame vectors in the concatenated sequence; embedding the position vector into each frame vector in the sequence to be encoded to obtain multiple corresponding position embedding vectors; and encoding the target sequence to generate the first encoding result corresponding to the concatenated sequence.

[0055] The target sequence includes the classification vector at the beginning and the plurality of position embedding vectors, and the first encoding result includes the encoding vector corresponding to each of the position embedding vectors and the first semantic vector corresponding to the classification vector.

[0056] Specifically, since the relative positions of the frame vectors in the concatenated sequence affect the semantic representation of the video to be classified, adding the classification vector to the beginning of the concatenated sequence will not affect the positional relationships between other frame vectors. The position vector is a learnable vector. Embedding the position vector into each frame vector adds corresponding positional information to each frame vector, resulting in position embedding vectors for each frame vector. These multiple position embedding vectors, along with the classification vector at the beginning, are then input as parameters to the encoder in the first conversion model for encoding.

[0057] like Figure 2 As shown, the Transformer network model includes multiple Encoders. Figure 2 Only four encoders are shown. The number of encoders can be customized as needed. Each encoder has the same structure. After the target sequence is processed by multiple encoders in sequence, the Transformer network model outputs the encoded vectors corresponding to the embedding vectors at each position and the first semantic vector corresponding to the classification vector.

[0058] Since the generation process of the second encoding result is the same as that of the first encoding result, the only difference being the input parameters of the Transformer network model, the generation process of the second encoding result will not be described again here.

[0059] In one embodiment, encoding the target sequence to generate the first encoding result corresponding to the concatenated sequence includes: determining the correlation between each target vector in the target sequence, wherein the target vector includes the position embedding vector and the classification vector; and generating an encoding vector corresponding to each position embedding vector and a first semantic vector corresponding to the classification vector based on the correlation between each target vector.

[0060] Specifically, such as Figure 3 As shown, the Encoder includes a normalization module (Norm), a multi-head attention module, a residual module, a normalization module, and a fully connected feedforward module (MLP). Linear transformation is performed in the normalization module, and then the linearly transformed target vector is input to the multi-head attention module.

[0061] In the multi-head attention module, based on the weight matrix (W) Q WK W V The weight vector corresponding to each target vector is obtained by multiplying the target vector with the target vector. The weight vector corresponding to each target vector includes the query vector (Q vector), the key vector (K vector), and the content vector (Value vector).

[0062] The correlation between each target vector is determined by taking the dot product of the Q-vector and the K-vector corresponding to each target vector. Then, the correlation value set corresponding to each target vector is obtained based on the correlation between each target vector and the target vector. The correlation value set is denoted as Score. That is, the Score corresponding to each target vector contains the correlation between itself and each other target vector. For example, if the target sequence includes two target vectors, target vector A and target vector B, the weight vector corresponding to target vector A includes (q1, k1, v1), and the weight vector corresponding to target vector B includes (q2, k2, v2). Then, the correlation between target vector A and target vector B is q1·k2, and the correlation value set corresponding to target vector A includes q1·k1 and q1·k2. Of course, each target vector has the strongest correlation with itself, that is, the current target vector has the greatest influence on the current position. The correlation between the current target vector and other target vectors is used to indicate the degree of influence of the current target vector on target vectors at other positions.

[0063] Normalize the correlation degree corresponding to each target vector, that is, set the correlation value set corresponding to each target vector to 1. Referring to the example above, set the Score corresponding to target vector A to 1, that is, q1·k1+q1·k2=1, 0<q1·k1<1, 0<q1·k2<1.

[0064] Multiply and sum the correlation degrees of each associated target vector in the set of associated values ​​corresponding to the target vector with the V vector of the associated target vector to obtain the output vector corresponding to each target vector. The output vector is denoted as Z. Referring to the example above, let q1·k1·v1+q1·k2·v2=Z.

[0065] However, the above only utilizes a single weight matrix. The target vector is encoded by a multi-head attention module, meaning that each target vector will utilize multiple weight matrices to obtain multiple output vectors corresponding to these weight matrices. These multiple output vectors are then merged to form a new output vector. This new output vector, combined with the corresponding target vector, is sequentially input into the residual module, normalization module, and fully connected feedforward module to finally obtain the transformation vector corresponding to the target vector. That is, when the target vector is a position embedding vector, the corresponding transformation vector is the corresponding encoded vector; when the target vector is a classification vector, the corresponding transformation vector is the corresponding first semantic vector.

[0066] In one embodiment, combining the difference sequence with the sampling sequence to generate a spliced ​​sequence includes: interleaving each difference frame in the difference sequence between two adjacent video frames in the sampling sequence that correspond to the vector difference of the difference frame, thereby generating the spliced ​​sequence.

[0067] Specifically, referring to the above embodiment, the sampling sequence includes frame vector 1, frame vector 2, frame vector 3, ..., frame vector n, while the difference sequence includes frame vector 2-frame vector 1, frame vector 3-frame vector 2, ..., frame vector n-frame vector (n-1). The splicing sequence formed by interspersing the difference frames in the sampling sequence is as follows: frame vector 1, frame vector 2-frame vector 1, frame vector 2, frame vector 3-frame vector 2, frame vector 3, ..., frame vector (n-1), frame vector n-frame vector (n-1), frame vector n.

[0068] In one embodiment, determining the classification result of the video to be classified based on the integrated result of the first encoding result and the second encoding result includes: adding the first semantic vector in the first encoding result to the second semantic vector in the second encoding result to generate an integrated semantic vector, wherein the integrated result includes the integrated semantic vector; determining the probability of the integrated semantic vector belonging to each classification category; and taking the classification category corresponding to the highest probability value as the classification result of the video to be classified.

[0069] Specifically, such as Figure 4 As shown, the concatenated sequence is input into Transformer1, and the difference sequence is input into Transformer2. The first semantic vector output by Transformer1 is added to the second semantic vector output by Transformer2 to obtain the integrated semantic vector. The integrated semantic vector is then input into the fully connected layer of the convolutional neural network to determine the probability that the integrated semantic vector belongs to each category. Based on the probability, the category corresponding to the integrated semantic vector can be determined. In other words, the semantic representations of the concatenated sequence and the difference sequence are combined to determine the classification result of the video to be classified, thus improving the accuracy of video classification.

[0070] Figure 1 This is a flowchart illustrating a video classification method in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0071] In one embodiment, such as Figure 5 As shown, a video classification device is provided, comprising:

[0072] The frame extraction module 210 is used to extract multiple video frames from the video sequence corresponding to the video to be classified and generate a sampling sequence, wherein the sampling sequence includes frame vectors corresponding to multiple video frames;

[0073] The generation module 220 is used to generate a difference sequence based on the vector difference between adjacent video frames in the sampling sequence, wherein each vector difference corresponds to a difference frame, and the difference sequence includes frame vectors corresponding to multiple difference frames;

[0074] The splicing module 230 is used to combine the difference sequence and the sampling sequence to generate a spliced ​​sequence;

[0075] The classification module 240 is used to determine the classification result of the video to be classified based on the first encoding result corresponding to the spliced ​​sequence and the second encoding result corresponding to the difference sequence.

[0076] In one embodiment, the classification module 240 is specifically used for:

[0077] Determine the first encoding result corresponding to the spliced ​​sequence and the second encoding result corresponding to the difference sequence, respectively;

[0078] Based on the integrated result of the first encoding result and the second encoding result, the classification result of the video to be classified is determined.

[0079] In one embodiment, the classification module 240 is specifically used for:

[0080] Obtain the location vector and classification vector;

[0081] The position vector, the classification vector, and the concatenated sequence are encoded to generate the first encoding result corresponding to the concatenated sequence.

[0082] In one embodiment, the classification module 240 is specifically used for:

[0083] The classification vector is added to the beginning of the concatenated sequence, and together with multiple frame vectors in the concatenated sequence, they form a sequence to be encoded.

[0084] The position vectors are embedded into each frame vector of the sequence to be encoded to obtain multiple corresponding position embedding vectors;

[0085] The target sequence is encoded to generate the first encoding result corresponding to the concatenated sequence. The target sequence includes the classification vector at the beginning and the plurality of position embedding vectors. The first encoding result includes the encoding vector corresponding to each of the position embedding vectors and the first semantic vector corresponding to the classification vector.

[0086] In one embodiment, the classification module 240 is specifically used for:

[0087] Determine the correlation degree between each target vector in the target sequence, wherein the target vector includes the position embedding vector and the classification vector;

[0088] Based on the correlation between the target vectors, an encoding vector corresponding to each position embedding vector and a first semantic vector corresponding to the classification vector are generated.

[0089] In one embodiment, the classification module 240 is specifically used for:

[0090] The first semantic vector in the first encoding result is added to the second semantic vector in the second encoding result to generate an integrated semantic vector, wherein the integrated result includes the integrated semantic vector;

[0091] Determine the probability that the integrated semantic vector belongs to each classification category;

[0092] The category corresponding to the highest probability value is taken as the classification result of the video to be classified.

[0093] In one embodiment, the splicing module 230 is specifically used for:

[0094] Each difference frame in the difference sequence is interleaved between two adjacent video frames in the sampling sequence that correspond to the vector difference of the difference frame, and the splicing sequence is generated by combining them.

[0095] Figure 6 An internal structural diagram of a computer device is shown in one embodiment. This computer device may specifically be a server. Figure 6As shown, the computer device includes a processor, memory, network interface, input device, and display screen connected via a system bus. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system and may also store a computer program. When executed by the processor, this computer program enables the processor to implement a video classification method. The internal memory may also store a computer program, which, when executed by the processor, enables the processor to implement the video classification method. The display screen can be an LCD screen or an e-ink screen. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0096] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0097] In one embodiment, the video classification device provided in this application can be implemented as a computer program, and the computer program can be implemented as follows: Figure 6 The device runs on the computer shown. The computer's memory can store the various program modules that make up the video classification device, for example, Figure 5 The diagram shows a frame extraction module 210, a generation module 220, a stitching module 230, and a classification module 240. The computer program comprised of these modules causes a processor to execute the steps of the video classification methods described in the various embodiments of this application.

[0098] Figure 6 The computer device shown can be used as follows Figure 5 The frame extraction module 210 in the video classification device illustrates extracts multiple video frames from the video sequence corresponding to the video to be classified, generating a sampling sequence. The sampling sequence includes frame vectors corresponding to the multiple video frames. The computer device, via the generation module 220, generates a difference sequence based on the vector differences between adjacent video frames in the sampling sequence. Each vector difference corresponds to a difference frame, and the difference sequence includes frame vectors corresponding to multiple difference frames. The computer device, via the splicing module 230, combines the difference sequence with the sampling sequence to generate a spliced ​​sequence. The computer device, via the classification module 240, determines the classification result of the video to be classified based on a first encoding result corresponding to the spliced ​​sequence and a second encoding result corresponding to the difference sequence.

[0099] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in any of the above embodiments.

[0100] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the above embodiments.

[0101] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0102] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0103] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A video classification method, characterized in that, The method includes: Multiple video frames are extracted from the video sequence corresponding to the video to be classified to generate a sampling sequence, wherein the sampling sequence includes frame vectors corresponding to the multiple video frames; Based on the vector difference between adjacent video frames in the sampling sequence, a difference sequence is generated, wherein each vector difference corresponds to a difference frame, and the difference sequence includes frame vectors corresponding to multiple difference frames. The difference sequence is used to characterize the motion information in the video to be classified. The difference sequence is combined with the sampling sequence to generate a spliced ​​sequence; Based on the first encoding result corresponding to the spliced ​​sequence and the second encoding result corresponding to the difference sequence, the classification result of the video to be classified is determined; The step of determining the classification result of the video to be classified based on the first encoding result corresponding to the spliced ​​sequence and the second encoding result corresponding to the difference sequence includes: Determine the first encoding result corresponding to the spliced ​​sequence and the second encoding result corresponding to the difference sequence, respectively; Based on the integrated result of the first encoding result and the second encoding result, the classification result of the video to be classified is determined; Determining the first encoding result corresponding to the concatenated sequence and the second encoding result corresponding to the difference sequence includes: The concatenated sequence is input into a first transformation model for encoding processing to obtain a first encoding result corresponding to the concatenated sequence. The difference sequence is input into a second transformation model for encoding processing to obtain a second encoding result corresponding to the difference sequence. The first transformation model is constructed based on the Transformer1 network model, and the second transformation model is constructed based on the Transformer2 network model. The process of determining the classification result of the video to be classified based on the integrated result of the first encoding result and the second encoding result includes: The first semantic vector in the first encoding result is added to the second semantic vector in the second encoding result to generate an integrated semantic vector, wherein the integrated result includes the integrated semantic vector.

2. The method according to claim 1, characterized in that, Determining the first encoding result corresponding to the concatenated sequence includes: Obtain the location vector and classification vector; The position vector, the classification vector, and the concatenated sequence are encoded to generate the first encoding result corresponding to the concatenated sequence.

3. The method according to claim 2, characterized in that, The step of encoding the position vector, the classification vector, and the concatenated sequence to generate the first encoding result corresponding to the concatenated sequence includes: The classification vector is added to the beginning of the concatenated sequence, and together with multiple frame vectors in the concatenated sequence, they form a sequence to be encoded. The position vectors are embedded into each frame vector of the sequence to be encoded to obtain multiple corresponding position embedding vectors; The target sequence is encoded to generate the first encoding result corresponding to the concatenated sequence. The target sequence includes the classification vector at the beginning and the plurality of position embedding vectors. The first encoding result includes the encoding vector corresponding to each of the position embedding vectors and the first semantic vector corresponding to the classification vector.

4. The method according to claim 3, characterized in that, The step of encoding the target sequence to generate the first encoding result corresponding to the concatenated sequence includes: Determine the correlation degree between each target vector in the target sequence, wherein the target vector includes the position embedding vector and the classification vector; Based on the correlation between the target vectors, an encoding vector corresponding to each position embedding vector and a first semantic vector corresponding to the classification vector are generated.

5. The method according to claim 4, characterized in that, The process of determining the classification result of the video to be classified based on the integrated result of the first encoding result and the second encoding result includes: Determine the probability that the integrated semantic vector belongs to each classification category; The category corresponding to the highest probability value is taken as the classification result of the video to be classified.

6. The method according to claim 1, characterized in that, The step of combining the difference sequence with the sampling sequence to generate a concatenated sequence includes: Each difference frame in the difference sequence is interleaved between two adjacent video frames in the sampling sequence that correspond to the vector difference of the difference frame, and the splicing sequence is generated by combining them.

7. A video classification device, characterized in that, The device includes: The frame extraction module is used to extract multiple video frames from the video sequence corresponding to the video to be classified and generate a sampling sequence, wherein the sampling sequence includes frame vectors corresponding to the multiple video frames; The generation module is used to generate a difference sequence based on the vector difference between adjacent video frames in the sampling sequence, wherein each vector difference corresponds to a difference frame, the difference sequence includes frame vectors corresponding to multiple difference frames, and the difference sequence is used to characterize the action information in the video to be classified; The splicing module is used to combine the difference sequence and the sampling sequence to generate a spliced ​​sequence; The classification module is used to determine the classification result of the video to be classified based on the first encoding result corresponding to the spliced ​​sequence and the second encoding result corresponding to the difference sequence; The classification module is specifically used for: Determine the first encoding result corresponding to the spliced ​​sequence and the second encoding result corresponding to the difference sequence, respectively; Based on the integrated result of the first encoding result and the second encoding result, the classification result of the video to be classified is determined; The classification module is specifically used for: The concatenated sequence is input into a first transformation model for encoding processing to obtain a first encoding result corresponding to the concatenated sequence. The difference sequence is input into a second transformation model for encoding processing to obtain a second encoding result corresponding to the difference sequence. The first transformation model is constructed based on the Transformer1 network model, and the second transformation model is constructed based on the Transformer2 network model. The classification module is specifically used for: The first semantic vector in the first encoding result is added to the second semantic vector in the second encoding result to generate an integrated semantic vector, wherein the integrated result includes the integrated semantic vector.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images