Video classification method and device

Abstract

The embodiment of the invention provides a video classification method, and the method comprises the steps: obtaining a training data set which comprises a plurality of training image sequences; the training image sequence is input into a converter model, each frame image in the training image sequence is segmented into a plurality of small blocks through the converter model, an encoder in the converter model comprises a plurality of converter modules, at least part of the converter modules comprise token merging units, and the token merging units are used for merging tokens; each small block corresponds to one token of the converter module; utilizing a token merging unit to merge part of the small blocks according to the similarity of the small blocks to obtain a first token after the training image sequence is updated; inputting the first token into a decoder of the converter model to obtain a classification result of the training image sequence; and after the converter model reaches the preset precision, obtaining a trained converter model. According to the scheme of the embodiment of the invention, the sensing range of the artificial intelligence model can be expanded, and the video classification accuracy is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular, to a video classification method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] When classifying videos, machine learning methods are usually used to classify videos according to business scenarios to meet the needs of specific business scenarios.

[0003] However, current machine learning methods have a small perception range and low classification efficiency when classifying videos.

[0004] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the invention

[0005] The embodiments of the present application provide a video classification method, apparatus, computer device, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems raised above.

[0006] One aspect of an embodiment of the present application provides a video classification method, the method comprising: Acquire a training data set, wherein the training data set includes a plurality of training image sequences, each of the training image sequences includes N frame images; Input the training image sequence into a converter model, and use the converter model to divide each frame image in the training image sequence into a plurality of small blocks, wherein the encoder in the converter model includes a plurality of converter modules, at least some of the converter modules include a token merging unit, and each of the small blocks corresponds to a token of the converter module; Utilizing the token merging unit to merge some of the small blocks according to the similarity of the small blocks, to obtain an updated first token of the training image sequence; Inputting the first token into a decoder of the converter model, and using the decoder to obtain a classification result of the training image sequence based on the first token; After the converter model reaches a preset accuracy, a trained converter model is obtained.

[0007] Optionally, after obtaining the trained converter model, the method further includes: Acquire a target video, extract frame images based on the target video and form a target image sequence, wherein the target image sequence includes N frame images; Inputting the target image sequence into the trained converter model, and dividing each frame image in the target image sequence into a plurality of target small blocks according to the trained converter model; Utilizing the token merging unit to merge some of the target small blocks according to the similarity of the target small blocks, to obtain a second token after the target image sequence is updated; Inputting the second token into a decoder of the trained converter model, and using the decoder to obtain a classification result of the target image sequence based on the second token; A classification result of the target video is obtained based on the classification result of the target image sequence.

[0008] Optionally, the method further comprises: Calculating the key vector of the small block and the dot product between the key vectors using the attention mechanism of the transformer module; The token merging unit is used to determine the similarity between the small blocks using cosine similarity according to the dot product.

[0009] Optionally, the using the token merging unit to merge some of the small blocks according to the similarity of the small blocks includes: Determine the similarity between the small blocks in each frame image by using the token merging unit; When the similarity between two or more small blocks reaches a preset threshold, the token merging unit is used to merge the small blocks whose similarity reaches the preset threshold to obtain a new token.

[0010] Optionally, when the similarity between two or more small blocks reaches a preset threshold, merging the small blocks whose similarity reaches the preset threshold to obtain a new token includes: Using the token merging unit to sum or average the features of the small blocks whose similarity reaches the preset threshold; The token merging unit is used to obtain a new token according to the summation or averaging result.

[0011] Optionally, the using the token merging unit to merge some of the small blocks according to the similarity of the small blocks includes: Using the token merging unit to divide all tokens of the current frame image into two sets: a first set and a second set; Utilizing the token merging unit to calculate the similarity between each token in the first set and all tokens in the second set, and determining K token pairs with the highest similarity; Using the token merging unit to perform weighted averaging on the K token pairs to obtain K new tokens; The token merging unit is used to obtain the updated token of the current frame image according to the K new tokens and the remaining tokens, and the remaining tokens are the remaining tokens in the first set and the second set.

[0012] Optionally, each of the converter modules includes the token merging unit.

[0013] Another aspect of an embodiment of the present application provides a video classification device, the device comprising: An acquisition module is used to acquire a training data set, wherein the training data set includes a plurality of training image sequences, and each of the training image sequences includes N frame images; A segmentation module, used for inputting the training image sequence into a converter model, and using the converter model to segment each frame image in the training image sequence into a plurality of small blocks, wherein the encoder in the converter model includes a plurality of converter modules, at least some of the converter modules include a token merging unit, and each of the small blocks corresponds to a token of the converter module; a merging module, configured to use the token merging unit to merge some of the small blocks according to the similarity of the small blocks, so as to obtain an updated first token of the training image sequence; A classification module, configured to input the first token into a decoder of the converter model, and obtain a classification result of the training image sequence based on the first token using the decoder; The determination module is used to obtain a trained converter model after the converter model reaches a preset accuracy.

[0014] Another aspect of an embodiment of the present application provides a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein: the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described above.

[0015] Another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method described above is implemented.

[0016] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which implements the method described above when executed by a processor.

[0017] The above technical solution adopted in the embodiment of the present application may have the following advantages: The training image sequence is input into the converter model, and the converter model is used to divide the training image into several small blocks (corresponding to tokens). The encoder in the converter model includes several converter modules, and at least some of the converter modules include a token merging unit. The token merging unit is used to merge some small blocks of the input image according to the similarity between the small blocks to obtain updated tokens of the training image sequence. The updated tokens are input into the decoder of the converter model, and the decoder is used to obtain classification results of the training image sequence based on the updated tokens. After the converter model reaches a preset accuracy, a trained converter model is obtained. By merging some small blocks with high similarity to obtain updated tokens, the number of tokens occupied by a single image can be reduced, so that the decoder of the converter model can cover more images, effectively expanding the perception range of the converter model and improving the efficiency and accuracy of the converter model in video classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0019] Figure 1 The following schematically shows an operating environment diagram of a video classification method according to Embodiment 1 of the present application; Figure 2 A flowchart of a video classification method according to Embodiment 1 of the present application is schematically shown; Figure 3 The schematic diagram shows a principle diagram of a video classification method according to the first embodiment of the present application; Figure 4 The newly added process of the video classification method according to the first embodiment of the present application is schematically shown; Figure 5 Schematically shows Figure 2 Flow chart of sub-steps of step S204; Figure 6 Schematically shows Figure 5 Flow chart of sub-steps of step S502; Figure 7 An example diagram schematically showing summing or averaging of images; Figure 8 Schematically shows Figure 2 Another sub-step flow chart of step S204; Fig. 9 Another newly added process of the video classification method according to the first embodiment of the present application is schematically shown; Fig.10A block diagram schematically shows a video classification device according to the second embodiment of the present application; and Fig.11 The hardware architecture diagram of the computer device according to the third embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0021] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0022] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0023] First, the following terms are explained: Transformer model: a deep learning model that uses an attention mechanism, which assigns different weights to different parts of the input data according to their importance. This model is mainly used in the fields of natural language processing and computer vision.

[0024] Attention mechanism: It is a technique in artificial neural networks that mimics cognitive attention. This mechanism can enhance the weight of some parts of the neural network input data while weakening the weight of other parts, so as to focus the network's attention on the most important small part of the data.

[0025] Secondly, in order to facilitate those skilled in the art to understand the technical solutions provided in the embodiments of the present application, the relevant technologies are described below: At present, when machine learning is used to classify videos, due to the limitation of the number of tokens of the artificial intelligence model, the perception range of the artificial intelligence model will be small, and it will not be able to perceive the video from a larger range, and the efficiency of video classification is low.

[0026] To this end, the embodiment of the present application provides a video classification technical solution. In this technical solution, similar parts in the frame images of the video can be merged to expand the perception range of the artificial intelligence model, improve the efficiency of the artificial intelligence model in classifying the video, and improve the accuracy of the classification. See below for details.

[0027] Finally, for ease of understanding, an exemplary operating environment is provided below.

[0028] like Figure 1 As shown, the environment diagram includes a service platform 2, a network 4, and a client 6, wherein: The service platform 2 may be composed of a single or multiple computing devices. The multiple computing devices may include virtualized computing instances. Virtualized computing instances may include virtual machines, such as simulations of computer systems, operating systems, servers, etc. The computing device may load a virtual machine based on a virtual image and / or other data defining specific software (e.g., operating system, dedicated application, server) for simulation. As the demand for different types of processing services changes, different virtual machines may be loaded and / or terminated on one or more computing devices. A hypervisor may be implemented to manage the use of different virtual machines on the same computing device.

[0029] The service platform 2 may be configured to communicate with the client 6 or the like via a network 4. The network 4 includes various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, and / or the like. The network 4 may include physical links, such as coaxial cable links, twisted pair cable links, optical fiber links, combinations thereof, and the like, or wireless links, such as cellular links, satellite links, Wi-Fi links, and the like.

[0030] The service platform 2 can provide storage, reading, writing, querying, deleting and other services, such as providing video uploading, classification of uploaded videos, transcoding, video transmission and other services for clients.

[0031] The client 6 may be an electronic device running an operating system such as Windows, Android™ or iOS, such as a smart phone, a tablet device, a laptop computer, a virtual reality device, a gaming device, a set-top box, a vehicle terminal, or a smart TV. Based on the above operating system, various applications may be run, such as an application for uploading and viewing videos.

[0032] The client 6 may provide / configure a user access page for manipulating the service platform 2 or uploading an object, etc.

[0033] It should be noted that the above devices are exemplary, and the number and type of devices are adjustable in different scenarios or according to different needs.

[0034] The following uses the service platform as the execution subject to introduce the technical solution of the present application through multiple embodiments. It should be noted that these embodiments can be implemented in a variety of different forms and should not be interpreted as being limited to the embodiments described here.

[0035] Embodiment 1 Figure 2 The flowchart of the video classification method according to the first embodiment of the present application is schematically shown.

[0036] like Figure 2 As shown, the video classification method may include steps S200 to S208, wherein: Step S200: acquiring a training data set, where the training data set includes a plurality of training image sequences, and each training image sequence includes N frame images.

[0037] Step S202: input the training image sequence into the converter model, and use the converter model to divide each frame image in the training image sequence into a number of small blocks, wherein the encoder in the converter model includes a number of converter modules, at least some of the converter modules include a token merging unit, and each small block corresponds to a token of the converter module.

[0038] Step S204: Utilize the token merging unit to merge some of the small blocks according to the similarity of the small blocks, and obtain the updated first token of the training image sequence.

[0039] Step S206: input the first token into the decoder of the converter model, and use the decoder to obtain a classification result of the training image sequence based on the first token.

[0040] Step S208, after the converter model reaches a preset accuracy, a trained converter model is obtained.

[0041] The video classification method provided in this embodiment inputs a training image sequence into a converter model, and uses the converter model to divide the training image into a number of small blocks (corresponding to tokens). The encoder in the converter model includes a number of converter modules, and at least some of the converter modules include a token merging unit. The token merging unit is used to merge some small blocks of the input image according to the similarity between the small blocks to obtain updated tokens of the training image sequence, and the updated tokens are input into the decoder of the converter model. The decoder is used to obtain classification results of the training image sequence based on the updated tokens. After the converter model reaches a preset accuracy, a trained converter model is obtained. By merging some small blocks with high similarity to obtain updated tokens, the number of tokens occupied by a single image can be reduced, so that the decoder of the converter model can cover more images, effectively expanding the perception range of the converter model, and improving the efficiency and accuracy of the converter model in video classification.

[0042] The following combination Figure 2 , each step in steps S200~S208 and other optional steps are explained in detail.

[0043] Step S200 , obtain a training data set, the training data set includes multiple training image sequences, and each training image sequence includes N frame images.

[0044] Specifically, N frames of images can be extracted from a video as a training image sequence. In a video, multiple training image sequences can be extracted multiple times. The training data set can be formed based on the training image sequences of multiple videos. N is a natural number, for example, it can be 32, that is, 32 frames of images are taken as a training image sequence.

[0045] Step S202 , input the training image sequence into the converter model, and use the converter model to divide each frame image in the training image sequence into a number of small blocks, wherein the encoder in the converter model includes a number of converter modules, at least some of the converter modules include a token merging unit, and each small block corresponds to a token of the converter module.

[0046] Please refer to Figure 3, which is a schematic diagram of the principle of the video classification method. As shown in the figure, the transformer model includes an encoder and a decoder, wherein the encoder includes several transformer blocks, some of which contain token merging units, while some of which do not. In an optional embodiment, each transformer module includes a token merging unit, that is, tokens are merged in each transformer module, so as to facilitate the capture of features at different levels through multiple transformer modules, accelerate the convergence of the transformer model, and improve the generalization ability of the transformer model.

[0047] In each converter module, a multi-head attention mechanism layer, a feedforward neural network layer, a normalization layer, and a residual connection layer may be included, and a token merging unit may be located between the multi-head attention mechanism layer and the feedforward neural network layer. After the image sequence is input into the converter model, each frame image in the image sequence will be divided into several small blocks and then input into the converter module.

[0048] Step S204 , a token merging unit is used to merge some small blocks according to the similarity of the small blocks to obtain the first token after the training image sequence is updated.

[0049] Please continue to refer to Figure 3 In the converter module including the token merging unit, the token merging unit determines the similarity between the small blocks of each frame image, and then merges some of the small blocks according to the similarity between the small blocks to obtain the updated token of each frame image, and then obtains the updated first token of the training image sequence according to the updated tokens of all frame images. Some exemplary schemes are provided below.

[0050] In an optional embodiment, if Figure 4 As shown, the video classification method of the embodiment of the present application may also include: Step S400, using the attention mechanism of the converter module to calculate the key vectors of the small blocks and the dot products between the key vectors.

[0051] Step S402: using a token merging unit to determine the similarity between small blocks using cosine similarity based on the dot product.

[0052] Specifically, the attention mechanism of the converter module can be used to calculate the dot product between the key vectors of the two small blocks in each small block pair consisting of two small blocks in the frame image, and calculate the key vector of each small block of the small block pair; then the cosine similarity calculation formula is used to calculate the similarity between the small blocks: cosθ=(A·B) / (︱A︱·︱B︱), where cosθ represents cosine similarity, A·B represents the dot product of small block A and small block B, and ︱A︱ and ︱B︱ represent the modulus of the vector of small block A and the modulus of the vector of small block B, respectively.

[0053] In this embodiment, by utilizing the attention mechanism of the converter module to calculate the key vector of the small block and the dot product between the key vectors, and utilizing the token merging unit to determine the similarity between the small blocks using the cosine similarity based on the dot product, the attention mechanism of the converter model can be utilized to effectively determine the similarity between the small blocks, thereby facilitating subsequent merging and improving the accuracy of determining the similarity between the small blocks.

[0054] In an optional embodiment, if Figure 5 As shown, in step S204, using the token merging unit to merge some small blocks according to the similarity of the small blocks may include: Step S500, using a token merging unit to determine the similarity between small blocks in each frame image.

[0055] Step S502: When the similarity between two or more small blocks reaches a preset threshold, the token merging unit is used to merge the small blocks whose similarity reaches the preset threshold to obtain a new token.

[0056] The similarity between small blocks can be measured by using the cosine similarity in the above embodiment, or by using other indicators (such as Euclidean distance).

[0057] Specifically, in a converter module including a token merging unit, the token merging unit is used to determine the similarity between small blocks in each frame image. When the similarity between two or more small blocks reaches a preset threshold, the small blocks whose similarity reaches the preset threshold are merged to obtain a new token.

[0058] It should be understood that in this embodiment, merging is performed when the similarity reaches a preset threshold, and all small blocks corresponding to the preset threshold are merged. In actual applications, different merging effects can be achieved by reasonably setting the preset threshold.

[0059] In an optional embodiment, if Figure 6 As shown, in step S502, when the similarity between two or more small blocks reaches a preset threshold, merging the small blocks whose similarity reaches the preset threshold to obtain a new token may include: Step S600: Using a token merging unit, the features of the small blocks whose similarities reach a preset threshold are summed or averaged.

[0060] Step S602, using a token merging unit to obtain a new token based on the summation or averaging result.

[0061] Specifically, after determining the similarity between small blocks in the frame image, the token merging unit sums the features of the small blocks whose similarity reaches a preset threshold, and obtains a new token based on the summation result; or, averages the features of the small blocks whose similarity reaches a preset threshold, and obtains a new token based on the average result.

[0062] For example, if the similarity between small block A and small block B reaches a preset threshold, the features corresponding to small block A and small block B are summed, and a new token is obtained according to the summation result.

[0063] Please refer to Figure 7 , which is a schematic diagram of summing or averaging an image. As shown in the figure, each frame image can be divided into multiple small blocks, and then some small blocks are summed or averaged according to the similarity of the small blocks to obtain the updated token. In the figure, the images in the third column are summed and merged, and the images in the fourth column are averaged and merged. In addition, in order to protect the privacy of the person, the eyes of the person are blurred, which is not the processing performed by the video classification method in this scheme.

[0064] In this embodiment, the features of small blocks whose similarity reaches a preset threshold are summed or averaged through a token merging unit, and a new token is obtained according to the result of the summation or averaging. The new token can represent the aggregated information of the merged small blocks, remove redundant information in a frame image, and reduce the number of tokens.

[0065] It is understandable that when the similarity of two or more small blocks reaches a threshold, the small blocks are merged to obtain new tokens. Since the number of small block pairs that reach similarity for each input image is not necessarily the same, the number of tokens after an input image is merged may not be fixed. In order to facilitate the unified application of the converter model, it is possible to consider fixing the number of tokens after merging.

[0066] In an optional embodiment, if Figure 8 As shown, in step S204, using the token merging unit to merge some small blocks according to the similarity of the small blocks may include: Step S300: using a token merging unit, divide all tokens of the current frame image into two sets: a first set and a second set.

[0067] Step S302: Utilize the token merging unit to calculate the similarity between each token in the first set and all tokens in the second set, and determine K token pairs with the highest similarity.

[0068] Step S304: Utilize the token merging unit to perform weighted averaging on the K token pairs to obtain K new tokens.

[0069] Step S306, using the token merging unit to obtain updated tokens for the current frame image according to the K new tokens and the remaining tokens, where the remaining tokens are the remaining tokens in the first set and the second set.

[0070] The value of K can be set according to the specific design of the converter model and is not limited here. For example, if K=10, the 10 token pairs with the highest similarity in the two sets are merged, and then the tokens after the 10 token pairs are merged are combined with other unmerged tokens to obtain new tokens for the current frame image, and the number of new tokens is less than the original number of tokens. In addition, the K corresponding to the token merging units of different converter modules can be the same or different. For example, fewer merges can be performed at the shallow layer, and more merges can be performed at the deep layer, so that more local information can be retained at the shallow layer to avoid losing important local information prematurely; and more global semantic information can be obtained at the deep layer to improve the accuracy and generalization ability of the converter model.

[0071] In this embodiment, by utilizing a token merging unit to divide all tokens of the current frame image into two sets, a first set and a second set, the similarity between each token in the first set and all tokens in the second set is calculated, the K token pairs with the highest similarity are determined, and a weighted average is performed on the K token pairs to obtain K new tokens. The updated tokens of the current frame image are obtained based on the K new tokens and the remaining tokens. The number of tokens merged can be accurately controlled, and the trained conversion model can be uniformly applied to the specified scene. At the same time, this merging method can achieve fast and accurate merging results.

[0072] Step S206 , the first token is input into the decoder of the converter model, and the decoder is used to obtain the classification result of the training image sequence based on the first token.

[0073] Specifically, the first token obtained by the encoder of the converter model can be input into the decoder of the converter model, and the decoder can be used to obtain the classification result of the training image sequence based on the first token. It can be understood that since the tokens are merged, the number of merged tokens is less than the original number of tokens, so the decoder can receive more image information, thereby improving the perception range of the converter model and improving the efficiency and accuracy of video classification.

[0074] For example, assuming that the number of tokens input to the decoder of the converter model is 2048, and the original number of tokens occupied by a single frame image is 256, the perception range of the decoder can only be 8 (2048 / 256) frame images; if the number of tokens occupied by a single frame image is now 64 through the merging of the token merging units, the perception range of the decoder can be 32 (2048 / 64) frame images, which is 4 times higher than the original perception range, greatly improving the perception range of the converter model, allowing the converter model to make judgments within a larger perception range, making it easier to make correct judgments on video classification, and effectively improving the efficiency and accuracy of the converter model in video classification.

[0075] Step S208 , after the converter model reaches the preset accuracy, the trained converter model is obtained.

[0076] Specifically, the classification results of the training image sequence can be annotated in advance, and the results output by the converter model can be compared with the annotated information for training. When the error of the judgment result output by the converter model drops to a reasonable range, the converter model is considered to have achieved the preset accuracy, and the trained converter model is obtained.

[0077] After obtaining the trained converter model, the trained converter model can be used for video classification in actual business.

[0078] In an optional embodiment, if Fig. 9 As shown, after step S208, that is, after obtaining the trained converter model, the video classification method of the embodiment of the present application may further include: Step S300 , obtaining a target video, extracting frame images based on the target video and forming a target image sequence, wherein the target image sequence includes N frame images.

[0079] Step S302: input the target image sequence into the trained converter model, and divide each frame image in the target image sequence into a plurality of target small blocks according to the trained converter model.

[0080] Step S304: Utilize the token merging unit to merge some of the target small blocks according to the similarity of the target small blocks, and obtain a second token after the target image sequence is updated.

[0081] Step S306: input the second token into the decoder of the trained converter model, and use the decoder to obtain a classification result of the target image sequence based on the second token.

[0082] Step S308: obtaining a classification result of the target video based on the classification result of the target image sequence.

[0083] The target video may be a video to be classified. The service platform may obtain the target video at a designated location, and then extract the frame images of the target video through a programming language or an existing video editing software to form a target image sequence. Among them, the target image sequence may be multiple, that is, multiple target image sequences are formed by extracting frame images from the target video multiple times. When there are multiple target image sequences, the classification result of the target video is obtained based on the classification result of the target image sequence in step S308, and the classification result of the target video may be obtained by synthesizing the classification results of multiple target image sequences. For example, if the classification result of the target image sequence is a classification score, after obtaining the classification score of each target image sequence, all classification scores are averaged to obtain the classification result of the target video. Of course, when there is one target image sequence, step S308 may be to directly use the classification result of the target image sequence as the classification result of the target video.

[0084] In this embodiment, a target image sequence is formed by extracting frame images based on a target video, and the target image sequence is input into a trained converter model. Each frame image in the target image sequence is segmented into a number of target small blocks according to the trained converter model. A token merging unit is used to merge some target small blocks according to the similarity of the target small blocks to obtain an updated second token of the target image sequence. The second token is input into a decoder of the trained converter model, and the decoder is used to obtain a classification result of the target image sequence based on the second token. The classification result of the target video is obtained based on the classification result of the target image sequence. The target video can be accurately classified through the larger perception range of the trained converter model, thereby improving the efficiency and accuracy of video classification.

[0085] Embodiment 2 Fig.10 The block diagram of the video classification device according to the second embodiment of the present application is schematically shown. The device can be divided into one or more program modules, one or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. Fig.10 As shown, the device 800 may include an acquisition module 810, a segmentation module 820, a merging module 830, a classification module 840, and a determination module 850, wherein: An acquisition module 810 is used to acquire a training data set, where the training data set includes a plurality of training image sequences, and each of the training image sequences includes N frame images; A segmentation module 820 is used to input the training image sequence into a converter model, and use the converter model to segment each frame image in the training image sequence into a plurality of small blocks, wherein the encoder in the converter model includes a plurality of converter modules, at least some of the converter modules include a token merging unit, and each of the small blocks corresponds to a token of the converter module; A merging module 830, configured to use the token merging unit to merge some of the small blocks according to the similarity of the small blocks, to obtain an updated first token of the training image sequence; A classification module 840, configured to input the first token into a decoder of the converter model, and obtain a classification result of the training image sequence based on the first token using the decoder; The determination module 850 is used to obtain a trained converter model after the converter model reaches a preset accuracy.

[0086] In an optional embodiment, the device 800 is further used for: Acquire a target video, extract frame images based on the target video and form a target image sequence, wherein the target image sequence includes N frame images; Inputting the target image sequence into the trained converter model, and dividing each frame image in the target image sequence into a plurality of target small blocks according to the trained converter model; Utilizing the token merging unit to merge some of the target small blocks according to the similarity of the target small blocks, to obtain a second token after the target image sequence is updated; Inputting the second token into a decoder of the trained converter model, and using the decoder to obtain a classification result of the target image sequence based on the second token; A classification result of the target video is obtained based on the classification result of the target image sequence.

[0087] In an optional embodiment, the device 800 is further used for: Calculating the key vector of the small block and the dot product between the key vectors using the attention mechanism of the transformer module; The token merging unit is used to determine the similarity between the small blocks using cosine similarity according to the dot product.

[0088] In an optional embodiment, the device 800 is further used for: Determine the similarity between the small blocks in each frame image by using the token merging unit; When the similarity between two or more small blocks reaches a preset threshold, the token merging unit is used to merge the small blocks whose similarity reaches the preset threshold to obtain a new token.

[0089] In an optional embodiment, the device 800 is further used for: Using the token merging unit to sum or average the features of the small blocks whose similarity reaches the preset threshold; The token merging unit is used to obtain a new token according to the summation or averaging result.

[0090] In an optional embodiment, the merging module 830 is further configured to: Using the token merging unit to divide all tokens of the current frame image into two sets: a first set and a second set; Utilizing the token merging unit to calculate the similarity between each token in the first set and all tokens in the second set, and determining K token pairs with the highest similarity; Using the token merging unit to perform weighted averaging on the K token pairs to obtain K new tokens; The token merging unit is used to obtain the updated token of the current frame image according to the K new tokens and the remaining tokens, and the remaining tokens are the remaining tokens in the first set and the second set.

[0091] In an optional embodiment, each of the converter modules includes the token merging unit.

[0092] Embodiment 3 Fig.11 The schematic diagram of the hardware architecture of a computer device 10000 suitable for implementing the video classification method according to the third embodiment of the present application is shown schematically. In some embodiments, the computer device 10000 may be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 may be a rack server, a blade server, a tower server, or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. Fig.11 As shown, the computer device 10000 includes but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other through a system bus. Among them: The memory 10010 includes at least one type of computer-readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 can be an internal storage module of the computer device 10000, such as a hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 can also be an external storage device of the computer device 10000, such as a plug-in hard disk equipped on the computer device 10000, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 10010 can also include both the internal storage module of the computer device 10000 and its external storage device. In this embodiment, the memory 10010 is generally used to store an operating system and various application software installed in the computer device 10000, such as program codes of a video classification method, etc. In addition, the memory 10010 can also be used to temporarily store various data that have been output or are to be output.

[0093] In some embodiments, the processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication with the computer device 10000. In this embodiment, the processor 10020 is used to run the program code stored in the memory 10010 or process data.

[0094] The network interface 10030 may include a wireless network interface or a wired network interface, and the network interface 10030 is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0095] It should be pointed out that Fig.11 Only a computer device having components 10010 - 10030 is shown, but it should be understood that implementation of all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0096] In this embodiment, the video classification method stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of the present application.

[0097] Embodiment 4 The embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the video classification method in the embodiment are implemented.

[0098] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of a computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the computer-readable storage medium can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various application software installed on the computer device, such as the program code of the video classification method in the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or are to be output.

[0099] Embodiment 5 An embodiment of the present application also provides a computer program product, including a computer program, which implements the method in the above embodiment when executed by a processor.

[0100] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices, optionally, they can be implemented by executable program codes of computer devices, so that they can be stored in a storage device and executed by the computer device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0101] It should be noted that the above are only preferred embodiments of the present application, and the patent protection scope of the present application is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A video classification method, characterized in that: The method comprises: Acquire a training data set, wherein the training data set includes a plurality of training image sequences, and each of the training image sequences includes N frame images; Input the training image sequence into a converter model, and use the converter model to divide each frame image in the training image sequence into a plurality of small blocks, wherein the encoder in the converter model includes a plurality of converter modules, at least some of the converter modules include a token merging unit, and each of the small blocks corresponds to a token of the converter module; Utilizing the token merging unit to merge some of the small blocks according to the similarity of the small blocks, to obtain an updated first token of the training image sequence; Inputting the first token into a decoder of the converter model, and using the decoder to obtain a classification result of the training image sequence based on the first token; After the converter model reaches a preset accuracy, a trained converter model is obtained.

2. The method according to claim 1, characterized in that After obtaining the trained converter model, the method further includes: Acquire a target video, extract frame images based on the target video and form a target image sequence, wherein the target image sequence includes N frame images; Inputting the target image sequence into the trained converter model, and dividing each frame image in the target image sequence into a plurality of target small blocks according to the trained converter model; Utilizing the token merging unit to merge some of the target small blocks according to the similarity of the target small blocks, to obtain a second token after the target image sequence is updated; Inputting the second token into a decoder of the trained converter model, and using the decoder to obtain a classification result of the target image sequence based on the second token; A classification result of the target video is obtained based on the classification result of the target image sequence.

3. The method according to claim 1, characterized in that The method further comprises: Calculating the key vector of the small block and the dot product between the key vectors using the attention mechanism of the transformer module; The token merging unit is used to determine the similarity between the small blocks using cosine similarity according to the dot product.

4. The method according to claim 3, characterized in that The step of merging some of the small blocks according to the similarity of the small blocks by using the token merging unit includes: Determine the similarity between the small blocks in each frame image by using the token merging unit; When the similarity between two or more small blocks reaches a preset threshold, the token merging unit is used to merge the small blocks whose similarity reaches the preset threshold to obtain a new token.

5. The method according to claim 4, characterized in that When the similarity between two or more small blocks reaches a preset threshold, merging the small blocks whose similarity reaches the preset threshold to obtain a new token includes: Using the token merging unit to sum or average the features of the small blocks whose similarity reaches the preset threshold; The token merging unit is used to obtain a new token according to the summation or averaging result.

6. The method according to any one of claims 1 to 5, characterized in that: The step of merging some of the small blocks according to the similarity of the small blocks by using the token merging unit includes: Using the token merging unit to divide all tokens of the current frame image into two sets: a first set and a second set; Utilizing the token merging unit to calculate the similarity between each token in the first set and all tokens in the second set, and determining K token pairs with the highest similarity; Using the token merging unit to perform weighted averaging on the K token pairs to obtain K new tokens; The token merging unit is used to obtain the updated token of the current frame image according to the K new tokens and the remaining tokens, and the remaining tokens are the remaining tokens in the first set and the second set.

7. The method according to any one of claims 1 to 5, characterized in that: Each of the converter modules includes the token merging unit.

8. A video classification device, characterized in that: The device comprises: An acquisition module is used to acquire a training data set, wherein the training data set includes a plurality of training image sequences, and each of the training image sequences includes N frame images; A segmentation module, used for inputting the training image sequence into a converter model, and using the converter model to segment each frame image in the training image sequence into a plurality of small blocks, wherein the encoder in the converter model includes a plurality of converter modules, at least some of the converter modules include a token merging unit, and each of the small blocks corresponds to a token of the converter module; a merging module, configured to use the token merging unit to merge some of the small blocks according to the similarity of the small blocks, so as to obtain an updated first token of the training image sequence; A classification module, configured to input the first token into a decoder of the converter model, and obtain a classification result of the training image sequence based on the first token using the decoder; The determination module is used to obtain a trained converter model after the converter model reaches a preset accuracy.

9. A computer device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images