Video tag recognition method, device, computer equipment and storage medium

Abstract

The present application relates to a method, device, computer equipment, storage medium and computer program product for identifying video tags. The method includes: determining the global information of the video frame based on the video frame and the candidate tags in the video; determining the local information of the video frame according to the image block in the video frame and the candidate tags; for each video frame in the video, performing spatiotemporal aggregation on the global information of each video frame and the local information of each video frame to obtain spatiotemporal global information and spatiotemporal local information; fusing the spatiotemporal global information and the spatiotemporal local information to obtain fused information; and identifying the tag to which the video belongs in the candidate tags based on the fused information. The use of this method can effectively improve the recognition efficiency and accuracy of video tags.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a method, device, computer equipment, storage medium and computer program product for identifying a video tag. Background Art

[0002] Video tag recognition is an important part of video content features. Automatically generating tags for massive videos can provide video content features of different granularities for the downstream content distribution link, improve the efficiency of content distribution, and significantly reduce the cost of manual content review.

[0003] In traditional technical solutions, the most common method is to annotate a large number of samples to train models to predict video labels. Specifically, a candidate label set is first defined, and a batch of data is sampled from the business data for manual labeling. The auditor watches each video and selects a label from the candidate label set as the label associated with the video. However, due to human factors, this label recognition process usually takes a long time and has a low accuracy rate, resulting in low recognition efficiency and accuracy of video labels. Summary of the invention

[0004] Based on this, it is necessary to provide a method, device, computer equipment, computer-readable storage medium and computer program product for identifying video tags in response to the above technical problems, which can effectively improve the recognition efficiency and accuracy of video tags.

[0005] In a first aspect, the present application provides a method for identifying a video tag, the method comprising:

[0006] Determining global information of the video frame based on the video frame and the candidate tags in the video;

[0007] Determining local information of the video frame according to the image block in the video frame and the candidate tag;

[0008] For each video frame in the video, respectively perform spatiotemporal aggregation on the global information of each video frame and the local information of each video frame to obtain spatiotemporal global information and spatiotemporal local information;

[0009] fusing the spatiotemporal global information and the spatiotemporal local information to obtain fused information;

[0010] Among the candidate tags, the tag to which the video belongs is identified based on the fusion information.

[0011] In a second aspect, the present application further provides a video tag recognition device, the device comprising:

[0012] A first determination module, configured to determine global information of a video frame based on a video frame and a candidate tag in the video;

[0013] A second determination module, configured to determine local information of the video frame according to the image block in the video frame and the candidate label;

[0014] An aggregation module, for performing spatiotemporal aggregation on the global information of each video frame and the local information of each video frame in the video, to obtain spatiotemporal global information and spatiotemporal local information;

[0015] A fusion module, used for fusing the spatiotemporal global information and the spatiotemporal local information to obtain fused information;

[0016] An identification module is used to identify the tag to which the video belongs from the candidate tags based on the fusion information.

[0017] In one embodiment, the device further comprises:

[0018] A processing module, used for performing temporal smoothing processing on the global information of each of the video frames to obtain smoothed global information; performing temporal smoothing processing on the local information of each of the video frames to obtain smoothed local information;

[0019] The aggregation module is further used to perform spatiotemporal aggregation on the smoothed global information of each video frame to obtain spatiotemporal global information; and to perform spatiotemporal aggregation on the smoothed local information of each video frame to obtain spatiotemporal local information.

[0020] In one embodiment, the global information includes global similarity information, and the apparatus further includes:

[0021] An encoding module, used to encode the video frame in the video and the first description information in the text sequence respectively, to obtain the first video feature of the video frame and the description feature of the first description information; the first description information is video description information formed based on the candidate tags in the tag set;

[0022] The first determination module is further used to determine the global similarity information of the video frame based on the first video feature and the description feature; the global similarity information includes a similarity value between the video frame and the candidate tag.

[0023] In one embodiment, the local information includes local similarity information, and the apparatus further includes:

[0024] An encoding module, used for encoding the image block of the video frame to obtain a second video feature of the image block;

[0025] The second determination module is further used to determine local similarity information of the video frame based on the second video feature and the description feature; the local similarity information includes a similarity value between an image block of the video frame and the candidate tag.

[0026] In one embodiment, the video frame includes a sampled video frame; the apparatus further includes:

[0027] A decoding module, used for decoding the video to obtain a video frame sequence;

[0028] The sampling module is used to determine a sampling interval, sample video frames in the video frame sequence according to the sampling interval, and obtain sampled video frames of the video.

[0029] In one embodiment, the fusion module is further used to determine the mean between the elements in the spatiotemporal global information and the elements in the spatiotemporal local information to obtain fused information; or, to perform weighted summation of the elements in the spatiotemporal global information and the elements in the spatiotemporal local information to obtain fused information.

[0030] In one embodiment, the device further comprises:

[0031] A segmentation module is used to segment the video frames in the video into the same number of image blocks; or to segment the video frames in the video in a pyramidal segmentation manner to obtain corresponding image blocks; wherein the number of image blocks of different video frames is different.

[0032] In one embodiment, the recognition module is also used to determine the multi-label distribution score of the video based on the fusion information; or, use the fusion information as the multi-label distribution score of the video; and among the candidate tags, determine the tag to which the video belongs according to the multi-label distribution score.

[0033] In one embodiment, the device further comprises:

[0034] A training module is used to use the labels whose multi-label distribution scores meet the scoring conditions as positive sample labels; use the labels whose multi-label distribution scores do not meet the scoring conditions as negative sample labels; use the multiple videos as training samples, and train the initial classification model based on the training samples, the positive sample labels and the negative sample labels to obtain a classification model.

[0035] In one embodiment, the device further comprises:

[0036] An acquisition module, used to acquire the audio of the training sample and extract audio features from the audio; acquire second description information of the training sample and extract description features from the second description information; the second description information is video description information of the training sample;

[0037] The training module is also used to encode the video frame samples in the training samples and the image block samples in the video frame samples respectively to obtain the third video features of the video frame samples and the fourth video features of the image block samples; based on the third video features, the fourth video features, the description features of the second description information and the audio features, the initial classification model is trained so that the similarity values ​​between the predicted labels of the training samples and the positive sample labels and the negative sample labels reach preset similarity conditions to obtain the classification model.

[0038] In one embodiment, the acquisition module is used to extract audio features in the audio through an audio feature extractor; encode the second description information through a target bidirectional encoder to obtain description features of the second description information;

[0039] The training module is further used to encode the video frame samples in the training samples and the image block samples in the video frame samples through a visual feature encoder to obtain the third video features of the video frame samples and the fourth video features of the image block samples.

[0040] In a third aspect, the present application further provides a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the video tag recognition method when executing the computer program.

[0041] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the video tag recognition method are implemented.

[0042] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the video tag recognition method are implemented.

[0043] The above-mentioned video tag recognition method, device, computer equipment, storage medium and computer program product determine the global information of the video frame based on the video frame and candidate tags in the video, and determine the local information of the video frame according to the image blocks and candidate tags in the video frame; for each video frame in the video, the global information of each video frame and the local information of each video frame are respectively spatiotemporally aggregated, so as to aggregate the global information of different video frames together, and aggregate the local information of different video frames together, and fully consider the global information and local information of the video space and time to obtain the spatiotemporal global information and the spatiotemporal local information; in addition, the spatiotemporal global information and the spatiotemporal local information are fused, so as to fuse the global information and local information of the spatiotemporal dimension of the video, so that even in the face of multi-tag recognition of the video, the tag to which the video belongs can be accurately identified based on the fused information, thereby effectively improving the recognition accuracy of the video tag; in addition, the influence of human factors can be effectively avoided, and the recognition efficiency of the video tag can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] FIG1 is an application environment diagram of a method for identifying video tags in one embodiment;

[0045] FIG2 is a schematic flow chart of a method for identifying video tags in one embodiment;

[0046] FIG3 is a schematic diagram of video frame segmentation in one embodiment;

[0047] FIG4 is a schematic flow chart of a method for identifying video tags in another embodiment;

[0048] FIG5 is a schematic diagram of model training and label recognition in one embodiment;

[0049] FIG6 is a schematic diagram of a process of model training in one embodiment;

[0050] FIG7 is a schematic diagram of a network structure used in model training in one embodiment;

[0051] FIG8 is a structural block diagram of a device for identifying video tags in one embodiment;

[0052] FIG9 is a structural block diagram of a device for identifying video tags in another embodiment;

[0053] FIG. 10 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] It should be noted that in the following description, the terms "first, second and third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first, second and third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0056] Before describing the embodiments of the present application, the technology involved in the present application is introduced as follows:

[0057] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0058] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, pre-trained models are also called large models and basic models. After fine-tuning, they can be widely used in downstream tasks in various major directions of artificial intelligence. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0059] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further processing graphics to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Large model technology has brought important changes to the development of computer vision technology. Pre-trained models in the visual field such as Swin-Transformer, ViT, V-MOE and MAE can be quickly and widely applied to downstream specific tasks after fine tuning. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, three-dimensional (3D) technology, virtual reality, augmented reality, simultaneous positioning and mapping, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0060] The video tag recognition method provided in the embodiment of the present application can be applied in the application environment shown in FIG1 . In which, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers.

[0061] The server 104 determines the global information of the video frame based on the video frame and the candidate tags in the video; determines the local information of the video frame based on the image blocks in the video frame and the candidate tags; for each video frame in the video, respectively, performs spatiotemporal aggregation on the global information of each video frame and the local information of each video frame to obtain spatiotemporal global information and spatiotemporal local information; fuses the spatiotemporal global information and the spatiotemporal local information to obtain fused information; and identifies the tag to which the video belongs in the candidate tags based on the fused information. After obtaining the tag of the video, the server can apply the tag to the model training, and can also use the tag to recommend a video to the terminal 102.

[0062] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, an IoT device, and a portable wearable device. The IoT device may be a smart speaker, a smart TV, a smart air conditioner, and a smart car device, etc. The portable wearable device may be a smart watch, a smart bracelet, and a head-mounted device, etc.

[0063] Server 104 can be an independent physical server or a service node in a blockchain system. Each service node in the blockchain system forms a peer-to-peer network. The peer-to-peer protocol is an application layer protocol running on the Transmission Control Protocol (TCP). In addition, server 104 can also be a server cluster composed of multiple physical servers, which can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0064] The terminal 102 and the server 104 may be connected via Bluetooth, USB (Universal Serial Bus) or a network or other communication connection method, and this application does not impose any limitation thereto.

[0065] In one embodiment, as shown in FIG. 2 , a method for identifying a video tag is provided. The method may be executed by a computer device, such as the server or terminal in FIG. 1 , or the server and the terminal in collaboration. The method is described by taking the server in FIG. 1 as an example, and includes the following steps:

[0066] S202, determining global information of the video frame based on the video frame and the candidate tags in the video.

[0067] The video can be any type of video on the video platform, including live video and non-live video. According to the length of time, the non-live video can be short video, medium video and long video; according to the type, the non-live video can be film and television video, information video, conference video, teaching video and life sharing video.

[0068] A video frame can be an image of a corresponding frame in a video, so the video frame can also be called an image. For example, assuming that the frame rate of a video is 30 and the duration is 1 second (s), the video has a total of 30 frames of images, and the 30 frames of images are all the video frames of the video. In order to reduce the amount of calculation, the server can use part of the video frames of the video for video tag recognition, that is, the video frames in the video used to determine the global information are the video frames obtained after sampling.

[0069] The candidate tag may be a candidate video tag, which is used to indicate the category to which a video, a video frame, or an object in a video frame belongs. The number of the candidate tags may be n, where n is a positive integer greater than or equal to 1. For example, the candidate tags are people, cats, dogs, mobile phones, and computers. When performing tag recognition on a video frame, the tag corresponding to the object in the video frame can be identified in the candidate tags based on the relevant information of the video; for another example, the candidate tags are sports, entertainment, emotions, actions, and plots. When performing tag recognition on a video, the tag corresponding to the video can be identified in the candidate tags based on the relevant information of the video.

[0070] The global information may be the association information between the objects in the entire video frame and the candidate tags. In some specific application scenarios, the global information may be global similarity information, global tag probability information or global score information. The global similarity information may be a global similarity vector, including the similarity values ​​between each object in the video frame and the candidate tags; the global tag probability information may be a global tag probability vector, including the distribution probability of each object in the video frame in the candidate tags, such as the probability that each object in the video frame belongs to multiple candidate video tags; the global score information may be a global score vector, including the predicted score of each object in the video frame on the candidate tags.

[0071] In one embodiment, considering that the objects in each video frame in a short period of time change little, such as the objects in the video frames within one second are basically the same, the server can perform video tag recognition by sampling. Therefore, the server can sample multiple video frames from the video at a specific sampling interval, and then determine the global information of the multiple video frames based on the multiple video frames and the candidate tags.

[0072] Specifically, the video frames include sampled video frames (i.e., sampled video frames); the server can decode the video to obtain a video frame sequence; determine a sampling interval, sample the video frames in the video frame sequence according to the sampling interval, and obtain sampled video frames of the video, thereby determining global information of the sampled video frames based on the sampled video frames and candidate tags.

[0073] The video frame sequence may be a sequence of video frames obtained after decoding the video. The sampling interval may refer to the number of interval frames when sampling, and the sampling interval may be determined according to the frame rate of the video. The larger the frame rate, the larger the sampling interval may be. For example, for a video with a frame rate of 30, sampling may be performed every 30 frames or 60 frames; for another example, for a video with a frame rate of 60, sampling may be performed every 60 frames or 90 frames.

[0074] In one embodiment, the server may perform similarity calculation on the video frames and candidate tags in the video to obtain global similarity information. For example, the server may perform similarity calculation on the video frames and candidate tags in the video through a contrastive language-image pre-training (CLIP) model to obtain global similarity information, which is recorded as , t represents the tth video frame. It should be noted that, through the above similarity calculation, each of the above video frames has a corresponding global similarity information.

[0075] Among them, the CLIP model is a multimodal learning model built on the CLIP algorithm, which can understand and process image and text data, and use contrastive learning to improve the performance of the model. The CLIP model is trained with a large number of images and related description information, and learns to map the description information and image (i.e., video frame) representations to the same semantic space, so that description information and images with similar meanings have similar representations in the semantic space. For example, the CLIP model can link the representation of the word "dog" with the representation of an image of a dog to obtain the corresponding similarity value (such as cosine similarity). During the training process, the CLIP model needs to distinguish between positive examples (i.e., correct description information that matches the image) and negative examples (i.e., description information that does not match the image), by maximizing the similarity between positive examples and minimizing the similarity between negative examples.

[0076] S204: Determine local information of the video frame according to the image blocks and the candidate tags in the video frame.

[0077] The image blocks may be image blocks obtained by segmenting a video frame. For example, a video frame may be segmented into 3×3 blocks to obtain 9 image blocks.

[0078] The local information may be the association information between the objects in the image block and the candidate tags. The local information has the same dimension as the global information mentioned above and is related to the number of candidate tags. In some specific application scenarios, the local information may be local similarity information, local label probability information or local score information. The local similarity information may be a local similarity vector, including the similarity values ​​between each object in the image block and the candidate tags. The local label probability information may be a local label probability vector, including the distribution probability of each object in the image block in the candidate tags, such as the probability that each object in the image block belongs to multiple candidate video tags; the local score information may be a local score vector, including the predicted scores of each object in the image block on the candidate tags.

[0079] In one embodiment, before S204, the server may divide the video frames in the video into the same number of image blocks; or, divide the video frames in the video in a pyramid-like manner to obtain corresponding image blocks; wherein the number of image blocks in different video frames is different.

[0080] Among them, the number of image blocks of each video frame obtained by the above segmentation method can be the same, or can be increased in geometric proportion. For example, the video frames in the video are segmented in the same segmentation method, so that the number of image blocks of each video frame is the same, such as each video frame is segmented in a 3×3 manner, as shown in Figure 3; in addition, the video frames in the video can also be segmented in a pyramidal segmentation method, such as the first video frame is segmented in a 2×2 manner to obtain 4 image blocks; the second video frame is segmented in a 3×3 manner to obtain 9 image blocks, and so on.

[0081] It should be noted that, when segmenting, each image block may have overlapping content or may not have overlapping content.

[0082] In one embodiment, the server may perform similarity calculation on the image blocks in the video frame and the candidate tags to obtain local similarity information. For example, the server may perform similarity calculation on the image blocks in the video frame and the candidate tags through the CLIP model to obtain local similarity information, which is recorded as , t represents the t-th video frame, and i represents the i-th image block of the t-th video frame.

[0083] It should be noted that, through the above similarity calculation, each of the above video frames has corresponding multiple local similarity information. For example, if each video frame is divided into 9 image blocks, each video frame has corresponding 9 local similarity information.

[0084] S206 , for each video frame in the video, perform spatiotemporal aggregation on the global information of each video frame and the local information of each video frame to obtain spatiotemporal global information and spatiotemporal local information.

[0085] The spatiotemporal global information may be global information obtained by performing spatiotemporal aggregation on global information. For a video, after performing spatiotemporal aggregation, a spatiotemporal global information may be obtained. The spatiotemporal local information may be local information obtained by performing spatiotemporal aggregation on local information. For a video, after performing spatiotemporal aggregation, a spatiotemporal local information may be obtained. The spatiotemporal global information and the spatiotemporal local information have the same dimension.

[0086] It should be pointed out that the video frames in the above video may be all the video frames in the video, or may be the video frames obtained by sampling in the video.

[0087] In one embodiment, for each video frame in the video, the server performs spatiotemporal aggregation on the global information of each video frame, such as performing maximum pooling processing on the global information of each video frame in the spatiotemporal dimension to obtain spatiotemporal global information.

[0088] Similarly, the spatiotemporal aggregation method of local information may include: for each video frame in the video, the server performs spatiotemporal aggregation on the local information of each video frame, such as performing maximum pooling processing on the local information of each video frame in the spatiotemporal dimension to obtain spatiotemporal local information.

[0089] Among them, the above-mentioned maximum pooling processing method may be different from the conventional pooling method. Considering that an object may appear in the first 3 seconds of the video, then leave the video screen, and will not appear in the video screen at a subsequent time, when performing the maximum pooling processing, a larger value is selected for each position in the global information.

[0090] In one embodiment, before S206, the server may first perform temporal smoothing on the global information and the local information, and then perform spatiotemporal aggregation on the smoothed global information and local information. Specifically: the server performs temporal smoothing on the global information of each video frame to obtain smoothed global information; performs temporal smoothing on the local information of each video frame to obtain smoothed local information; then, performs spatiotemporal aggregation on the smoothed global information of each video frame to obtain spatiotemporal global information; and performs spatiotemporal aggregation on the smoothed local information of each video frame to obtain spatiotemporal local information.

[0091] Among them, time series smoothing can suppress noise and help improve the accuracy of video tag recognition.

[0092] For adjacent video frames of a video, the video contents are usually highly similar, and the positions of objects in the video will not be too large between adjacent video frames. For example, an object usually does not instantly move from a certain position in a video frame to a farther position, such as usually does not instantly move from the lower left corner of video frame t to the upper right corner of video frame t+1.

[0093] The process of temporally smoothing the global information and the local information may include: the server may perform average pooling processing on the global information of each video frame on the time axis to obtain smoothed global information; and, perform average pooling processing on the local information of each video frame on the time axis to obtain smoothed local information.

[0094] S208, fusing the spatiotemporal global information and the spatiotemporal local information to obtain fused information.

[0095] The elements in the fused information may be the mean or weighted sum of the elements in the spatiotemporal global information and the elements in the spatiotemporal local information. When the global information and the local information are global similarity information and local similarity information, respectively, the fused information may be fused similarity information; when the global information and the local information are global label probability information and local label probability information, respectively, the fused information may be fused label probability information; when the global information and the local information are global score information and local score information, respectively, the fused information may be fused score information.

[0096] In one embodiment, the server determines the mean between the elements in the spatiotemporal global information and the elements in the spatiotemporal local information to obtain fused information; or, performs weighted summation of the elements in the spatiotemporal global information and the elements in the spatiotemporal local information to obtain fused information.

[0097] S210: Identify the tag to which the video belongs among the candidate tags based on the fusion information.

[0098] Among them, the label to which the video belongs can be a pseudo-label of the video, a video label of the same level, or a video label of a different level, such as video labels of different levels. In addition, the ranges covered by video labels of different levels are different. For example, if the first level is higher than the second level, the range covered by the video label of the first level is greater than the range covered by the video label of the second level. It should be pointed out that the video label recognition method of the present application can identify the labels of multiple videos (such as all videos on a video platform). After obtaining the labels to which these videos belong, they can be used for training the initial classification model, so that manual labeling is not required during the training phase, effectively avoiding a lot of time and labor costs consumed by labeling. In addition, after obtaining the labels to which these videos belong, they can also be used in the downstream content distribution link of the video (such as recommendation systems and content operations) to improve content distribution efficiency, while greatly reducing the cost of manual content review.

[0099] There are multiple candidate tags, and the server can use the fusion information to identify the tag to which the video belongs from the candidate tags. It should be noted that there can be multiple tags to which the video belongs, and the number is less than or equal to the number of candidate tags.

[0100] In one embodiment, the server determines the multi-label distribution score of the video based on the fusion information; or, uses the fusion information as the multi-label distribution score of the video; and determines the label to which the video belongs among the candidate labels according to the multi-label distribution score.

[0101] When the fusion information is fusion score information, the server can determine the label to which the video belongs based on the elements in the fusion score information. The fusion score information can be a multi-label distribution score (i.e., the distribution score of the video on all candidate labels). For example, each element in the fusion score information is the maximum score value selected from the score values ​​between the candidate labels of each video frame of the video, so the label to which the video belongs can be determined based on each element in the fusion score information. In addition, when determining the label to which the video belongs, it is also possible to first determine whether the elements in the fusion score information meet the recognition conditions (such as whether they are greater than or equal to the score threshold), and use the candidate label corresponding to the element that meets the recognition conditions as the label of the video.

[0102] When the fused information is fused similarity information, the server can determine the label to which the video belongs based on the elements in the fused similarity information. For example, each element in the fused similarity information is the maximum similarity value selected from the similarity values ​​between the candidate labels of each video frame of the video, so the label to which the video belongs can be determined based on each element in the fused similarity information; in addition, the multi-label distribution score of the video can also be determined based on the fused similarity information, and then the label to which the video belongs can be determined based on the multi-label distribution score. In addition, when determining the label to which the video belongs, it is also possible to first determine whether the elements in the fused similarity information meet the recognition conditions (such as whether they are greater than or equal to the similarity threshold), and use the candidate label corresponding to the element that meets the recognition conditions as the label of the video.

[0103] When the fusion information is fused label probability information, the server can determine the label to which the video belongs based on the elements in the fused label probability information. For example, each element in the fused label probability information is the maximum label probability value selected from the label probability values ​​between the candidate labels of each video frame of the video. Therefore, the label to which the video belongs can be determined based on each element in the fused label probability information; in addition, the multi-label distribution score of the video can also be determined based on the fused label probability information, and then the label to which the video belongs can be determined based on the multi-label distribution score. In addition, when determining the label to which the video belongs, it is also possible to first determine whether the elements in the fused label probability information meet the recognition conditions (such as whether they are greater than or equal to the probability threshold), and use the candidate label corresponding to the element that meets the recognition conditions as the label of the video.

[0104] In order to better understand the above technical solution, it is described here in conjunction with Figures 4 and 5, as follows:

[0105] As shown in FIG4 , in the process of identifying the pseudo label of a video, the CLIP model needs to be used to calculate the global information and the local information, and then the global information and the local information are subjected to spatiotemporal aggregation and information fusion, and then the pseudo label of the video can be obtained according to the fused information. Next, the CLIP model, the calculation of the global information and the local information, and the spatiotemporal aggregation and information fusion of the global information and the local information are described as follows:

[0106] 1) CLIP model:

[0107] The CLIP model is a model of multimodal learning technology. It improves the performance of the model by contrastive training, so that the model learns to map text and image representations to the same semantic space. Text and images with similar meanings have similar representations in this semantic space, so the model can quickly understand and process image and text data. For example, the model can associate the representation of the word "dog" with the representation of an image of a dog. During the training process, the model needs to distinguish between positive examples (i.e., the image matches the description information) and negative examples (i.e., the image does not match the description information), by maximizing the similarity between positive examples and minimizing the similarity between negative examples.

[0108] Specifically, given an image and text, the goal of the model is to maximize the similarity between matching images and texts, while minimizing the similarity between unmatched images and texts. In addition, the CLIP model uses the InfoNCE loss function to measure the similarity value. For a pair of positive sample images and text , as well as K negative sample images and texts, the InfoNCE loss function is defined as follows:

[0109]

[0110] where τ is a hyperparameter used to control the distribution of similarity values, Representing images The characteristic vector of Represents text A positive sample image can refer to a label that matches the image content, and the image at this time is a positive sample image; a negative sample image can refer to a label that does not match the image content, and the image at this time is a negative sample image.

[0111] After the model training is completed, it can be used to calculate the similarity between the image and the text. First, construct description information for the candidate label, such as using the following template to construct description information:

[0112] "an image containing {label}";

[0113] Among them, {label} is a placeholder. Fill in the label to get the description information of each label, and get the text feature t through the text feature encoder. Assuming that there are C labels in the label set, the feature of the label set can be expressed as { , ,…, }.

[0114] The image passes through the visual feature encoder image feature encoder in the model to obtain feature v. The similarity value between the image and each label is:

[0115]

[0116]

[0117] Among them, τ is the hyperparameter learned by CLIP, Represents the similarity value between image and label i, and is denoted as S as a whole.

[0118] 2) Use the CLIP model to calculate the similarity between the video frame and multiple labels to obtain global and local information. The calculation process is as follows:

[0119] Considering that the conventional CLIP model is only trained for a single object in each video frame, it is not suitable for multi-label classification. The goal of this application is to identify all labels of the video frame. Therefore, the idea of ​​fusing global information and local information is adopted to identify multiple labels. The specific approach is as follows:

[0120] S1, the video can be sampled in a uniform interval, such as one frame every 0.2 seconds, to obtain a video frame sequence { , ,…, , ,…};

[0121] S2, processes each frame separately, based on the video frame For example, the input here is a complete video frame, and the CLIP model calculates the global information of the video frame, which is recorded as ;

[0122] S3, video frame Get the block { ,t}, can be evenly divided into N×N image blocks, such as the 3×3 segmentation shown in Figure 3, to obtain 9 image blocks. The CLIP model is used to calculate a local information for each image block separately, which is recorded as , t represents the tth video frame (i.e., video frame ), i represents the i-th image block of the t-th video frame.

[0123] 3) Process the global and local information to identify the pseudo labels of the video, as follows:

[0124] The global and local information obtained in the above way may contain noise. In this case, it is necessary to suppress the noise and aggregate the spatiotemporal global and local information to obtain the pseudo label of the video in preparation for the next training process. This link has three main steps:

[0125] S1, time series smoothing: mainly performs average pooling on global information and local information on the time axis to suppress noise.

[0126] The design of the temporal smoothing process is based on two main assumptions: 1) the contents of adjacent video frames should be highly similar, and 2) the displacement of objects between adjacent video frames will not be too large. For example, there is generally no situation where an object teleports from the upper left corner of a video frame to other farther positions.

[0127] S2, spatiotemporal aggregation: After time series smoothing, max pooling is performed in the spatiotemporal dimension. A different pooling method from S1 is used here, that is, the maximum similarity value is selected from all global and local information. The corresponding calculation formula is as follows:

[0128]

[0129]

[0130] Among them, considering that an object may appear in the first 3 seconds of the video and then leave the video screen, it will not appear in the video screen at any subsequent time, so a larger similarity value should be selected.

[0131] S3, the global information and local information after spatiotemporal aggregation are fused in the following way:

[0132]

[0133] After completing the above steps, each video obtains a multi-label distribution score S, so that the multi-label distribution score S can be used to determine the pseudo label of the video.

[0134] In the above embodiment, the global information of the video frame is determined based on the video frame and the candidate tags in the video, and the local information of the video frame is determined according to the image blocks and the candidate tags in the video frame; for each video frame in the video, the global information of each video frame and the local information of each video frame are respectively spatiotemporally aggregated, so that the global information of different video frames are aggregated together, and the local information of different video frames are aggregated together, and the global information and local information of the video space and time are fully considered to obtain the spatiotemporal global information and the spatiotemporal local information; in addition, the spatiotemporal global information and the spatiotemporal local information are fused, so as to fuse the global information and local information of the spatiotemporal dimension of the video. Therefore, even in the face of multi-label recognition of the video, the label to which the video belongs can be accurately identified based on the fused information, thereby effectively improving the recognition accuracy of the video label; in addition, the influence of human factors can be effectively avoided, and the recognition efficiency of the video label can be effectively improved.

[0135] In one embodiment, as shown in FIG6 , the method further includes:

[0136] S602: labels whose multi-label distribution scores satisfy the scoring conditions are taken as positive sample labels.

[0137] The positive sample label may refer to a label that matches the video content, so the label is the actual label of the video, and the video at this time is a positive sample video. The score condition may refer to being greater than or equal to a score threshold.

[0138] S604: labels whose multi-label distribution scores do not meet the score conditions are taken as negative sample labels.

[0139] Among them, the negative sample label can refer to a label that does not match the video, so the label is not the actual label of the video. At this time, the video is a negative sample video.

[0140] For example, the server sets a score threshold, and uses the labels whose multi-label distribution scores are greater than or equal to the score threshold as positive sample labels, and the corresponding videos are positive sample videos; in addition, the labels whose multi-label distribution scores are less than the score threshold are used as negative sample labels, and the corresponding videos are negative sample videos.

[0141] S606, taking the multiple videos as training samples, training the initial classification model based on the training samples, positive sample labels and negative sample labels to obtain a classification model.

[0142] The initial classification model may be an untrained multi-label classification model for classifying videos, such as a Transformer model.

[0143] In one embodiment, the server may encode the training samples to obtain video features of the training samples, and train an initial classification model based on the video features of the training samples, positive sample labels, and negative sample labels to obtain a classification model.

[0144] When training the initial classification model, the server can train the initial classification model in combination with the audio of the training sample and the second description information. Specifically, the server trains the initial classification model based on the training sample, the positive sample label, the negative sample label, the audio of the training sample and the second description information to obtain the classification model.

[0145] The second description information is video description information of the training sample, such as description information of the positive sample video and the negative sample video. As an example, considering that the training sample is a video, the second description information can be the title, introduction or other description information of the video.

[0146] Specifically, the server obtains the audio of the training sample and extracts audio features from the audio; obtains the second description information of the training sample, which is the video description information of the training sample; encodes the video frame samples in the training sample, the image block samples in the video frame samples, and the second description information, respectively, to obtain the third video features of the video frame samples, the fourth video features of the image block samples, and the description features of the second description information; trains the initial classification model based on the third video features, the fourth video features, the description features, and the audio features, so that the similarity values ​​between the predicted labels of the training samples and the positive sample labels and the negative sample labels reach a preset similarity condition, and obtains the classification model.

[0147] During the training process, the loss value between the predicted label of the training sample and the positive sample label, as well as the loss value between the predicted label of the training sample and the negative sample label are calculated, and the loss value is used to optimize the parameters of the initial classification model until the model converges to obtain the final classification model.

[0148] It should be pointed out that in the process of parameter optimization, the similarity between the predicted label of the training sample and the positive sample label is maximized, such as the similarity is 1, or a value close to 1; in addition, the similarity between the predicted label of the training sample and the negative sample label is minimized, such as the similarity is 0, or a value close to 0, so that a classification model with high inference accuracy can be obtained. Among them, 1 means that the predicted label of the training sample is 100% similar to the positive sample label, and 0 means that the predicted label of the training sample is not similar to the negative sample label.

[0149] For audio feature extraction, the server may extract audio features from the audio through an audio feature extractor, such as using a Vggish model to extract audio features from the audio.

[0150] For feature extraction of video frame samples and image block samples in the training samples, the server may encode the video frame samples and image block samples in the training samples through a visual feature encoder to obtain a third video feature of the video frame samples and a fourth video feature of the image block samples. For example, a visual feature encoder (such as a CNN model or a ViT model, etc.) is used to extract the third video feature of the video frame samples from the video frame samples in the training samples, and to extract the fourth video feature of the image block samples from the image block samples in the video frame samples.

[0151] For feature extraction of the second description information, the server may encode the second description information through a target bidirectional encoder to obtain the description features of the second description information. For example, the BERT model may be used to extract an embedded representation of the second description information, and then the position encoding and type encoding may be added to the embedded representation to obtain the description features of the second description information.

[0152] In order to more clearly understand the above technical solution of the present application, it is described here in conjunction with FIG. 7, as follows:

[0153] After obtaining the multi-label distribution score S of the video through the above S1-S3 method, a threshold θ can be set. The labels whose multi-label distribution scores S exceed the threshold θ are regarded as positive sample labels, and the labels whose multi-label distribution scores S are less than the threshold θ are negative sample labels. In this way, the pseudo-label of the video is obtained, which can be used to train the multi-label classification model of the video.

[0154] For example, as shown in Figure 7, the idea of ​​video multimodal classification is adopted here, that is, a single-stream Transformer structure is used to tokenize the input of multimodal video information, and the Transformer's self-attention mechanism is used to align and fuse multimodal features.

[0155] For the text part, BERT is used to extract the embedding representation of the video description information, and then the video embedding is added with the position encoding and type encoding to obtain the text features.

[0156] For the video frame part, a visual feature encoder (such as a CNN model or a ViT model) is used to extract video features;

[0157] Similarly, audio features are extracted through the Vggish model. In this way, video features, audio features, and text features can be input into the Transformer in the form of tokens for training, and the fusion features can be learned for prediction. After the training is completed, a multi-label classification model can be obtained.

[0158] In the above embodiment, the identified video tags are applied to model training, thereby eliminating the need for any manual operation and greatly improving the training efficiency of the model.

[0159] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0160] Based on the same inventive concept, the embodiment of the present application also provides a video tag recognition device for implementing the above-mentioned video tag recognition method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the embodiment of the device for recognizing one or more video tags provided below can refer to the limitations of the video tag recognition method above, and will not be repeated here.

[0161] In one embodiment, as shown in FIG. 8 , a video tag recognition device is provided, including: a first determination module 802 , a second determination module 804 , an aggregation module 806 , a fusion module 808 and a recognition module 810 , wherein:

[0162] A first determination module 802, configured to determine global information of a video frame based on the video frame and the candidate tags in the video;

[0163] A second determination module 804 is used to determine local information of the video frame according to the image blocks and candidate tags in the video frame;

[0164] Aggregation module 806, for performing spatiotemporal aggregation on the global information and the local information of each video frame in the video, to obtain spatiotemporal global information and spatiotemporal local information;

[0165] A fusion module 808 is used to fuse the spatiotemporal global information and the spatiotemporal local information to obtain fused information;

[0166] The identification module 810 is used to identify the tag to which the video belongs from the candidate tags based on the fusion information.

[0167] In one embodiment, as shown in FIG9 , the device further includes:

[0168] The processing module 812 is used to perform time-series smoothing processing on the global information of each video frame to obtain smoothed global information; and perform time-series smoothing processing on the local information of each video frame to obtain smoothed local information;

[0169] The aggregation module 806 is further used to perform spatiotemporal aggregation on the smoothed global information of each video frame to obtain spatiotemporal global information; and perform spatiotemporal aggregation on the smoothed local information of each video frame to obtain spatiotemporal local information.

[0170] In one embodiment, the global information includes global similarity information. As shown in FIG9 , the apparatus further includes:

[0171] An encoding module 814 is used to encode the video frame in the video and the first description information in the text sequence respectively to obtain a first video feature of the video frame and a description feature of the first description information; the first description information is video description information formed based on the candidate tags in the tag set;

[0172] The first determination module 802 is further configured to determine global similarity information of the video frame based on the first video feature and the description feature; the global similarity information includes a similarity value between the video frame and the candidate tag.

[0173] In one embodiment, the local information includes local similarity information. As shown in FIG9 , the apparatus further includes:

[0174] An encoding module 814, configured to encode an image block of a video frame to obtain a second video feature of the image block;

[0175] The second determination module 804 is further configured to determine local similarity information of the video frame based on the second video feature and the description feature; the local similarity information includes a similarity value between an image block of the video frame and a candidate tag.

[0176] In one embodiment, the video frame includes a sampled video frame; as shown in FIG9 , the apparatus further includes:

[0177] A decoding module 816, configured to decode the video to obtain a video frame sequence;

[0178] The sampling module 818 is used to determine a sampling interval, and sample video frames in the video frame sequence according to the sampling interval to obtain sampled video frames of the video.

[0179] In one embodiment, the fusion module 808 is further used to determine the mean between the elements in the spatiotemporal global information and the elements in the spatiotemporal local information to obtain fused information; or, to perform weighted summation of the elements in the spatiotemporal global information and the elements in the spatiotemporal local information to obtain fused information.

[0180] In one embodiment, as shown in FIG9 , the device further includes:

[0181] The segmentation module 820 is used to segment the video frames in the video into the same number of image blocks; or to segment the video frames in the video according to a pyramid segmentation method to obtain corresponding image blocks; wherein the number of image blocks in different video frames is different.

[0182] In one embodiment, the identification module 810 is further used to determine the multi-label distribution score of the video based on the fusion information; or, to use the fusion information as the multi-label distribution score of the video; and to determine the label to which the video belongs among the candidate labels according to the multi-label distribution score.

[0183] In the above embodiment, the global information of the video frame is determined based on the video frame and the candidate tags in the video, and the local information of the video frame is determined according to the image blocks and the candidate tags in the video frame; for each video frame in the video, the global information of each video frame and the local information of each video frame are respectively spatiotemporally aggregated, so that the global information of different video frames are aggregated together, and the local information of different video frames are aggregated together, and the global information and local information of the video space and time are fully considered to obtain the spatiotemporal global information and the spatiotemporal local information; in addition, the spatiotemporal global information and the spatiotemporal local information are fused, so as to fuse the global information and local information of the spatiotemporal dimension of the video. Therefore, even in the face of multi-label recognition of the video, the label to which the video belongs can be accurately identified based on the fused information, thereby effectively improving the recognition accuracy of the video label; in addition, the influence of human factors can be effectively avoided, and the recognition efficiency of the video label can be effectively improved.

[0184] In one embodiment, as shown in FIG9 , the device further includes:

[0185] The training module 822 is used to use the labels whose multi-label distribution scores meet the scoring conditions as positive sample labels; use the labels whose multi-label distribution scores do not meet the scoring conditions as negative sample labels; use multiple videos as training samples, and train the initial classification model based on the training samples, positive sample labels and negative sample labels to obtain a classification model.

[0186] In one embodiment, as shown in FIG9 , the device further includes:

[0187] The acquisition module 824 is used to acquire the audio of the training sample and extract the audio features from the audio; acquire the second description information of the training sample and extract the description features from the second description information; the second description information is the video description information of the training sample;

[0188] The training module 822 is also used to encode the video frame samples in the training samples and the image block samples in the video frame samples respectively to obtain the third video features of the video frame samples and the fourth video features of the image block samples; based on the third video features, the fourth video features, the description features of the second description information and the audio features, the initial classification model is trained so that the similarity values ​​between the predicted labels of the training samples and the positive sample labels and the negative sample labels reach the preset similarity conditions to obtain the classification model.

[0189] In one embodiment, the acquisition module 824 is used to extract audio features from the audio through an audio feature extractor; encode the second description information through a target bidirectional encoder to obtain description features of the second description information;

[0190] The training module 822 is further configured to encode the video frame samples and the image block samples in the training samples through a visual feature encoder to obtain the third video features of the video frame samples and the fourth video features of the image block samples.

[0191] In the above embodiment, the identified video tags are applied to model training, thereby eliminating the need for any manual operation and greatly improving the training efficiency of the model.

[0192] Each module in the above-mentioned video tag recognition device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0193] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG10. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store video data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for identifying a video tag is implemented.

[0194] In one embodiment, a computer device is provided, which may be a terminal, and includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for identifying a video tag is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0195] Those skilled in the art will understand that the structure shown in FIG. 10 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0196] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned video tag recognition method when executing the computer program.

[0197] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned video tag recognition method are implemented.

[0198] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the above-mentioned video tag recognition method when executed by a processor.

[0199] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0200] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0201] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0202] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for identifying video tags, characterized in that: The method comprises: Calculating similarity between video frames and candidate tags in a video to obtain global information of the video frame; the video includes live video and non-live video, the candidate tags include tags of people, animals and objects, which are used to indicate the category to which the object in the video frame belongs, and the global information is the similarity value between the object in the entire video frame and the candidate tag; Performing similarity calculation on the image block in the video frame and the candidate tag to obtain local information of the video frame; the local information is a similarity value between the object in the image block and the candidate tag; Performing temporal smoothing processing on the global information of each of the video frames to obtain smoothed global information; performing temporal smoothing processing on the local information of each of the video frames to obtain smoothed local information; For each video frame in the video, the smoothed global information of each video frame is subjected to spatiotemporal aggregation to obtain spatiotemporal global information; the smoothed local information of each video frame is subjected to spatiotemporal aggregation to obtain spatiotemporal local information; wherein the spatiotemporal aggregation is achieved by performing maximum pooling processing on the smoothed global information of each video frame and the smoothed local information of each video frame in the spatiotemporal dimension, wherein the maximum pooling processing includes selecting the maximum similarity value from all the smoothed global information and the smoothed local information, and each video corresponds to one spatiotemporal global information and one spatiotemporal local information respectively; The spatiotemporal global information and the spatiotemporal local information are fused to obtain fused information; the fused information includes fused similarity information; Determine the multi-label distribution score of the video based on the fusion information; or use the fusion information as the multi-label distribution score of the video; and determine the label to which the video belongs among the candidate labels according to the multi-label distribution score; The labels whose multi-label distribution scores meet the scoring conditions are used as positive sample labels; the labels whose multi-label distribution scores do not meet the scoring conditions are used as negative sample labels; the multiple videos are used as training samples, and an initial classification model is trained based on the training samples, the positive sample labels and the negative sample labels to obtain a classification model.

2. The method according to claim 1, characterized in that The global information includes global similarity information, and the method further includes: Encoding the video frame in the video and the first description information in the text sequence respectively to obtain the first video feature of the video frame and the description feature of the first description information; the first description information is video description information formed based on the candidate tags in the tag set; The similarity calculation of the video frame and the candidate tag in the video to obtain the global information of the video frame includes: The global similarity information of the video frame is determined based on the first video feature and the description feature; the global similarity information includes a similarity value between the video frame and the candidate tag.

3. The method according to claim 2, characterized in that The local information includes local similarity information, and the method further includes: Encoding the image block of the video frame to obtain a second video feature of the image block; The performing similarity calculation on the image block in the video frame and the candidate label to obtain the local information of the video frame includes: The local similarity information of the video frame is determined based on the second video feature and the description feature; the local similarity information includes a similarity value between an image block of the video frame and the candidate tag.

4. The method according to claim 1, characterized in that The video frames include sampled video frames; Before determining the global information of the video frame based on the video frame and the candidate tags in the video, the method further includes: Performing video decoding on the video to obtain a video frame sequence; A sampling interval is determined, and video frames are sampled in the video frame sequence according to the sampling interval to obtain sampled video frames of the video.

5. The method according to claim 1, characterized in that The fusing the spatiotemporal global information and the spatiotemporal local information to obtain the fused information includes: Determine the mean between the elements in the spatiotemporal global information and the elements in the spatiotemporal local information to obtain fusion information; or, The elements in the spatiotemporal global information and the elements in the spatiotemporal local information are weightedly summed to obtain fused information.

6. The method according to claim 1, characterized in that Before determining the local information of the video frame according to the image block in the video frame and the candidate tag, the method further includes: Divide the video frames in the video into the same number of image blocks; or, The video frames in the video are segmented in a pyramidal segmentation manner to obtain corresponding image blocks; wherein the number of image blocks in different video frames is different.

7. The method according to claim 1, characterized in that The method further comprises: Acquire the audio of the training sample and extract audio features from the audio; Acquire second description information of the training sample, and extract description features from the second description information; the second description information is video description information of the training sample; The training of the initial classification model based on the training samples, the positive sample labels and the negative sample labels to obtain the classification model comprises: Encoding the video frame samples in the training samples and the image block samples in the video frame samples respectively to obtain the third video features of the video frame samples and the fourth video features of the image block samples; The initial classification model is trained based on the third video feature, the fourth video feature, the description feature of the second description information and the audio feature so that the similarity value between the predicted label of the training sample and the positive sample label and the negative sample label reaches a preset similarity condition, thereby obtaining a classification model.

8. The method according to claim 7, characterized in that The extracting audio features from the audio comprises: Extracting audio features from the audio by an audio feature extractor; The extracting description features from the second description information comprises: encoding the second description information by a target bidirectional encoder to obtain description features of the second description information; The step of respectively encoding the video frame samples in the training samples and the image block samples in the video frame samples to obtain the third video features of the video frame samples and the fourth video features of the image block samples comprises: The video frame samples in the training samples and the image block samples in the video frame samples are encoded by a visual feature encoder to obtain a third video feature of the video frame samples and a fourth video feature of the image block samples.

9. A video tag recognition device, characterized in that: The device comprises: A first determination module is used to calculate the similarity between a video frame and a candidate tag in a video to obtain global information of the video frame; the video includes a live video and a non-live video, the candidate tags include tags of people, animals and objects, which are used to indicate the category to which the object in the video frame belongs, and the global information is a similarity value between the object in the entire video frame and the candidate tag; A second determination module is used to perform similarity calculation on the image block in the video frame and the candidate tag to obtain local information of the video frame; the local information is a similarity value between the object in the image block and the candidate tag; A processing module, used for performing temporal smoothing processing on the global information of each of the video frames to obtain smoothed global information; performing temporal smoothing processing on the local information of each of the video frames to obtain smoothed local information; An aggregation module is used for performing spatiotemporal aggregation on the smoothed global information of each video frame in the video to obtain spatiotemporal global information; performing spatiotemporal aggregation on the smoothed local information of each video frame to obtain spatiotemporal local information; wherein the spatiotemporal aggregation is achieved by performing maximum pooling processing on the smoothed global information of each video frame and the smoothed local information of each video frame in the spatiotemporal dimension, wherein the maximum pooling processing includes selecting the maximum similarity value from all the global information and the local information, respectively, and each video corresponds to one spatiotemporal global information and one spatiotemporal local information; A fusion module, used for fusing the spatiotemporal global information and the spatiotemporal local information to obtain fused information; the fused information includes fused similar information; an identification module, configured to determine a multi-label distribution score of the video based on the fusion information; or, to use the fusion information as the multi-label distribution score of the video; and to determine, among the candidate labels, the label to which the video belongs according to the multi-label distribution score; A training module is used to use the labels whose multi-label distribution scores meet the scoring conditions as positive sample labels; use the labels whose multi-label distribution scores do not meet the scoring conditions as negative sample labels; use the multiple videos as training samples, and train the initial classification model based on the training samples, the positive sample labels and the negative sample labels to obtain a classification model.

10. The device according to claim 9, characterized in that The global information includes global similarity information, and the device further includes: An encoding module, used for encoding the video frame in the video and the first description information in the text sequence respectively, to obtain the first video feature of the video frame and the description feature of the first description information; the first description information is video description information formed based on the candidate tags in the tag set; The first determination module is further used to determine the global similarity information of the video frame based on the first video feature and the description feature; the global similarity information includes a similarity value between the video frame and the candidate tag.

11. The device according to claim 10, characterized in that The local information includes local similarity information, and the device further includes: An encoding module, used for encoding the image block of the video frame to obtain a second video feature of the image block; The second determination module is further used to determine local similarity information of the video frame based on the second video feature and the description feature; the local similarity information includes a similarity value between an image block of the video frame and the candidate tag.

12. The device according to claim 9, characterized in that The video frame includes a sampled video frame; the device also includes: A decoding module, used for decoding the video to obtain a video frame sequence; The sampling module is used to determine a sampling interval, sample video frames in the video frame sequence according to the sampling interval, and obtain sampled video frames of the video.

13. The device according to claim 9, characterized in that The fusion module is also used to determine the mean between the elements in the spatiotemporal global information and the elements in the spatiotemporal local information to obtain fused information; or to perform weighted summation on the elements in the spatiotemporal global information and the elements in the spatiotemporal local information to obtain fused information.

14. The device according to claim 9, characterized in that Before determining the local information of the video frame according to the image block in the video frame and the candidate tag, the device further includes: A segmentation module is used to segment the video frames in the video into the same number of image blocks; or to segment the video frames in the video in a pyramidal segmentation manner to obtain corresponding image blocks; wherein the number of image blocks of different video frames is different.

15. The device according to claim 9, characterized in that The device also includes: An acquisition module, used to acquire the audio of the training sample and extract audio features from the audio; acquire second description information of the training sample and extract description features from the second description information; the second description information is video description information of the training sample; The training module is also used to encode the video frame samples in the training samples and the image block samples in the video frame samples respectively to obtain the third video features of the video frame samples and the fourth video features of the image block samples; based on the third video features, the fourth video features, the description features of the second description information and the audio features, the initial classification model is trained so that the similarity values ​​between the predicted labels of the training samples and the positive sample labels and the negative sample labels reach preset similarity conditions to obtain the classification model.

16. The device according to claim 15, characterized in that The acquisition module is used to extract the audio features in the audio through an audio feature extractor; encode the second description information through a target bidirectional encoder to obtain the description features of the second description information; The training module is further used to encode the video frame samples in the training samples and the image block samples in the video frame samples through a visual feature encoder to obtain the third video features of the video frame samples and the fourth video features of the image block samples.

17. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images