Video recognition method, device and storage medium

Abstract

The present application provides a method, device and storage medium for video recognition, which relates to the fields of artificial intelligence and computer technology, and is used to improve the accuracy of identifying video tags. The video to be processed is input into a tag classification model to obtain the matching probability value corresponding to each preset tag; the video to be processed is input into a tag retrieval model to obtain each candidate annotation tag that successfully matches the video to be processed and the corresponding target similarity. The target similarity corresponding to each candidate annotation tag is determined based on the video data matching similarity between each associated historical video and the video to be processed; the preset tags whose matching probability value reaches the probability threshold are screened out to form a first tag set, and the candidate annotation tags whose target similarity reaches the similarity threshold are screened out to form a second tag set; the first tag set and the second tag set are fused to determine the target tag corresponding to the video to be processed; the tag recall and precision are improved, and the recognition accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device and storage medium for video recognition. Background Art

[0002] Video platforms generally require labeling of videos. In addition to users adding corresponding labels when uploading videos to the video platform, video platform reviewers can also add corresponding labels to each video during the review process. Machine learning methods can also be used to identify the labels corresponding to the videos.

[0003] When using machine learning methods to identify labels corresponding to videos, it is mainly based on the text features of the video title. The text information of the video title and the labels in the platform's massive label pool are modeled separately and their correlation is calculated. Then, based on the correlation, the label corresponding to the video is obtained from the platform's massive label pool, resulting in low accuracy of the identified labels corresponding to the video. Summary of the Invention

[0004] The present application provides a video recognition method, device, and storage medium to improve the accuracy of video tag recognition.

[0005] In a first aspect, an embodiment of the present application provides a method for video recognition, the method comprising:

[0006] Input the video to be processed into the trained label classification model, match the video to be processed with each preset label through the label classification model, and obtain the matching probability value corresponding to each preset label;

[0007] Input the video to be processed into the trained label retrieval model. The label retrieval model is used to match the video to be processed with each historical video to obtain each candidate label that successfully matches the video to be processed and the corresponding target similarity. The target similarity corresponding to each candidate label is determined based on the video data matching similarity between each historical video associated with the candidate label and the video to be processed.

[0008] Screening out preset tags whose matching probability values ​​reach a probability threshold to form a first tag set, and screening out candidate annotation tags whose target similarity reaches a similarity threshold to form a second tag set;

[0009] The first label set and the second label set are fused to determine the target label corresponding to the video to be processed.

[0010] In a second aspect, an embodiment of the present application provides a video recognition device, the device comprising:

[0011] The first obtaining unit is configured to input the video to be processed into a trained label classification model, match the video to be processed with each preset label through the label classification model, and obtain a matching probability value corresponding to each preset label;

[0012] The second obtaining unit is configured to input the to-be-processed video into the trained label retrieval model, match the to-be-processed video with each historical video through the label retrieval model, and obtain each candidate annotation label successfully matched with the to-be-processed video and the corresponding target similarity, wherein the target similarity corresponding to each candidate annotation label is determined based on the video data matching similarity between each historical video associated with the candidate annotation label and the to-be-processed video;

[0013] a screening unit, configured to screen out preset tags whose matching probability values ​​reach a probability threshold to form a first tag set, and to screen out candidate annotation tags whose target similarity reaches a similarity threshold to form a second tag set;

[0014] The determination unit is configured to fuse the first tag set and the second tag set to determine a target tag corresponding to the video to be processed.

[0015] In a possible implementation, the trained label classification model in the first obtaining unit is obtained by training in the following manner:

[0016] According to the training samples in the training sample data set, the label classification model to be trained is subjected to cyclic iterative training, and when the preset convergence conditions are met, the trained label classification model is output. The following operations are performed during one cyclic iterative training process:

[0017] Selecting training samples from a training sample dataset, wherein the training samples are historical videos with annotated labels;

[0018] Input the training samples into the label classification model;

[0019] Based on the fully connected layer in the label classification model, the training samples are mapped to various preset labels, and the matching probability value of each preset label is predicted;

[0020] Based on the matching probability values ​​of the labeled labels corresponding to the training samples and the predicted preset labels, the binary cross entropy corresponding to each preset label is determined;

[0021] A loss function is constructed based on the binary cross entropy of each preset label, and the parameters of the label classification model are adjusted based on the loss function.

[0022] In one possible implementation, the video to be processed is input into a label classification model, and the label identification model is used to match the video to be processed with each preset label to obtain a matching probability value corresponding to each preset label, including:

[0023] Input the video to be processed into the trained label classification model;

[0024] Based on the fully connected layer in the label classification model, the video to be processed is mapped to each preset label, and the label matching similarity between the video to be processed and each preset label is determined respectively;

[0025] Based on the Sigmoid function in the label classification model, the obtained label matching similarities are converted into matching probability values ​​of the corresponding preset labels.

[0026] In a possible implementation, the tag retrieval model in the second acquisition unit includes at least one of a text retrieval sub-model and a knowledge mining sub-model;

[0027] If the tag retrieval model includes a text retrieval sub-model, each candidate tag successfully matched by the to-be-processed video and the corresponding target similarity are obtained based on the first candidate tag successfully matched by the to-be-processed video and the corresponding first target similarity;

[0028] If the label retrieval model includes a knowledge mining sub-model, each candidate labeling label successfully matched with the to-be-processed video and the corresponding target similarity are obtained based on each second candidate labeling label successfully matched with the to-be-processed video and the corresponding second target similarity obtained with the knowledge mining sub-model.

[0029] In a possible implementation, if the label retrieval model in the second acquisition unit includes a text retrieval sub-model;

[0030] The second acquisition unit obtains each first candidate annotation label and the corresponding first target similarity that are successfully matched to the video to be processed by the following method:

[0031] Input the video to be processed into the text retrieval sub-model, match the video to be processed with each historical video through the text retrieval sub-model, and obtain the video data matching similarity between each historical video and the video to be processed;

[0032] Based on the obtained video data matching similarity associated with each historical video, historical videos whose video data matching similarity reaches the video data matching similarity threshold and the associated annotation tags are screened out;

[0033] Based on the various annotation tags associated with all the screened historical videos, the first candidate annotation tags that successfully match the video to be processed are obtained, and based on the video data matching similarity corresponding to the various historical videos associated with the various first candidate annotation tags, the first target similarity of each first candidate annotation tag is determined.

[0034] In a possible implementation, the second acquiring unit is specifically configured to:

[0035] The video data matching similarities corresponding to the historical videos associated with the first candidate label tags are respectively used as candidate similarities associated with the corresponding first candidate label tags, and the maximum value among the candidate similarities associated with the first candidate label tags is respectively used as the first target similarity of the corresponding first candidate label tags; or

[0036] The video data matching similarities corresponding to each historical video associated with each first candidate annotation tag are respectively used as the candidate similarities associated with the corresponding first candidate annotation tag, and the candidate similarities associated with each first candidate annotation tag are weighted averaged to obtain the corresponding weighted average processing results, and the first target similarities of each first candidate annotation tag are respectively determined based on the weighted average processing results.

[0037] In a possible implementation, if the label retrieval model in the second acquisition unit includes a knowledge mining sub-model;

[0038] The second acquisition unit obtains each second candidate annotation label and the corresponding second target similarity that are successfully matched with the video to be processed in the following manner:

[0039] The video to be processed is input into the knowledge mining sub-model. Through the knowledge mining sub-model, the to-be-processed segmentation associated with the video to be processed is matched with each historical segmentation associated with each historical video, and the video data matching similarity between each historical segmentation and the to-be-processed segmentation is obtained respectively; wherein each historical segmentation is a segmentation that coexists with at least one annotated label associated with the corresponding historical video, and each historical segmentation corresponds to at least one annotated label, and each annotated label is associated with a coexistence probability of the corresponding historical segmentation;

[0040] Based on the obtained video data matching similarity associated with each historical segmentation, the historical segmentations and associated annotation tags whose video data matching similarity reaches the video data matching similarity threshold are screened out;

[0041] Based on the various annotation tags associated with all the filtered historical word segmentations, the second candidate annotation tags that successfully match the video to be processed are obtained, and based on the coexistence probability associated with each second candidate annotation tag and the video data matching similarity between the corresponding historical word segmentation and the video to be processed, the second target similarity corresponding to each second candidate annotation tag is determined.

[0042] In a possible implementation, the second acquiring unit is specifically configured to:

[0043] The video data matching similarity between each historical segmentation and the segmentation to be processed is used as the candidate similarity associated with each second candidate annotation label corresponding to each historical segmentation;

[0044] Based on the coexistence probability associated with each second candidate annotation tag, each corresponding candidate similarity is weighted to obtain a corresponding processing result, and the second target similarity corresponding to each second candidate annotation tag is determined based on the processing result, wherein the weight coefficient of each weighted processing is the coexistence probability associated with the corresponding second candidate annotation tag.

[0045] In a possible implementation, if the label retrieval model in the second acquisition unit includes a text retrieval sub-model and a knowledge mining sub-model;

[0046] The screening unit is specifically used for:

[0047] Based on the first target similarity, selecting a first candidate annotation tag whose first target similarity reaches a first similarity threshold from each first candidate annotation tag; and

[0048] Based on the second target similarity, selecting second candidate annotation tags whose second target similarity reaches a second similarity threshold from each second candidate annotation tag;

[0049] Based on the same candidate labeling tags among the screened first candidate labeling tags and the screened second candidate labeling tags, a second labeling set is formed;

[0050] The determination unit is specifically used for:

[0051] All tags in the first tag set and the second tag set are used as target tags corresponding to the video to be processed.

[0052] In a possible implementation, the screening unit is specifically configured to:

[0053] If it is determined based on each matching probability value that the target label exists in the video to be processed, and each matching probability value is less than the probability threshold, a first number of candidate annotation labels whose target similarity reaches the similarity threshold are screened out to form a second label set;

[0054] If it is determined based on the matching probability value that the target tag exists in the video to be processed, and at least one matching probability value reaches the probability threshold, then the preset tags whose matching probability values ​​reach the probability threshold are screened out to form a first tag set, and a second number of candidate annotation tags whose target similarities reach the similarity threshold are screened out to form a second tag set, wherein the first number is greater than the second number;

[0055] If it is determined based on each matching probability value that the target tag does not exist in the video to be processed, then both the first tag set and the second tag set are empty sets.

[0056] In a possible implementation, each preset label included in the label classification model in the first acquisition unit is determined in the following manner:

[0057] Based on the acquired historical videos, a prediction sample set is constructed;

[0058] Performing encoding processing on each prediction sample in the prediction sample set, determining a sample feature vector of each prediction sample, and determining a feature matching similarity between each sample feature vector and a first feature vector associated with each preset category;

[0059] Based on the feature matching similarity corresponding to each preset category, at least one target category is selected from each preset category, and the prediction samples are divided into each target category;

[0060] Based on the prediction samples contained in each target category, the prediction label corresponding to the target category is determined.

[0061] In a possible implementation, the prediction sample set is a prediction sample directed graph structure, wherein each node in the prediction sample directed graph structure represents a sample consisting of a historical video;

[0062] Encoding is performed on each sample in the prediction sample set, and the sample feature vector of each sample is determined respectively, including:

[0063] Determine at least one neighbor node connected to each node and the in-degree of at least one neighbor node based on the predicted sample directed graph structure, and determine at least one target neighbor node based on the in-degree of each neighbor node;

[0064] Perform weighted sum processing on the second eigenvector of each node and the third eigenvector of each corresponding target neighbor node to obtain the sample eigenvector of each sample;

[0065] The weighted coefficients in the weighted summation process are obtained by performing dot multiplication of the second eigenvector of each node and the third eigenvector of each corresponding target neighbor node with the second eigenvector of the corresponding node and then normalizing the result.

[0066] In a third aspect, an embodiment of the present application provides a video recognition device, comprising: a memory and a processor, wherein the memory is used to store computer instructions; and the processor is used to execute computer instructions to implement the video recognition method provided by the embodiment of the present application.

[0067] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a processor, the video recognition method provided by the embodiment of the present application is implemented.

[0068] The beneficial effects of this application are as follows:

[0069] The embodiment of the present application provides a method, device and storage medium for video recognition, which are used to improve the accuracy of video tag recognition. In the embodiment of the present application, the video to be processed is matched with each preset tag through a tag classification model, and the matching probability value corresponding to each preset tag is obtained respectively, and the preset tags whose matching probability value reaches the probability threshold are screened out from each preset tag to form a first tag set; the video to be processed contains audio text information, picture text information, and title text information, and the first tag corresponding to the text to be processed is determined based on the text information in the video to be processed and the preset tags. When forming the first tag set, the reference information is rich, and the tag of the video to be processed is no longer identified based solely on the title text information, so that the video tag predicted by the tag classification model is more accurate; and the video to be processed is matched with each historical video through a tag retrieval model to obtain each candidate annotation tag that successfully matches the video to be processed. and the corresponding target similarity. The target similarity corresponding to each candidate annotation tag is determined based on the video data matching similarity between each historical video associated with the candidate annotation tag and the video to be processed, and based on the target similarity, the candidate annotation tags whose target similarity reaches the similarity threshold are screened out from the candidate annotation tags to form a second tag set. In the process of determining the tag for the video to be processed based on the historical videos with tags, based on the means of finding similar videos, the historical videos with high similarity to the video to be processed and with tags are more accurately determined in the historical videos. By applying the tags with higher accuracy of the determined historical videos, the accuracy of the tags determined for the video to be processed is significantly improved. The first tag set and the second tag set are fused to determine the target tag corresponding to the video to be processed. In the process of identifying the target tag corresponding to the video to be processed, the tag classification model and the tag retrieval model are combined to improve the recall and precision of the target tag corresponding to the video to be processed, so as to improve the accuracy of identifying the target tag corresponding to the video to be processed.

[0070] Other features and advantages of the present application will be described in the following description, and will become apparent from the description or understood by practicing the present application. The objectives and other advantages of the present application can be achieved and obtained through the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0072] Figure 1 is a schematic diagram of a directed graph;

[0073] Figure 2 A diagram showing a model structure for identifying target tags corresponding to a video provided in an embodiment of the present application;

[0074] Figure 3 Another model structure diagram for identifying target tags corresponding to videos provided in an embodiment of the present application;

[0075] Figure 4 This is a schematic diagram of the application scenario of video recognition;

[0076] Figure 5 A flow chart of a video recognition method provided in an embodiment of the present application;

[0077] Figure 6 A flowchart of a method for training a label classification model provided in an embodiment of the present application;

[0078] Figure 7 A flowchart of a method for using a label classification model provided in an embodiment of the present application;

[0079] Figure 8 A flow chart of a method for determining each preset tag provided in an embodiment of the present application;

[0080] Figure 9 A flowchart of a method for using a text retrieval sub-model provided in an embodiment of the present application;

[0081] Figure 10 A schematic diagram of screening a first candidate annotation label provided in an embodiment of the present application;

[0082] Figure 11 A flow chart of a method for using a knowledge mining sub-model provided in an embodiment of the present application;

[0083] Figure 12 A schematic diagram of screening a second candidate annotation label provided in an embodiment of the present application;

[0084] Figure 13 A schematic diagram of determining a target label for a video to be processed provided in an embodiment of the present application;

[0085] Figure 14 A flowchart of an overall method for video recognition provided in an embodiment of the present application;

[0086] Figure 15 A structural diagram of a video recognition device provided in an embodiment of the present application;

[0087] Figure 16 A structural diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0088] In order to make the purpose, technical solutions and beneficial effects of this application more clear and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0089] The terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, such that the embodiments of the application described herein can be practiced in orders other than those illustrated or described herein.

[0090] The following explains some of the terms used in the embodiments of the present application to facilitate understanding by those skilled in the art.

[0091] 1. Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0092] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0093] 2. Key technologies in speech technology include automatic speech recognition (ASR), text-to-speech (TTS), and voiceprint recognition. Enabling computers to hear, see, speak, and feel is the future direction of human-computer interaction, with speech becoming one of the most promising methods of human-computer interaction.

[0094] The goal of automatic speech recognition (ASR) technology is to convert the vocabulary content in human speech into text or computer-readable input, such as keystrokes, binary codes, or character sequences.

[0095] 3. Optical Character Recognition (ORC) is a technology that converts text in images into text. It primarily involves image preprocessing, text feature extraction, comparative recognition, word post-processing, and text output. In this embodiment, ORC primarily converts the text contained in each frame of the video being processed into corresponding text information.

[0096] 4. Machine Learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0097] 5. A directed graph is a graph composed entirely of directed edges, which refers to an ordered triple (V(D), A(D), ψ(D)), where ψ(D) is the association function that makes each element in A(D) (called a directed edge or arc) correspond to an ordered element in V(D) (called a vertex or point).

[0098] A directed graph contains out-degree and in-degree. The out-degree is the number of edges starting from a certain vertex, and the in-degree is the number of edges ending at a certain vertex. The out-degree + in-degree is called the degree of a vertex. Figure 1 An example of a directed graph is provided. Taking point A as an example, among all the edges associated with point A, the number of edges starting from point A is called out-degree. The in-degree is just the opposite, and the number of edges ending at point A is called in-degree. Among them, in-degree + out-degree is called the degree of point A. If point A has a self-loop, with both the starting point and the end point being itself, then the out-degree is counted as one degree, and the in-degree is also counted as one degree. Therefore, Figure 1 The out-degree of midpoint A is 3, the in-degree of point A is also 2, and the degree of point A is 5.

[0099] 6. K-Means clustering is an unsupervised clustering algorithm that is relatively simple to implement and has good clustering effects. The idea of ​​K-Means clustering is very simple. For a given sample set, the sample set is divided into K clusters according to the distance between the samples. The points within the cluster are connected as closely as possible, and the distance between clusters is as large as possible. In the embodiment of the present application, K clusters are pre-set, and historical videos are given. Each historical video is divided into at least one cluster, and each cluster corresponds to a preset label. Then, based on the common characteristics of the historical videos contained in each cluster, the preset labels corresponding to each cluster are determined.

[0100] 7. Annotation tags are concise summaries of certain aspects of historical videos, and can be used to label the types of historical videos. For example, the annotation tags can be entertainment, life, etc. Under entertainment, the annotation tags can be further subdivided into games, TV series clips, movie clips, celebrity profiles, etc. Under life, the annotation tags can be further subdivided into life skills, cooking, shopping, etc. Different historical videos have different corresponding annotation tags, so there are a large number of annotation tags. The above are just examples.

[0101] Therefore, the target tag is a concise summary of a certain aspect of the video to be processed. The target tag is mainly used to recommend other videos that the user is interested in based on the video to be processed, or the video platform can quickly make recommendations by associating the target tag with other video resumes.

[0102] The following is a brief introduction to the design concept of the embodiments of this application.

[0103] This application is for video recognition, mainly identifying the tags corresponding to the video. In the related technologies of video recognition, the common way to identify the tags corresponding to the video is to use traditional machine learning methods such as Naive Bayes, Support Vector Machine (SVM) or deep learning, and to implement it based on the text features of the video title. By modeling the text information of the video title and the tags in the platform's massive tag pool respectively and calculating their correlation, the tag corresponding to the video is obtained from the platform's massive tag pool based on the correlation. At this time, the accuracy of identifying the tag corresponding to the video is low.

[0104] Furthermore, the tags in the platform's massive tag pool in related technologies are primarily determined using clustering, topic discovery technology, and manual construction. There are two main clustering methods: one that uses similarity between samples, and the other that uses graph structure clustering between samples. This results in a large and complex tag pool with excessively fine semantic granularity and a mix of coarse and fine details. Ultimately, identifying the tags corresponding to a video is time-consuming and difficult.

[0105] In view of this, the embodiments of the present application adopt artificial intelligence and propose a video recognition method, device, and storage medium. In the embodiments of the present application, a tag classification model and a tag retrieval model are combined to fully utilize the text features of historical videos, where the text features include the annotated tags of historical videos, the title information of historical videos, the OCR text information of historical videos, and the ASR text information of historical videos to construct a video recognition model for identifying the tags corresponding to the videos. Figure 2 A schematic diagram of a video recognition model provided in an embodiment of the present application is provided as an example. Figure 2 It can be seen that the video recognition model includes a label classification model and a label retrieval model.

[0106] In the embodiment of this application, combined with Figure 2 Perform the following operations to identify the video and determine the label corresponding to the video:

[0107] Input the video to be processed into the trained label classification model, match the video to be processed with each preset label through the label classification model, and obtain the matching probability value corresponding to each preset label;

[0108] Input the video to be processed into the trained label retrieval model. The label retrieval model is used to match the video to be processed with each historical video to obtain each candidate label that successfully matches the video to be processed and the corresponding target similarity. The target similarity corresponding to each candidate label is determined based on the video data matching similarity between each historical video associated with the candidate label and the video to be processed.

[0109] Screening out preset tags whose matching probability values ​​reach a probability threshold to form a first tag set, and screening out candidate annotation tags whose target similarity reaches a similarity threshold to form a second tag set;

[0110] The first tag set and the second tag set are fused to determine the target tag corresponding to the video to be processed.

[0111] Figure 3 A schematic diagram of another video recognition model in the embodiment of the present application is provided as an example. Figure 3 As can be seen in the figure, the video recognition model includes a label classification model and a label retrieval model, wherein the label retrieval model includes a text retrieval sub-model and a knowledge mining sub-model. It should be noted that the label classification model, the text retrieval sub-model, and the knowledge mining sub-model can be three parallel models. In this case, the video to be processed is input into the label classification model, the text retrieval sub-model, and the knowledge mining sub-model respectively for video recognition to determine the label corresponding to the video;

[0112] In the embodiment of this application, Figure 3 Perform the following operations to identify the video and determine the label corresponding to the video:

[0113] Input the video to be processed into the trained label classification model, match the video to be processed with each preset label through the label classification model, and obtain the matching probability value corresponding to each preset label;

[0114] Input the video to be processed into the trained text retrieval sub-model, and match the video to be processed with each historical video through the text retrieval sub-model to obtain each first candidate annotation label successfully matched with the video to be processed and the corresponding first target similarity, wherein the first target similarity corresponding to each first candidate annotation label is determined based on the video data matching similarity between each historical video associated with the first candidate annotation label and the video to be processed;

[0115] Input the video to be processed into the trained text retrieval sub-model, and match the to-be-processed segmentation of the video to be processed with the historical segmentation of each historical video through the text retrieval sub-model to obtain each second candidate annotation label successfully matched with the video to be processed and the corresponding second target similarity, wherein the second target similarity corresponding to each second candidate annotation label is determined based on the video data matching similarity between each historical segmentation associated with the second candidate annotation label and the to-be-processed segmentation of the video to be processed;

[0116] Filtering out preset tags whose matching probability values ​​reach a probability threshold to form a first tag set; and

[0117] Screening out first candidate annotation tags whose first target similarity reaches a first similarity threshold, and screening out second candidate annotation tags whose second target similarity reaches a second similarity threshold, and forming a second tag set based on the screened out first candidate annotation tags and the screened out second candidate annotation tags;

[0118] The first tag set and the second tag set are fused to determine the target tag corresponding to the video to be processed.

[0119] In order to streamline the labels in the label pool, change the semantic granularity of the labels, and save the time of identifying the labels corresponding to the video, an improved k-means clustering method is used in the embodiment of the present application to obtain the various preset labels used in the label classification model, which greatly simplifies the number of labels used for video recognition.

[0120] In the embodiment of the present application, the preset labels used in the label classification model are determined by performing the following operations:

[0121] Based on the acquired historical videos, a prediction sample set is constructed;

[0122] Performing encoding processing on each prediction sample in the prediction sample set, determining a sample feature vector of each prediction sample, and determining a feature matching similarity between each sample feature vector and a first feature vector associated with each preset category;

[0123] Based on the feature matching similarity corresponding to each preset category, at least one target category is selected from each preset category, and the prediction samples are divided into each target category;

[0124] Based on the prediction samples contained in each target category, the prediction label corresponding to the target category is determined.

[0125] By using the feature vectors of historical videos, clustering technology is used to construct the preset tags used in the label classification model, thereby building a more accurate labeling system. Subsequently, the video recognition model is constructed using label classification models, text retrieval, knowledge mining, and other technologies. This improves the ability and accuracy of identifying the tags corresponding to videos, improves the quality of the tags associated with videos, and reduces the cost of manually setting labels for videos.

[0126] After introducing the design concepts of the embodiments of the present application, the following briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limiting. In the specific implementation process, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0127] Figure 4An exemplary application scenario diagram of an embodiment of the present application is provided, which includes a terminal device 40 (such as but not limited to 40 - 1 or 40 - 2 shown in the figure) and a server 41;

[0128] The terminal device 40 is an electronic device used by the user, and various application software and websites with functions such as video uploading and video playing are installed and run on the terminal device 40. The terminal device 40 can be a computer device such as a personal computer, mobile phone, tablet computer, notebook computer, car terminal, etc.

[0129] Server 41 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0130] In one possible implementation, the terminal device 40 and the server 41 can communicate via a communication network, which can be a wired network or a wireless network. Therefore, the terminal device 40 and the server 41 can be connected directly or indirectly via wired or wireless communication. For example, the terminal device 40 can be indirectly connected to the server 41 via a wireless access point 42, or directly connected to the server 41 via the Internet, although this application does not limit this.

[0131] In a possible application scenario, when a video viewer browses videos on an application platform in a terminal device 40, the server 41 will receive the video currently browsed by the video viewer on the application platform, and identify the tag corresponding to the video currently browsed by the video viewer. Then, based on the tag corresponding to the video, the server 41 selects a target video with the same tag from multiple videos stored corresponding to the application platform, and recommends the target video to the video viewer, so as to accurately recommend videos of interest to the video viewer based on the tag of the video.

[0132] In a possible application scenario, when a video uploader uploads a video in a certain application platform in a terminal device 40, the server 41 receives the video uploaded by the video uploader on the application platform, and identifies the tag corresponding to the video uploaded by the video uploader. Then, based on the tag corresponding to the video and a plurality of videos stored corresponding to the application platform, an association relationship is established to facilitate video recommendation for video viewers. Obviously, by determining the tagging tags for the video uploader's video, on the one hand, the tagging workload of the tagging personnel can be reduced, and on the other hand, video clustering can be performed through the tags to improve the platform distribution quality. When a user is watching a video, the platform can recommend related videos to the video viewer based on the tags, thereby improving the viewing experience of the video viewer. In addition, when the video viewer is searching for a video, high-quality tags also help the video viewer to accurately obtain related videos.

[0133] It should be noted that the above two application scenarios of video recognition are only examples. The video recognition method of the embodiment of the present application is not only applicable to the above two scenarios. For example, it is also applicable to the tag display below the video, video viewers select videos by tags, operators manage videos by tags, and tags are used as recommendation system features to recommend videos.

[0134] In one possible implementation, in an embodiment of the present application, cloud storage technology can be used to save the above-mentioned training sample data set for training the label classification model; or to store various preset labels obtained through training; or to store historical video related information matched with the video to be processed, wherein the historical video related information includes text information corresponding to the historical video, historical word segmentation corresponding to the historical video, etc.; wherein cloud storage (Cloud Storage) is a new concept extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as the storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and jointly provide data storage and business access functions to the outside world.

[0135] In one possible implementation, to reduce communication latency, servers 41 can be deployed in various regions. Alternatively, for load balancing, different servers 41 can serve the regions corresponding to each terminal device 40. Multiple servers 41 can also share data through blockchain, effectively creating a data sharing system comprised of multiple servers 41. For example, a terminal device 40 located at location a communicates with server 41, while a terminal device 40 located at location b communicates with other servers 41.

[0136] Each server 41 in the data sharing system has a node identifier corresponding to that server 41. Each server 41 in the data sharing system can store the node identifiers of other servers 41 in the data sharing system so that the generated blocks can be broadcast to other servers 41 in the data sharing system based on the node identifiers of other servers 41. Each server 41 can maintain a node identifier list as shown in the table below, and store the server 41 name and node identifier in the node identifier list. The node identifier can be an Internet Protocol (IP) address for interconnection between networks or any other information that can be used to identify the node. Table 1 only uses the IP address as an example for explanation.

[0137] Table 1

[0138] Backend server name Node ID Node 1 119.113.131.174 Node 2 118.116.189.143 … … Node N 119.124.789.238

[0139] Based on the above application scenarios, the video recognition method provided by the exemplary embodiment of the present application is described below in combination with the application scenarios described above according to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the implementation methods of the present application are not subject to any restrictions in this regard.

[0140] Figure 5 A flowchart of a video recognition method in an embodiment of the present application is provided as an example, including the following steps:

[0141] In step S500 , the video to be processed is input into a trained label classification model, and the label classification model is used to match the video to be processed with each preset label to obtain a matching probability value corresponding to each preset label.

[0142] In an embodiment of the present application, the label classification model is pre-trained. Before training the label classification model, a training sample data set for training the label classification model should be obtained first. The training sample data set contains a large number of historical videos with annotated labels.

[0143] Then, according to the training samples in the training sample data set, the label classification model to be trained is subjected to cyclic iterative training, and when the preset convergence conditions are met, the trained label classification model is output, where the preset convergence conditions include but are not limited to the number of training times reaching a threshold and the loss value meeting the convergence value.

[0144] Figure 6 An exemplary flow chart of a method for training a label classification model in an embodiment of the present application is provided, taking one-loop iterative training as an example, wherein the following operations are performed during one-loop iterative training:

[0145] Step S600: Select training samples from the training sample data set, wherein the training samples are historical videos with annotated labels.

[0146] In one possible implementation, the training samples are historical text information corresponding to historical videos, and the historical text information includes but is not limited to the annotation labels of historical videos, historical video title information, historical video ORC text information, and historical video ASR text information. Obviously, in the label classification model training process, training is no longer based solely on a single title information, which improves training reliability and makes the label classification model predict video labels more accurately.

[0147] Step S601: input the training samples into the label classification model.

[0148] In the embodiment of the present application, the label classification model includes but is not limited to an input layer, an encoding layer, fully connected layers (FC) and an output layer.

[0149] The encoding layer is used to convert the historical text information of the input training sample into a historical text feature vector. Its structure is Transformer, and the model parameters select the parameters of the label classification model.

[0150] The fully connected layer acts as a classifier in the network, mapping the historical text feature vector to each preset label and determining the matching similarity between the historical feature vector and each preset label.

[0151] The output layer corresponds to a Sigmoid function, which is used to map the determined matching similarity to a matching probability value. The matching probability value is between (0, 1) and is used to represent the degree of matching between the video and each preset label.

[0152] In this application, during the training process, the training samples are first input into the input layer of the label classification model, and then the input layer inputs the training samples into the encoding layer for encoding processing to determine the historical text feature vector of the historical text information corresponding to the training samples.

[0153] In one possible implementation, the input layer concatenates the various historical text information contained in the training sample, encodes the concatenated historical text information, determines the historical text feature vector, and then maps the historical text feature vector to various preset labels through the fully connected layer.

[0154] When performing the splicing process, a [CLS] classification token is set at the beginning, and the maximum length is set to 256; and the annotation labels of the historical video, the historical video title information, the historical video ORC text information, and the historical video are connected with the special symbol [SEP].

[0155] Step S602 : Based on the fully connected layer in the label classification model, the training samples are mapped to various preset labels, and the matching probability value of each preset label is predicted.

[0156] Exemplarily, the historical text feature vector of the last layer after encoding corresponding to "[CLS]" is mapped to each preset label through the fully connected layer in the label classification model, and the matching probability value of each preset label is predicted.

[0157] In a possible implementation, each preset tag may include a special tag, which is used to indicate that the historical video corresponding to the training sample does not have a corresponding annotated tag.

[0158] Step S603 : Based on the labeled labels corresponding to the training samples and the predicted matching probability values ​​of the preset labels, the binary cross entropy corresponding to the preset labels is determined.

[0159] In one possible implementation, the binary cross entropy corresponds to the following formula:

[0160]

[0161] in, It is the probability that the label classification model predicts each predicted label as the annotated label of the historical video, that is, the matching probability of this application. y is the annotated label. If a predicted label belongs to a positive example, the value is 1, otherwise the value is 0.

[0162] Step S604: construct a loss function based on the binary cross entropy of each preset label, and adjust the parameters of the label classification model based on the loss function.

[0163] In one possible implementation, the results of the binary cross entropy corresponding to each preset label determined based on the above formula are added together to obtain a loss function, and the parameters of the label classification model are adjusted based on the determined loss function.

[0164] In the embodiment of the present application, the matching probability values ​​between the video to be processed and each preset label are obtained respectively through the trained label classification model. Figure 7 A flowchart of a method for determining a matching probability value corresponding to each preset tag using a trained tag classification model in an embodiment of the present application is provided as an example, including the following steps:

[0165] Step S700: Input the video to be processed into the trained label classification model.

[0166] In one possible implementation, the text information of the video to be processed is input into a trained label classification model. The text information to be processed includes but is not limited to the title information of the video to be processed, the ORC text information of the video to be processed, and the ASR text information of the video to be processed.

[0167] Step S701 : Based on the fully connected layer in the tag classification model, the video to be processed is mapped to each preset tag, and the tag matching similarity between the video to be processed and each preset tag is determined respectively.

[0168] In one possible implementation, after the text information to be processed of the video to be processed is input into the label classification model, the text information to be processed of the video to be processed is spliced ​​at the input layer of the label classification model, and then the spliced ​​text information to be processed is converted into a text feature vector to be processed at the encoding layer of the label classification model. For details, please refer to the training process of the label classification model, which will not be repeated here.

[0169] The feature vector of the text to be processed is input into the fully connected layer of the label classification model. Based on the fully connected layer in the label classification model, the video to be processed is mapped to each preset label, and the label matching similarity between the video to be processed and each preset label is determined respectively.

[0170] Step S702 : Based on the Sigmoid function in the tag classification model, convert each tag matching similarity obtained into a matching probability value of the corresponding preset tag.

[0171] The fully connected output is input to the output layer of the label classification model, and the Sigmoid function in the output layer is used to convert the obtained label matching similarities into matching probability values ​​of the corresponding preset labels.

[0172] In the embodiment of the present application, each preset tag used in the tag classification model is pre-constructed based on historical videos. Figure 8 A flowchart of a method for constructing a preset label in an embodiment of the present application is provided as an example, including the following steps:

[0173] Step S800: construct a prediction sample set based on the acquired historical videos.

[0174] At least one of the videos recently viewed by each video viewer on the video platform, that is, the historical videos played, and the videos recently uploaded by each video uploader on the video platform is obtained.

[0175] Historical videos are provided with annotation labels, titles, text information, and voice information. Therefore, text information such as annotation labels, titles, ORC text, and ASR text can be obtained from historical videos. The ORC text is generated by converting the text contained in each frame of the video using ORC technology, while the ASR text is generated by converting the voice information contained in the video using ASR technology.

[0176] Afterwards, the text information such as the annotation labels, titles, ORC texts, ASR texts, etc. corresponding to each historical video are sequentially spliced ​​together to form multiple prediction samples, and multiple prediction samples constitute a sample set.

[0177] In one possible implementation, the prediction sample set is structured as a directed graph of prediction samples. Each node in the graph represents a sample formed by sequentially concatenating the annotation label, title, ORC text, ASR text, and other textual information corresponding to a historical video. The weight of the edge between nodes is the number of shared annotation labels between the two samples. After the directed graph is constructed, the weights of all in-degree edges of each node are normalized by dividing the weights of all in-degree edges by the sum of the in-degrees of the node.

[0178] Step S801 : performing encoding processing on each prediction sample in the prediction sample set, and determining a sample feature vector of each prediction sample.

[0179] In one possible implementation, a deep learning model is used to encode each prediction sample in a sample set, and the first feature vector of the last layer is used as the sample feature vector, so that the sample feature vector of each prediction sample can be determined.

[0180] In an embodiment of the present application, according to the idea of ​​the Graph Neural Networks-Attention (GNN-Attention) mechanism, based on the predicted sample directed graph structure, at least one neighbor node connected to each node and the in-degree of at least one neighbor node are determined, and based on the in-degree of each neighbor node, at least one target neighbor node is determined.

[0181] The target neighbor node is the neighbor node with the largest in-degree or the in-degree reaching the preset value that is selected from each neighbor node based on the in-degree of each neighbor node. Figure 1 For example, the target node is node A. The neighbor nodes of node A include node B, node C, and node D. The in-degree of node B is 1, the in-degree of node C is 2, and the in-degree of node D is 1. If the target neighbor node is selected based on the largest in-degree, the target neighbor node of node A is node C.

[0182] Then, the second eigenvector of each node and the third eigenvector of each corresponding target neighbor node are weighted and summed to obtain the sample eigenvector of each sample;

[0183] The weighted coefficients in the weighted summation process are obtained by performing dot multiplication of the second eigenvector of each node and the third eigenvector of each corresponding target neighbor node with the second eigenvector of the corresponding node and then normalizing the result.

[0184] For example, the number of determined target neighbor nodes is K. When determining the eigenvector corresponding to the target node based on the target node and the determined K target neighbor nodes, the eigenvectors of the (K+1) first-layer outputs are weightedly summed to obtain the eigenvector of the second-layer output of the target node, and the output of the previous layer is sequentially used as the input of the next layer until the output of the last layer, and the first eigenvector of the last layer is used as the sample eigenvector.

[0185] Step S802: determining the feature matching similarity between each sample feature vector and the first feature vector associated with each preset category.

[0186] Step S803 : Based on the feature matching similarities corresponding to the preset categories, at least one target category is selected from the preset categories, and the prediction samples are divided into the target categories.

[0187] It should be noted that the above steps S802-S803 use clustering to determine which preset category the predicted sample is specifically divided into, that is, k-means clustering is used for the obtained sample feature vector. Given that the video corresponding to each sample feature vector can correspond to at least one label, this method modifies the k-means clustering so that each sample can belong to multiple categories.

[0188] In a possible implementation, each preset category is preselected, the number of preset categories is set, and the center point of each preset category is randomly initialized. The center point of each preset category is the first eigenvector associated with the corresponding preset category.

[0189] For each sample feature vector, determining the feature matching similarity between the sample feature vector and the first feature vector of each preset category;

[0190] For example, there are 10 preset categories, namely preset category 1, preset category 2, preset category 3, preset category 4, preset category 5, preset category 6, preset category 7, preset category 8, preset category 9 and preset category 10, and the sample feature vector of the predicted sample A has a feature matching similarity of 95, 90, 88, 86, 97, 99, 80, 93, 60, and 98 respectively.

[0191] Based on the feature matching similarity corresponding to each preset category, at least one target category is selected from each preset category, and the prediction samples are divided into each target category;

[0192] wherein, at least one target category is selected from each preset type in a manner that the feature matching similarity is greater than a feature matching similarity threshold; or the target category is selected from each preset type in a manner that the preset type with the greatest feature matching similarity is selected; or the target category is selected from each preset type in a manner that first determines whether the feature matching similarity reaches the feature matching similarity threshold, and if so, selects the preset type that reaches the feature matching similarity threshold as the target type, otherwise selects the preset type with the greatest feature matching similarity as the target category;

[0193] Taking the example of selecting at least one target category from each preset type in a manner that the feature matching similarity is greater than the feature matching similarity threshold, assuming that the feature matching similarity threshold is 97, preset category 5, preset category 6 and preset category 10 are selected from each preset category as target types, and the predicted sample A is divided into preset category 5, preset category 6 and preset category 10.

[0194] After classifying the prediction sample into each target category, the center point of the corresponding target category is updated. The center point of a target category is the mean of all data in the corresponding target category. That is, after classifying prediction sample A into preset category 5, preset category 6, and preset category 10, the center points of preset category 5, preset category 6, and preset category 10 are updated respectively. Then, steps S802-S803 are repeated.

[0195] It should be noted that in the embodiment of the present application, the center points of each preset category can be randomly initialized, and then the one with the best running results can be selected.

[0196] Step S804 : determining a prediction label corresponding to the target category based on each prediction sample included in each target category.

[0197] Each prediction sample contained in the target category represents the clustering effect, and the corresponding target category is further named based on the clustering effect of each target category, that is, each preset label is obtained.

[0198] The clustering method of the embodiment of the present application can be used to simplify the various preset labels in the label classification model, reduce the difficulty of video recognition, and save time for video recognition.

[0199] In step S501, the video to be processed is input into the trained label retrieval model. The video to be processed is matched with each historical video through the label retrieval model to obtain each candidate annotation label that successfully matches the video to be processed and the corresponding target similarity. The target similarity corresponding to each candidate annotation label is determined based on the video data matching similarity between each historical video associated with the candidate annotation label and the video to be processed.

[0200] In an embodiment of the present application, the tag retrieval model includes at least one of a text retrieval sub-model and a knowledge mining sub-model.

[0201] 1: The tag retrieval model includes the text retrieval sub-model.

[0202] If the tag retrieval model includes a text retrieval sub-model, each candidate tag successfully matched by the video to be processed and the corresponding target similarity are obtained based on the first candidate tag successfully matched by the video to be processed and the corresponding first target similarity.

[0203] Figure 9 A flowchart of a method for obtaining first candidate annotation labels and corresponding first target similarities that successfully match a video to be processed by using a text retrieval sub-model is provided as an example in an embodiment of the present application, including the following steps:

[0204] Step S900: Input the video to be processed into the text retrieval sub-model, match the video to be processed with each historical video through the text retrieval sub-model, and obtain the video data matching similarity between each historical video and the video to be processed;

[0205] The text information to be processed of the video to be processed is input into the text retrieval sub-model and matched with the historical text information of each historical video pre-imported in the text retrieval sub-model to determine the video data matching similarity between the text information to be processed and each historical text information.

[0206] Step S901 : Based on the obtained video data matching similarity associated with each historical video, historical videos whose video data matching similarity reaches a video data matching similarity threshold and associated annotation tags are screened out.

[0207] Please refer to Figure 10 , Figure 10 A schematic diagram of screening a first candidate annotation label in an embodiment of the present application is exemplarily provided.

[0208] The text retrieval sub-model is provided with N historical video information, wherein each historical video corresponds to at least one annotation label.

[0209] At this time, determine the video data matching similarity between the video to be processed and each historical video. Figure 10 It can be seen that the video data matching similarities corresponding to historical video 1, historical video 2, historical video 3, historical video 4, historical video 5, historical video 6...historical video N are 99, 88, 98.5, 97, 89, 74, and 90 respectively;

[0210] The historical videos selected whose video data matching similarity reaches the video data matching similarity threshold of 97 include historical video 1, historical video 3, and historical video 4, among which the annotation tags associated with historical video 1 are annotation tag 1, annotation tag 2, annotation tag 3, and annotation tag 4; the annotation tags associated with historical video 3 are annotation tag 3, annotation tag 8, and annotation tag 9; and the annotation tags associated with historical video 4 are annotation tag 1, annotation tag 3, annotation tag 5, and annotation tag 6.

[0211] Step S902: Based on the annotation tags associated with all the filtered historical videos, obtain the first candidate annotation tags that successfully match the video to be processed, and determine the first target similarity of each first candidate annotation tag based on the video data matching similarity corresponding to each historical video associated with each first candidate annotation tag.

[0212] In the embodiment of the present application, when determining the first target similarity of each first candidate annotation tag based on the video data matching similarity corresponding to each historical video associated with each first candidate annotation tag:

[0213] Case 1: The video data matching similarities corresponding to each historical video associated with each first candidate label tag are respectively used as the candidate similarities associated with the corresponding first candidate label tag, and the maximum value among the candidate similarities associated with each first candidate label tag is respectively used as the first target similarity of the corresponding first candidate label tag;

[0214] At this time, the first candidate annotation tags successfully matched with the video to be processed are obtained as annotation tag 1, annotation tag 2, annotation tag 3, annotation tag 4, annotation tag 5, annotation tag 6, annotation tag 8, and annotation tag 9. Annotation tag 1 is associated with historical video 1 and historical video 4, and the video data matching similarity corresponding to historical video 1 is 99, and the video data matching similarity corresponding to historical video 4 is 97. Therefore, the candidate similarities associated with annotation tag 1 in the first candidate annotation tags are 99 and 97. Similarly, the candidate similarity associated with annotation tag 2 is 99, the candidate similarities associated with annotation tag 3 are 99, 98.5, and 97, the candidate similarity associated with annotation tag 4 is 99, the candidate similarity associated with annotation tag 5 is 97, the candidate similarity associated with annotation tag 6 is 97, the candidate similarity associated with annotation tag 8 is 98.5, and the candidate similarity associated with annotation tag 9 is 98.5.

[0215] If the maximum value of the candidate similarities associated with each first candidate label is taken as the first target similarity of the corresponding first candidate label, the first target similarity of label 1 is 99. Similarly, the first target similarity of label 2 is 99, the first target similarity of label 3 is 99, the first target similarity of label 4 is 99, the first target similarity of label 5 is 97, the first target similarity of label 6 is 97, the first target similarity of label 8 is 98.5, and the first target similarity of label 9 is 98.5.

[0216] Case 2: The video data matching similarities corresponding to each historical video associated with each first candidate labeling tag are respectively used as candidate similarities associated with the corresponding first candidate labeling tag, and the candidate similarities associated with each first candidate labeling tag are weighted averaged to obtain corresponding weighted average processing results. The first target similarity of each first candidate labeling tag is determined based on each weighted average processing result;

[0217] If the candidate similarities associated with each first candidate labeling tag are weighted averaged to obtain the corresponding weighted average processing results, and the first target similarities of each first candidate labeling tag are determined based on each weighted average processing result, the first target similarity of labeling tag 1 is 98. Similarly, the first target similarity of labeling tag 2 is 99, the first target similarity of labeling tag 3 is 98.1, the first target similarity of labeling tag 4 is 99, the first target similarity of labeling tag 5 is 97, the first target similarity of labeling tag 6 is 97, the first target similarity of labeling tag 8 is 98.5, and the first target similarity of labeling tag 9 is 98.5.

[0218] 2: The label retrieval model includes the knowledge mining sub-model.

[0219] If the label retrieval model includes a knowledge mining sub-model, each candidate labeling label successfully matched with the to-be-processed video and the corresponding target similarity are obtained based on each second candidate labeling label successfully matched with the to-be-processed video and the corresponding second target similarity obtained with the knowledge mining sub-model.

[0220] Figure 11 A flowchart of a method for obtaining each second candidate annotation label and the corresponding second target similarity that successfully matches the video to be processed by using a knowledge mining sub-model in an embodiment of the present application is exemplified, including the following steps:

[0221] Step S1100: Input the video to be processed into the knowledge mining sub-model. The knowledge mining sub-model matches the to-be-processed segmentation associated with the video to be processed with each historical segmentation associated with each historical video, and obtains the video data matching similarity between each historical segmentation and the to-be-processed segmentation.

[0222] Each historical segmentation is a segmentation that coexists with at least one annotated tag associated with the corresponding historical video, and each historical segmentation corresponds to at least one annotated tag, and each annotated tag is associated with a coexistence probability of the corresponding historical segmentation.

[0223] The text information to be processed of the video to be processed is input into the knowledge mining sub-model. Through the knowledge mining model, the word to be processed in the text information to be processed is matched with the historical word of each historical video pre-imported into the knowledge mining sub-model to determine the video data matching similarity between the text information to be processed and each historical word.

[0224] Among them, each pre-imported historical segmentation is based on the historical text information of historical videos selected offline. After deleting high-frequency words or stop words in the historical text information, the probability of the co-existence between the remaining segmentations in the historical text information and the annotated labels of the corresponding historical videos, that is, the co-existence probability, is determined. For example, if the co-existence probability of segmentation A and annotated label A in historical video A reaches a threshold, the co-existence probability of segmentation B and annotated label B reaches a threshold, the co-existence probability of segmentation B and annotated label C reaches a threshold, and so on, then the relationship between segmentation A and annotated label A and the co-existence probability between annotated label A and segmentation A, the relationship between segmentation B and annotated label B and the co-existence probability between annotated label B and segmentation B, the relationship between segmentation B and annotated label C and the co-existence probability between annotated label C and segmentation B are stored in the knowledge mining sub-model. At this time, the historical segmentations include segmentation A and segmentation B.

[0225] Step S1101: Based on the obtained video data matching similarity associated with each historical segmentation, filter out the historical segmentations and the associated annotation tags whose video data matching similarity reaches the video data matching similarity threshold;

[0226] Please refer to Figure 12 , Figure 12 A schematic diagram of screening a second candidate annotation label in an embodiment of the present application is exemplarily provided.

[0227] There are N historical participles set in the knowledge mining sub-model, where each historical participle corresponds to at least one annotation label. The annotation label corresponding to the historical participle is the annotation label that can coexist with the historical participle, and each annotation label corresponds to the coexistence probability of the corresponding historical participle.

[0228] At this time, determine the video data matching similarity between the to-be-processed segmentation and each historical segmentation. Figure 11 It can be seen that the video data matching similarities corresponding to historical segmentation word 1, historical segmentation word 2, historical segmentation word 3, historical segmentation word 4, historical segmentation word 5, historical segmentation word 6... historical segmentation word N are 99, 88, 98.5, 97, 89, 74, and 90 respectively;

[0229] The historical segmentations whose video data matching similarity reaches the video data matching similarity threshold of 97 are screened out, including historical segmentation 1, historical segmentation 3, and historical segmentation 4, among which the annotation tags associated with historical segmentation 1 are annotation tag 1, annotation tag 2, annotation tag 3, and annotation tag 4; the annotation tags associated with historical segmentation 3 are annotation tag 3, annotation tag 8, and annotation tag 9; and the annotation tags associated with historical segmentation 4 are annotation tag 1, annotation tag 3, annotation tag 5, and annotation tag 6.

[0230] In step S1102, based on the various annotation tags associated with all the filtered historical word segmentations, the second candidate annotation tags that successfully match the video to be processed are obtained, and based on the coexistence probability associated with each second candidate annotation tag and the video data matching similarity between the corresponding historical word segmentation and the video to be processed, the second target similarity corresponding to each second candidate annotation tag is determined.

[0231] In the embodiment of the present application, based on the co-existence probability associated with each second candidate annotation tag and the video data matching similarity between the corresponding historical segmentation and the video to be processed, when determining the second target similarity corresponding to each second candidate annotation tag:

[0232] The video data matching similarity between each historical segmentation and the segmentation to be processed is used as the candidate similarity associated with each second candidate annotation label corresponding to each historical segmentation;

[0233] At this time, the second candidate annotation labels successfully matched with the video to be processed are annotation label 1, annotation label 2, annotation label 3, annotation label 4, annotation label 5, annotation label 6, annotation label 8, and annotation label 9. Annotation label 1 is associated with historical segmentation 1 and historical segmentation 4, and the video data matching similarity corresponding to historical segmentation 1 is 99, and the video data matching similarity corresponding to historical segmentation 4 is 97. Therefore, the candidate similarities associated with annotation label 1 in the second candidate annotation labels are 99 and 97. Similarly, the candidate similarity associated with annotation label 2 is 99, the candidate similarities associated with annotation label 3 are 99, 98.5, and 97, the candidate similarity associated with annotation label 4 is 99, the candidate similarity associated with annotation label 5 is 97, the candidate similarity associated with annotation label 6 is 97, the candidate similarity associated with annotation label 8 is 98.5, and the candidate similarity associated with annotation label 9 is 98.5.

[0234] Based on the coexistence probability associated with each second candidate annotation tag, each corresponding candidate similarity is weighted to obtain a corresponding processing result, and the second target similarity corresponding to each second candidate annotation tag is determined based on the processing result, wherein the weight coefficient of each weighted processing is the coexistence probability associated with the corresponding second candidate annotation tag.

[0235] In one possible implementation, when a label is associated with at least two candidate similarities, the coexistence probability between the historical participles and the label corresponding to each candidate similarity is determined respectively, and the corresponding candidate similarities are weighted based on the determined coexistence probability to obtain corresponding processing results, and based on the processing results, the maximum value of the processing results is selected as the second target similarity corresponding to each second candidate label or the weighted average of the processing results is selected as the second target similarity corresponding to the second candidate label.

[0236] For example, the candidate similarities of the annotation tag 1 are 99 and 97, and the corresponding historical segmentations are historical segmentations 1 and historical segmentations 4, respectively. The coexistence probabilities between historical segmentations 1 and historical segmentations 4 and the annotation tags are 99% and 97%, respectively. At this time, the video data matching similarity of 99 corresponding to historical segmentation 1 and the coexistence probability with the annotation tag 1 are weighted (99*99%) to obtain a corresponding processing result of 98. The video data matching similarity of 97 corresponding to historical segmentation 4 and the coexistence probability with the annotation tag 1 are weighted (97*97%) to obtain a corresponding processing result of 94. If the maximum value method is used to determine the second target similarity corresponding to the second candidate annotation tag, the second target similarity of the annotation tag 1 is 98. The second similarities of other annotation tags are determined in the same way, which will not be repeated here.

[0237] 3: The tag retrieval model includes a text retrieval sub-model and a knowledge mining sub-model.

[0238] For details, please refer to Case 1 and Case 2 above, which will not be repeated here.

[0239] Step S502 , screening out preset tags whose matching probability values ​​reach a probability threshold to form a first tag set, and screening out candidate annotation tags whose target similarity reaches a similarity threshold to form a second tag set.

[0240] Reference Figure 10 For illustration, if the similarity threshold is 98.5, the candidate annotation tags screened out are annotation tag 2, annotation tag 4, annotation tag 8 and annotation tag 9 respectively. At this time, the second tag set includes annotation tag 2, annotation tag 4, annotation tag 8 and annotation tag 9.

[0241] Reference Figure 12For illustration, if the similarity threshold is 96, the candidate annotation labels screened out are annotation label 1, annotation label 2, annotation label 3 and annotation label 5 respectively. At this time, the second label set includes annotation label 1, annotation label 2, annotation label 3 and annotation label 5.

[0242] In the embodiment of the present application, whether the first tag set and the second tag set contain tags is related to the matching probability values ​​corresponding to the respective preset tags obtained based on the tag classification model. That is, after determining that the target tag exists in the video to be processed based on the matching probability values ​​corresponding to the respective preset tags obtained based on the tag classification model, preset tags whose matching probability values ​​reach a probability threshold are screened out to form the first tag set, and candidate annotated tags whose target similarity reaches a similarity threshold are screened out to form the second tag set. In this case, both the first tag set and the second tag set include at least one tag.

[0243] In a possible implementation, if each preset tag includes a special tag for indicating that a video does not have a marked tag, then it is determined whether the video to be processed has a target tag based on the matching probability value corresponding to the feature tag;

[0244] For example, if the matching probability value corresponding to the feature label is greater than or equal to 0.5, it is determined that the target label does not exist in the video to be processed; otherwise, it is determined that the label exists in the video to be processed.

[0245] It should be noted that the above determination conditions are merely examples, and this application may adjust the determination conditions based on actual circumstances. For example, when the matching probability value corresponding to the feature tag is greater than 0.6, it is determined that the target tag does not exist in the video to be processed. The number of target tags corresponding to the video to be processed may also be determined based on the matching probability value corresponding to the feature tag. For example, when the matching probability value corresponding to the feature tag is less than 0.3, the video to be processed corresponds to 10 target tags.

[0246] In another possible implementation, if the preset tags do not include a special tag for indicating that the video does not have a marked tag, it is determined whether the to-be-processed video has a target tag based on the matching probability values ​​of the preset tags.

[0247] For example, determine whether the matching probability values ​​of each preset tag are all lower than the set threshold. If so, it means that the target tag does not exist in the video to be processed; or determine whether the target tag exists in the video to be processed based on the average value of the matching probability values ​​of each preset tag. If the average value is greater than the set threshold, it means that the target tag exists in the video to be processed. Those skilled in the art can set the judgment conditions according to actual conditions, and the examples are not listed here in an exhaustive manner.

[0248] In the embodiment of the present application, the above-mentioned tag retrieval model includes a text retrieval sub-model, and the tag retrieval model includes a knowledge mining sub-model:

[0249] If it is determined based on each matching probability value that the target label exists in the video to be processed, and each matching probability value is less than the probability threshold, a first number of candidate annotation labels whose target similarity reaches the similarity threshold are screened out to form a second label set; taking the above-mentioned label retrieval model including the text retrieval sub-model as an example, according to Figure 10 To illustrate, if the first number is set to 4 and the similarity threshold is 98, the four target labels with the highest similarity are selected as label 2, label 4, label 8 and label 9. At this time, the second label set includes label 2, label 4, label 8 and label 9.

[0250] If it is determined based on the matching probability value that the target tag exists in the video to be processed, and at least one matching probability value reaches the probability threshold, the preset tags whose matching probability value reaches the probability threshold are screened out to form a first tag set, and a second number of candidate annotation tags whose target similarity reaches the similarity threshold are screened out to form a second tag set, wherein the first number is greater than the second number; taking the above-mentioned tag retrieval model including the text retrieval sub-model as an example, according to Figure 10 For example, if the second number is set to 2 and the similarity threshold is 98, the two tags with the highest target similarity are selected as tag 2 and tag 4, and the second tag set includes tag 2 and tag 4.

[0251] If it is determined based on each matching probability value that the target tag does not exist in the video to be processed, then both the first tag set and the second tag set are empty sets.

[0252] In the embodiment of the present application, for the case where the above-mentioned tag retrieval model includes a text retrieval sub-model and a knowledge mining sub-model:

[0253] If it is determined based on each matching probability value that the target label exists in the video to be processed, and each matching probability value is less than the probability threshold, a first number of first candidate label labels whose first target similarity reaches a first similarity threshold are screened out from each first candidate label label; and a first number of second candidate label labels whose second target similarity reaches a second similarity threshold are screened out from each second candidate label label; and a second label set is formed based on the same candidate label labels among the screened out first candidate label labels and the screened out second candidate label labels;

[0254] Reference Figure 10 and Figure 12, when the first number is set to 4, the first similarity threshold is set to 98.5, and the second similarity threshold is set to 96, the first candidate annotation labels screened out include annotation label 2, annotation label 4, annotation label 8, and annotation label 9, and the second candidate annotation labels screened out are annotation label 1, annotation label 2, annotation label 3, and annotation label 5;

[0255] Combine Figure 10 and Figure 12 , refer to Figure 13 An embodiment of the present application provides a schematic diagram of determining a target tag for a video to be processed. It can be seen that the second tag set only includes the annotation tag 2.

[0256] If it is determined based on the matching probability value that the target tag exists in the video to be processed, and at least one matching probability value reaches the probability threshold, preset tags whose matching probability values ​​reach the probability threshold are screened out to form a first tag set; and a second number of first candidate tag tags whose first target similarity reaches the first similarity threshold are screened out from each first candidate tag tag; and a second number of second candidate tag tags whose second target similarity reaches the second similarity threshold are screened out from each second candidate tag tag; and a second tag set is formed based on the same candidate tag tags among the screened out first candidate tag tags and the screened out second candidate tag tags, wherein the first number is greater than the second number;

[0257] Reference Figure 10 and Figure 12 , when the second number is set to 3, the first similarity threshold is 98.5, and the second similarity threshold is 96, the first candidate annotation labels screened out include annotation label 2, annotation label 4, and annotation label 8, and the second candidate annotation labels screened out are annotation label 1, annotation label 2, and annotation label 3; combined Figure 10 and Figure 12 , refer to Figure 13 An embodiment of the present application provides a schematic diagram of determining a target label for a video to be processed. It can be seen that the second label set only includes the annotation label 2.

[0258] If it is determined based on each matching probability value that the target tag does not exist in the video to be processed, then both the first tag set and the second tag set are empty sets.

[0259] Step S503: The first tag set and the second tag set are fused to determine a target tag corresponding to the video to be processed.

[0260] In the embodiment of the present application, all tags in the first tag set and the second tag set are used as target tags corresponding to the video to be processed.

[0261] Reference Figure 13It can be seen that the first label set includes label 1, label 2, label 3 and label 5. It should be noted that the matching probability values ​​corresponding to label 1, label 2, label 3 and label 5 in the first label set are all greater than the matching probability threshold; the second label set only includes label 2, so the target labels corresponding to the video to be processed are label 1, label 2, label 3 and label 5.

[0262] The following is combined with Figure 3 The overall method of video recognition in the embodiment of this application is explained. Figure 14 An exemplary flow chart of an overall method for video recognition in an embodiment of the present application is provided, including the following steps:

[0263] Step S1400, obtaining the text information to be processed corresponding to the video to be processed;

[0264] The text information to be processed includes the title, description, ORC text, ASR text and other text information corresponding to the video to be processed.

[0265] In step S1401 , the text information to be processed is input into a label classification model, and the label classification model is used to match the text information to be processed with various preset labels to obtain matching probability values ​​corresponding to various preset labels.

[0266] Step S1402 : Based on the matching probability values, preset tags whose matching probability values ​​reach a probability threshold are screened out to form a first tag set.

[0267] Step S1403 , input the text information to be processed into the text retrieval sub-model, and match the text information to be processed with the historical text information corresponding to each historical video through the text retrieval sub-model to obtain the historical text information that successfully matches the text information to be processed.

[0268] Step S1404: Determine a first candidate annotation tag corresponding to the text information to be processed based on the annotation tags corresponding to the successfully matched historical text information; and determine a first target similarity corresponding to the first candidate annotation tag based on the video data matching similarity between each historical text information associated with the first candidate annotation tag and the text information to be processed.

[0269] Step S1405 : Based on the first target similarity, first candidate annotation tags whose first target similarity reaches a first similarity threshold are screened out.

[0270] Step S1406: input the text information to be processed into the knowledge mining sub-model, and match the to-be-processed segmented words in the text information to be processed with the historical segmented words corresponding to each historical video through the knowledge mining sub-model to obtain the historical segmented words that successfully match the text information to be processed.

[0271] Step S1407: Determine the second candidate annotation tag corresponding to the text information to be processed based on the annotation tags that coexist with the successfully matched historical segmentations and the coexistence probabilities corresponding to the annotation tags; and determine the second target similarity corresponding to the second candidate annotation tag based on the video data matching similarities between each historical segmentation associated with the second candidate annotation tag and the segmentation to be processed and the corresponding coexistence probabilities.

[0272] Step S1408 : Based on the second target similarity, second candidate annotation tags whose second target similarity reaches a second similarity threshold are screened out.

[0273] Step S1409 : forming a second tag set based on the same candidate tagging tags among the filtered first candidate tagging tags and the filtered second candidate tagging tags.

[0274] Step S1410: All tags in the first tag set and the second tag set are used as target tags corresponding to the video to be processed.

[0275] Based on the same inventive concept, the embodiment of the present application further provides a video recognition device 1500, Figure 15 An exemplary embodiment of the present application provides a video recognition device, which includes:

[0276] The first obtaining unit 1501 is configured to input the video to be processed into a trained label classification model, match the video to be processed with each preset label through the label classification model, and obtain a matching probability value corresponding to each preset label;

[0277] The second obtaining unit 1502 is configured to input the video to be processed into the trained label retrieval model, match the video to be processed with each historical video through the label retrieval model, and obtain each candidate annotation label successfully matched with the video to be processed and the corresponding target similarity, wherein the target similarity corresponding to each candidate annotation label is determined based on the video data matching similarity between each historical video associated with the candidate annotation label and the video to be processed;

[0278] The screening unit 1503 is used to screen out preset tags whose matching probability values ​​reach a probability threshold to form a first tag set, and screen out candidate annotation tags whose target similarity reaches a similarity threshold to form a second tag set;

[0279] The determining unit 1504 is configured to fuse the first tag set and the second tag set to determine a target tag corresponding to the video to be processed.

[0280] In a possible implementation, the trained label classification model in the first obtaining unit is obtained by training in the following manner:

[0281] According to the training samples in the training sample data set, the label classification model to be trained is subjected to cyclic iterative training, and when the preset convergence conditions are met, the trained label classification model is output. The following operations are performed during one cyclic iterative training process:

[0282] Selecting training samples from a training sample dataset, wherein the training samples are historical videos with annotated labels;

[0283] Input the training samples into the label classification model;

[0284] Based on the fully connected layer in the label classification model, the training samples are mapped to various preset labels, and the matching probability value of each preset label is predicted;

[0285] Based on the matching probability values ​​of the labeled labels corresponding to the training samples and the predicted preset labels, the binary cross entropy corresponding to each preset label is determined;

[0286] A loss function is constructed based on the binary cross entropy of each preset label, and the parameters of the label classification model are adjusted based on the loss function.

[0287] In one possible implementation, the video to be processed is input into a label classification model, and the label identification model is used to match the video to be processed with each preset label to obtain a matching probability value corresponding to each preset label, including:

[0288] Input the video to be processed into the trained label classification model;

[0289] Based on the fully connected layer in the label classification model, the video to be processed is mapped to each preset label, and the label matching similarity between the video to be processed and each preset label is determined respectively;

[0290] Based on the Sigmoid function in the label classification model, the obtained label matching similarities are converted into matching probability values ​​of the corresponding preset labels.

[0291] In a possible implementation, the tag retrieval model in the second acquisition unit 1502 includes at least one of a text retrieval sub-model and a knowledge mining sub-model;

[0292] If the tag retrieval model includes a text retrieval sub-model, each candidate tag successfully matched by the to-be-processed video and the corresponding target similarity are obtained based on the first candidate tag successfully matched by the to-be-processed video and the corresponding first target similarity;

[0293] If the label retrieval model includes a knowledge mining sub-model, each candidate labeling label successfully matched with the to-be-processed video and the corresponding target similarity are obtained based on each second candidate labeling label successfully matched with the to-be-processed video and the corresponding second target similarity obtained with the knowledge mining sub-model.

[0294] In a possible implementation, if the tag retrieval model in the second acquisition unit 1502 includes a text retrieval sub-model;

[0295] The second acquisition unit 1502 obtains each first candidate annotation label and the corresponding first target similarity that are successfully matched to the video to be processed by the following method:

[0296] Input the video to be processed into the text retrieval sub-model, match the video to be processed with each historical video through the text retrieval sub-model, and obtain the video data matching similarity between each historical video and the video to be processed;

[0297] Based on the obtained video data matching similarity associated with each historical video, historical videos whose video data matching similarity reaches the video data matching similarity threshold and the associated annotation tags are screened out;

[0298] Based on the various annotation tags associated with all the screened historical videos, the first candidate annotation tags that successfully match the video to be processed are obtained, and based on the video data matching similarity corresponding to the various historical videos associated with the various first candidate annotation tags, the first target similarity of each first candidate annotation tag is determined.

[0299] In a possible implementation, the second acquiring unit 1502 is specifically configured to:

[0300] The video data matching similarities corresponding to the historical videos associated with the first candidate label tags are respectively used as candidate similarities associated with the corresponding first candidate label tags, and the maximum value among the candidate similarities associated with the first candidate label tags is respectively used as the first target similarity of the corresponding first candidate label tags; or

[0301] The video data matching similarities corresponding to each historical video associated with each first candidate annotation tag are respectively used as the candidate similarities associated with the corresponding first candidate annotation tag, and the candidate similarities associated with each first candidate annotation tag are weighted averaged to obtain the corresponding weighted average processing results, and the first target similarities of each first candidate annotation tag are respectively determined based on the weighted average processing results.

[0302] In a possible implementation, if the label retrieval model in the second acquisition unit 1502 includes a knowledge mining sub-model;

[0303] The second acquisition unit 1502 obtains each second candidate annotation label and the corresponding second target similarity that are successfully matched with the video to be processed by the following method:

[0304] The video to be processed is input into the knowledge mining sub-model. Through the knowledge mining sub-model, the to-be-processed segmentation associated with the video to be processed is matched with each historical segmentation associated with each historical video, and the video data matching similarity between each historical segmentation and the to-be-processed segmentation is obtained respectively; wherein each historical segmentation is a segmentation that coexists with at least one annotated label associated with the corresponding historical video, and each historical segmentation corresponds to at least one annotated label, and each annotated label is associated with a coexistence probability of the corresponding historical segmentation;

[0305] Based on the obtained video data matching similarity associated with each historical segmentation, the historical segmentations and associated annotation tags whose video data matching similarity reaches the video data matching similarity threshold are screened out;

[0306] Based on the various annotation tags associated with all the filtered historical word segmentations, the second candidate annotation tags that successfully match the video to be processed are obtained, and based on the coexistence probability associated with each second candidate annotation tag and the video data matching similarity between the corresponding historical word segmentation and the video to be processed, the second target similarity corresponding to each second candidate annotation tag is determined.

[0307] In a possible implementation, the second acquiring unit 1502 is specifically configured to:

[0308] The video data matching similarity between each historical segmentation and the segmentation to be processed is used as the candidate similarity associated with each second candidate annotation label corresponding to each historical segmentation;

[0309] Based on the coexistence probability associated with each second candidate annotation tag, each corresponding candidate similarity is weighted to obtain a corresponding processing result, and the second target similarity corresponding to each second candidate annotation tag is determined based on the processing result, wherein the weight coefficient of each weighted processing is the coexistence probability associated with the corresponding second candidate annotation tag.

[0310] In a possible implementation, if the label retrieval model in the second acquisition unit 1502 includes a text retrieval sub-model and a knowledge mining sub-model;

[0311] The screening unit 1503 is specifically used for:

[0312] Based on the first target similarity, selecting a first candidate annotation tag whose first target similarity reaches a first similarity threshold from each first candidate annotation tag; and

[0313] Based on the second target similarity, selecting second candidate annotation tags whose second target similarity reaches a second similarity threshold from each second candidate annotation tag;

[0314] Based on the same candidate labeling tags among the screened first candidate labeling tags and the screened second candidate labeling tags, a second labeling set is formed;

[0315] The determining unit 1504 is specifically configured to:

[0316] All tags in the first tag set and the second tag set are used as target tags corresponding to the video to be processed.

[0317] In a possible implementation, the screening unit 1503 is specifically configured to:

[0318] If it is determined based on each matching probability value that the target label exists in the video to be processed, and each matching probability value is less than the probability threshold, a first number of candidate annotation labels whose target similarity reaches the similarity threshold are screened out to form a second label set;

[0319] If it is determined based on the matching probability value that the target tag exists in the video to be processed, and at least one matching probability value reaches the probability threshold, then the preset tags whose matching probability values ​​reach the probability threshold are screened out to form a first tag set, and a second number of candidate annotation tags whose target similarities reach the similarity threshold are screened out to form a second tag set, wherein the first number is greater than the second number;

[0320] If it is determined based on each matching probability value that the target tag does not exist in the video to be processed, then both the first tag set and the second tag set are empty sets.

[0321] In a possible implementation, each preset label included in the label classification model in the first acquisition unit 1501 is determined in the following manner:

[0322] Based on the acquired historical videos, a prediction sample set is constructed;

[0323] Performing encoding processing on each prediction sample in the prediction sample set, determining a sample feature vector of each prediction sample, and determining a feature matching similarity between each sample feature vector and a first feature vector associated with each preset category;

[0324] Based on the feature matching similarity corresponding to each preset category, at least one target category is selected from each preset category, and the prediction samples are divided into each target category;

[0325] Based on the prediction samples contained in each target category, the prediction label corresponding to the target category is determined.

[0326] In a possible implementation, the prediction sample set is a prediction sample directed graph structure, wherein each node in the prediction sample directed graph structure represents a sample consisting of a historical video;

[0327] Encoding is performed on each sample in the prediction sample set, and the sample feature vector of each sample is determined respectively, including:

[0328] Determine at least one neighbor node connected to each node and the in-degree of at least one neighbor node based on the predicted sample directed graph structure, and determine at least one target neighbor node based on the in-degree of each neighbor node;

[0329] Perform weighted sum processing on the second eigenvector of each node and the third eigenvector of each corresponding target neighbor node to obtain the sample eigenvector of each sample;

[0330] The weighted coefficients in the weighted summation process are obtained by performing dot multiplication of the second eigenvector of each node and the third eigenvector of each corresponding target neighbor node with the second eigenvector of the corresponding node and then normalizing the result.

[0331] For the convenience of description, the above sub-models are divided into units (or modules) according to their functions and described separately. Of course, when implementing this application, the functions of each unit (or module) can be implemented in the same or multiple software or hardware.

[0332] After introducing the method and apparatus for video recognition according to an exemplary embodiment of the present application, a computing device for video recognition according to another exemplary embodiment of the present application is introduced next.

[0333] Those skilled in the art will appreciate that various aspects of the present application can be implemented as systems, methods, or program products. Therefore, various aspects of the present application can be specifically implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation that combines hardware and software aspects, which may be collectively referred to herein as a "circuit," "module," or "system."

[0334] In one possible implementation, the video recognition computing device provided in the embodiments of the present application may include at least a processor and a memory. The memory stores program code that, when executed by the processor, causes the processor to perform any step of the video recognition method of the various exemplary embodiments of the present application.

[0335] Refer to the following Figure 16 The video recognition computing device 1600 according to this embodiment of the present application is described as follows. Figure 16The video recognition computing device 1600 is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0336] like Figure 16 As shown, the sub-models of the computing device 1600 may include but are not limited to: the at least one processor 1601 mentioned above, the at least one memory 1602 mentioned above, and a bus 1603 connecting different system sub-models (including the memory 1602 and the processor 1601).

[0337] Bus 1603 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a processor or local bus using any of a variety of bus architectures.

[0338] The memory 1602 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 16021 and / or a cache memory 16022 , and may further include a read-only memory (ROM) 16023 .

[0339] Memory 1602 may also include a program / utility 16025 having a set (at least one) of program modules 16024, such program modules 16024 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0340] The computing device 1600 may also communicate with one or more external devices 1604 (e.g., a keyboard, a pointing device, etc.), one or more devices that enable a user to interact with the computing device 1600, and / or any device that enables the computing device 1600 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication may occur through an input / output (I / O) interface 1605. Furthermore, the computing device 1600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 1606. Figure 16 As shown, the network adapter 1606 communicates with the other modules of the computing device 1600 via the bus 1603. Figure 16 Not shown, other hardware and / or software modules may be used in conjunction with computing device 1600, including but not limited to microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0341] In some possible implementations, various aspects of the video recognition method provided in the present application may also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the video recognition method according to various exemplary embodiments of the present application described above in this specification.

[0342] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0343] The program product for controlling the sending of short messages according to the embodiment of the present application may be in the form of a portable compact disk read-only memory (CD-ROM) and include program codes, and may be run on a computing device.

[0344] A readable signal medium may include a data signal propagated in baseband or as a carrier wave, wherein the data signal carries a readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with a command execution system, apparatus, or device.

[0345] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0346] The program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages.

[0347] It should be noted that although several units or subunits of the device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, depending on the embodiment of the present application, the feature vectors and functions of two or more units described above can be embodied in a single unit. Conversely, the feature vectors and functions of a single unit described above can be further divided and embodied by multiple units.

[0348] Furthermore, although the operations of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0349] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A video recognition method, characterized in that: The method includes: Input the video to be processed into the trained label classification model, match the video to be processed with each preset label through the label classification model, and obtain the matching probability value corresponding to each preset label; The video to be processed is input into a trained label retrieval model to obtain each candidate labeling label successfully matched by the video to be processed and the corresponding target similarity; wherein the label retrieval model includes a knowledge mining sub-model, and each candidate labeling label successfully matched by the video to be processed and the corresponding target similarity include each second candidate labeling label successfully matched by the video to be processed obtained based on the knowledge mining sub-model and the corresponding second target similarity; each second candidate labeling label and the corresponding second target similarity are obtained in the following manner: through the knowledge mining sub-model, the to-be-processed segmentation associated with the video to be processed is matched with each historical segmentation associated with each historical video, and the video data matching similarity between each historical segmentation and the to-be-processed segmentation is obtained respectively; wherein each historical segmentation is a segmentation that coexists with at least one annotation tag associated with the corresponding historical video, and each historical segmentation corresponds to at least one annotation tag, and each annotation tag is associated with a coexistence probability of the corresponding historical segmentation; based on the obtained video data matching similarity associated with each historical segmentation, historical segmentations and associated annotation tags whose video data matching similarity reaches a video data matching similarity threshold are screened out; based on the annotation tags associated with all the screened historical segmentations, each second candidate annotation tag that successfully matches the to-be-processed video is obtained, and based on the coexistence probability associated with each second candidate annotation tag and the video data matching similarity between the corresponding historical segmentation and the to-be-processed video, a second target similarity corresponding to each second candidate annotation tag is determined; Screening out preset tags whose matching probability values ​​reach a probability threshold to form a first tag set, and screening out candidate annotation tags whose target similarity reaches a similarity threshold to form a second tag set; The first tag set and the second tag set are fused to determine a target tag corresponding to the video to be processed.

2. The method according to claim 1, wherein The trained label classification model is obtained by training in the following way: According to the training samples in the training sample data set, the label classification model to be trained is subjected to cyclic iterative training, and when the preset convergence conditions are met, the trained label classification model is output. The following operations are performed during one cyclic iterative training process: Selecting a training sample from the training sample data set, wherein the training sample is a historical video with annotated labels; Inputting the training samples into the label classification model; Based on the fully connected layer in the label classification model, the training samples are mapped to various preset labels, and the matching probability values ​​of the various preset labels are predicted; Determining the binary cross entropy corresponding to each preset label based on the annotated label corresponding to the training sample and the predicted matching probability value of each preset label; A loss function is constructed based on the binary cross entropy of the preset labels, and parameters of the label classification model are adjusted based on the loss function.

3. The method according to claim 1, wherein Inputting the video to be processed into the label classification model, matching the video to be processed with each preset label through the label classification model, and obtaining a matching probability value corresponding to each preset label, including: Inputting the video to be processed into the trained label classification model; Based on the fully connected layer in the label classification model, the video to be processed is mapped to each preset label, and the label matching similarity between the video to be processed and each preset label is determined respectively; Based on the Sigmoid function in the tag classification model, each obtained tag matching similarity is converted into a matching probability value of the corresponding preset tag.

4. The method according to claim 1, wherein The tag retrieval model also includes a text retrieval sub-model; The candidate annotation tags successfully matched to the video to be processed and the corresponding target similarities also include the first candidate annotation tags successfully matched to the video to be processed and the corresponding first target similarities obtained based on the text retrieval sub-model.

5. The method according to claim 4, wherein If the tag retrieval model includes a text retrieval sub-model; The first candidate annotation labels and the corresponding first target similarities of the successfully matched video to be processed are obtained in the following manner: Inputting the video to be processed into the text retrieval sub-model, matching the video to be processed with each of the historical videos through the text retrieval sub-model, and obtaining the video data matching similarity between each of the historical videos and the video to be processed; Based on the obtained video data matching similarity associated with each historical video, historical videos whose video data matching similarity reaches the video data matching similarity threshold and the associated annotation tags are screened out; Based on the various annotation tags associated with all the screened historical videos, the various first candidate annotation tags that successfully match the video to be processed are obtained, and based on the video data matching similarity corresponding to the various historical videos associated with the various first candidate annotation tags, the first target similarity of the various first candidate annotation tags is determined.

6. The method according to claim 5, wherein The determining of the first target similarity of each first candidate annotation tag based on the video data matching similarity corresponding to each historical video associated with each first candidate annotation tag includes: The video data matching similarities corresponding to the historical videos associated with the first candidate annotation tags are respectively used as candidate similarities associated with the corresponding first candidate annotation tags, and the maximum value among the candidate similarities associated with the first candidate annotation tags is respectively used as the first target similarity of the corresponding first candidate annotation tag; or The video data matching similarities corresponding to the historical videos associated with the first candidate annotation tags are respectively used as candidate similarities associated with the corresponding first candidate annotation tags, and the candidate similarities associated with the first candidate annotation tags are weighted averaged to obtain corresponding weighted average processing results, and the first target similarities of the first candidate annotation tags are respectively determined based on the weighted average processing results.

7. The method according to claim 1, wherein The determining of the second target similarity corresponding to each second candidate annotation tag based on the co-existence probability associated with each second candidate annotation tag and the video data matching similarity between the corresponding historical segmentation word and the video to be processed includes: The video data matching similarities between each historical segmentation and the segmentation to be processed are respectively used as candidate similarities associated with each second candidate annotation label corresponding to each historical segmentation; Based on the coexistence probabilities associated with the respective second candidate annotation tags, weighted processing is performed on the corresponding candidate similarities to obtain corresponding processing results, and the second target similarities corresponding to the respective second candidate annotation tags are determined based on the processing results, wherein the weight coefficient of each weighted processing is the coexistence probability associated with the corresponding second candidate annotation tag.

8. The method according to claim 4, wherein If the tag retrieval model includes the text retrieval sub-model and the knowledge mining sub-model; The candidate annotation tags whose target similarity reaches the similarity threshold are screened out to form a second tag set, including: Based on the first target similarity, selecting a first candidate labeling label whose first target similarity reaches a first similarity threshold from the first candidate labeling labels; and Based on the second target similarity, screening out second candidate labeling labels whose second target similarity reaches a second similarity threshold from the second candidate labeling labels; forming a second label set based on the same candidate labeling labels among the first candidate labeling labels selected and the second candidate labeling labels selected; The fusing the first tag set and the second tag set to determine a target tag corresponding to the video to be processed includes: All tags in the first tag set and the second tag set are used as target tags corresponding to the video to be processed.

9. The method according to any one of claims 1 to 8, wherein: The preset tags whose matching probability values ​​reach the probability threshold are screened out to form a first tag set, and the candidate annotation tags whose target similarity reaches the similarity threshold are screened out to form a second tag set, including: If it is determined that the target tag exists in the video to be processed based on each of the matching probability values, and each of the matching probability values ​​is less than the probability threshold, a first number of candidate annotation tags whose target similarities reach the similarity threshold are screened out to form a second tag set; If it is determined based on the matching probability value that the target tag exists in the video to be processed, and at least one of the matching probability values ​​reaches the probability threshold, then the preset tags whose matching probability values ​​reach the probability threshold are screened out to form a first tag set, and a second number of candidate annotation tags whose target similarities reach the similarity threshold are screened out to form a second tag set, wherein the first number is greater than the second number; If it is determined based on each of the matching probability values ​​that the video to be processed does not have a target tag, then both the first tag set and the second tag set are empty sets.

10. The method according to claim 1, wherein The preset labels are determined as follows: Based on the acquired historical videos, a prediction sample set is constructed; Performing encoding processing on each prediction sample in the prediction sample set, determining a sample feature vector of each prediction sample, and determining a feature matching similarity between each sample feature vector and a first feature vector associated with each preset category; Based on the feature matching similarities corresponding to the respective preset categories, at least one target category is selected from the respective preset categories, and the prediction samples are divided into the respective target categories; Based on the prediction samples included in the target categories, a prediction label corresponding to the target category is determined.

11. The method according to claim 10, wherein The prediction sample set is a prediction sample directed graph structure, wherein each node in the prediction sample directed graph structure represents a sample consisting of a historical video; The encoding process is performed on each sample in the prediction sample set to determine the sample feature vector of each sample, including: Determining, based on the predicted sample directed graph structure, at least one neighbor node connected to each node and an in-degree of the at least one neighbor node, and determining at least one target neighbor node based on the in-degree of each neighbor node; Performing weighted summation processing on the second eigenvector of each node and the third eigenvector of each corresponding target neighbor node to obtain a sample eigenvector of each sample; The weighted coefficients in the weighted summation process are obtained by performing dot multiplication of the second eigenvector of each node and the third eigenvector of each corresponding target neighbor node with the second eigenvector of the corresponding node and then normalizing the result.

12. A video recognition device, characterized in that: The device includes: A first obtaining unit is configured to input a to-be-processed video into a trained label classification model, match the to-be-processed video with each preset label through the label classification model, and obtain a matching probability value corresponding to each preset label; The second obtaining unit is used to input the video to be processed into the trained label retrieval model to obtain each candidate labeling label that successfully matches the video to be processed and the corresponding target similarity; wherein, the label retrieval model includes a knowledge mining sub-model, and each candidate labeling label that successfully matches the video to be processed and the corresponding target similarity include each second candidate labeling label that successfully matches the video to be processed obtained based on the knowledge mining sub-model and the corresponding second target similarity; each second candidate labeling label and the corresponding second target similarity are obtained in the following manner: through the knowledge mining sub-model, the to-be-processed segmentation associated with the video to be processed is matched with each historical segmentation associated with each historical video, and the video data matching between each historical segmentation and the to-be-processed segmentation is obtained respectively. Similarity; wherein, each historical segmentation is a segmentation that coexists with at least one annotation tag associated with the corresponding historical video, and each historical segmentation corresponds to at least one annotation tag, and each annotation tag is associated with a coexistence probability of the corresponding historical segmentation; based on the obtained video data matching similarity associated with each historical segmentation, historical segmentations and associated annotation tags whose video data matching similarity reaches a video data matching similarity threshold are screened out; based on the annotation tags associated with all the screened historical segmentations, each second candidate annotation tag that successfully matches the to-be-processed video is obtained, and based on the coexistence probability associated with each second candidate annotation tag and the video data matching similarity between the corresponding historical segmentation and the to-be-processed video, a second target similarity corresponding to each second candidate annotation tag is determined; a screening unit, configured to screen out preset tags whose matching probability values ​​reach a probability threshold to form a first tag set, and to screen out candidate annotation tags whose target similarity reaches a similarity threshold to form a second tag set; The determining unit is configured to fuse the first tag set and the second tag set to determine a target tag corresponding to the video to be processed.

13. A video recognition device, characterized in that: The device comprises: a memory and a processor, wherein the memory is used to store computer instructions; and the processor is used to execute the computer instructions to implement the method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images