Video marking method, device and electronic equipment

Abstract

The present application provides a video tagging method, device and electronic device, the method comprising: obtaining candidate concept tags corresponding to the video to be tagged; inputting the video to be tagged, the associated text corresponding to the video to be tagged, and the candidate concept tags into the target tag confidence model to obtain the confidence; determining the target concept tag according to the confidence, and tagging the video to be tagged. The multimodal features corresponding to the video to be tagged can be determined according to the video to be tagged and the associated text corresponding to the video to be tagged, and then the candidate concept tags corresponding to the video to be tagged can be screened according to the multimodal features corresponding to the video to be tagged, and the target concept tags that can accurately represent the superordinate concepts of the video to be tagged are obtained, and the video to be tagged is marked with the target concept tags, which improves the richness of video tags, provides more reference basis for video operation platforms to manage videos, and also provides users with richer means to find videos.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of multimedia technology, and in particular to a video marking method and device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the popularization of streaming media and the Internet, more and more videos are uploaded to video platforms by users. The number of videos on video platforms is increasing day by day. In order to facilitate the operators of video platforms to manage videos and users to find videos, corresponding tags are usually generated for videos based on the content of the videos.

[0003] In related technologies, technicians usually first build a tag library based on the categories, fields, subjects, etc. that the video may involve, and perform word segmentation on the title content of the video, and compare the word segmentation results with the tags in the tag library. If the word segmentation results match the tags in the tag library, the tags that match the word segmentation results in the tag library will be used as the tags of the video.

[0004] In the current solution, the form of labeling for videos is relatively simple, and label matching can only reflect the content of the video itself, which makes it difficult to summarize the video content. It is not conducive to users' retrieval of videos, nor is it conducive to the management of videos by video operation platforms. Summary of the invention

[0005] The embodiments of the present application provide a video tagging method and device, an electronic device, a computer-readable storage medium, and a computer program product to solve the problem in the related art that video tags are less rich and cannot summarize video content.

[0006] In a first aspect, an embodiment of the present application provides a video marking method, the method comprising:

[0007] Obtaining a candidate concept tag corresponding to the video to be marked, wherein the candidate concept tag is used to represent a candidate superordinate concept of the content of the video to be marked;

[0008] The video to be marked, the associated text corresponding to the video to be marked, and the candidate concept label are input into a target label confidence model to determine the confidence of the candidate concept label, wherein the confidence is used to characterize the probability that the candidate concept label is a superordinate concept of the video to be marked. The target label confidence model is used to determine the multimodal features of the video to be marked according to the video to be marked and the associated text, determine the label features of the candidate concept label according to the associated text, and determine the confidence according to the multimodal features and the label features;

[0009] A target concept tag is determined from the candidate concept tags according to the confidence level, and the to-be-tagged video is tagged according to the target concept tag.

[0010] In an optional implementation manner, obtaining candidate concept tags corresponding to the video to be labeled includes:

[0011] Obtaining a video content tag of the video to be marked, wherein the video content tag is used to represent the content of the video to be marked;

[0012] According to the video content tag, a candidate concept tag matching the video content tag is obtained from a concept tag library, wherein the concept tag library stores a correspondence between content tags and concept tags.

[0013] In an optional implementation, the method further includes:

[0014] Get preset content tags from the preset content tag library;

[0015] Searching a preset dictionary and / or web terms to determine a concept sentence associated with the preset content tag;

[0016] Inputting the preset content tag and the concept sentence into a target concept matching model to obtain a concept sentence matching degree output by the target concept matching model, wherein the concept sentence matching degree is used to represent the probability that the concept sentence is a superordinate concept of the preset content tag;

[0017] Determining a candidate concept tag corresponding to the preset content tag from the concept tag according to the concept sentence matching degree;

[0018] The concept tag library is constructed according to the correspondence between the preset content tags and the candidate concept tags.

[0019] In an optional implementation manner, obtaining the video content tag of the video to be tagged includes:

[0020] A semantic analysis is performed on the video to be marked and / or the associated text corresponding to the video to be marked, and a video content tag is determined from a preset content tag library according to the semantic analysis result.

[0021] In an optional implementation, the method further includes:

[0022] Acquire a sample content label, a sample concept sentence corresponding to the sample content label, and sentence annotation information of each of the sample concept sentences, wherein the sentence annotation information is used to indicate whether the sample concept sentence is a superordinate concept of the sample content label or not;

[0023] Inputting the first correspondence between the sample content label and the sample concept sentence into the initial concept matching model to obtain the sample concept sentence matching degree output by the initial concept matching model, wherein the sample concept sentence matching degree is used to characterize the probability that the sample concept sentence is a superordinate concept of the sample content label;

[0024] A first loss value is determined according to the sample concept sentence matching degree and the sentence annotation information, and the initial concept matching model is trained according to the first loss value and a preset loss function to obtain the target concept matching model.

[0025] In a second aspect, an embodiment of the present application provides a method for generating a target label confidence model, the method comprising:

[0026] Inputting a sample video and a sample candidate concept label corresponding to the sample video as training data into an initial label confidence model, obtaining the confidence of the sample candidate concept label output by the initial label confidence model, wherein the confidence is used to characterize the probability that the sample candidate concept label is a superordinate concept of the sample video;

[0027] A second loss value is determined based on the confidence and the label annotation information corresponding to the sample video, and the initial label confidence model is trained based on the second loss value and a preset loss function to obtain a target label confidence model as described in any one of claims 1 to 5, wherein the label annotation information is used to characterize whether the sample candidate concept label is or is not a superordinate concept of the sample video.

[0028] In an optional implementation, the step of inputting the sample video and the sample candidate concept labels corresponding to the sample video as training data into the initial label confidence model to obtain the confidence of the sample candidate concept labels output by the initial label confidence model includes:

[0029] Acquire a second correspondence between a sample video, a sample associated text of the sample video, a sample candidate concept label corresponding to the sample video, and label annotation information of each sample candidate concept label, wherein the sample associated text is a text content describing the sample video;

[0030] The second corresponding relationship is input as training data into the initial label confidence model so that the initial label confidence model determines the video features of the sample video, the first text features of the sample associated text, and the second text features of the sample candidate concept label, and outputs the confidence of the sample candidate concept label based on the video features, the first text features, and the second text features.

[0031] In an optional implementation, the initial label confidence model includes a first subnetwork and a second subnetwork, and the second corresponding relationship is input into the initial label confidence model as training data so that the initial label confidence model determines the video feature of the sample video, the first text feature of the sample associated text, and the second text feature of the sample candidate concept label, including:

[0032] Inputting the sample video in the second corresponding relationship into the first sub-network, and obtaining the video features of the sample video output by the first sub-network;

[0033] The sample associated text and the sample candidate concept label in the second corresponding relationship are input into the second sub-network, and the first text feature of the sample associated text and the second text feature of the sample candidate concept label output by the second sub-network are obtained.

[0034] In an optional implementation manner, inputting the sample video in the second corresponding relationship into the first sub-network, and obtaining the video features of the sample video output by the first sub-network includes:

[0035] Inputting a sample video frame of the sample video into the first sub-network, and obtaining a video frame feature of each video frame output by the first sub-network;

[0036] The video frame features are summed and averaged to obtain the video features.

[0037] In an optional implementation, outputting the confidence of the sample candidate concept label according to the video feature, the first text feature, and the second text feature includes:

[0038] Adding the video feature and the first text feature to obtain a multimodal feature, wherein the multimodal feature is used to represent a fusion feature of the video feature and the associated text feature of the sample video;

[0039] Outputting the confidence of the sample candidate concept label according to the multimodal feature and the second text feature.

[0040] In an optional implementation, outputting the confidence of the sample candidate concept label according to the multimodal feature and the second text feature includes:

[0041] Obtaining a dot product result of the multimodal feature and the second text feature;

[0042] The dot product result is normalized to obtain and output the confidence of the sample candidate concept label.

[0043] In a third aspect, an embodiment of the present application provides a video marking device, the device comprising:

[0044] A candidate concept tag acquisition module is configured to acquire a candidate concept tag corresponding to the video to be tagged, wherein the candidate concept tag is used to represent a candidate superordinate concept of the content of the video to be tagged;

[0045] The label confidence determination module is configured to input the video to be labeled, the associated text corresponding to the video to be labeled, and the candidate concept label into a target label confidence model to determine the confidence of the candidate concept label, wherein the confidence is used to characterize the probability that the candidate concept label is a superordinate concept of the video to be labeled, and the target label confidence model is used to determine the multimodal features of the video to be labeled based on the video to be labeled and the associated text, determine the label features of the candidate concept label based on the associated text, and determine the confidence based on the multimodal features and the label features;

[0046] The target concept label determination module is configured to determine a target concept label from the candidate concept labels according to the confidence level, and label the to-be-labeled video according to the target concept label.

[0047] In an optional embodiment, the device further comprises:

[0048] A content tag determination module is configured to obtain a video content tag of the video to be tagged, wherein the video content tag is used to represent the content of the video to be tagged;

[0049] The concept tag library module is configured to obtain a candidate concept tag matching the video content tag from a concept tag library according to the video content tag, wherein the concept tag library stores a correspondence between content tags and concept tags.

[0050] In an optional embodiment, the device further comprises:

[0051] A content tag acquisition module is configured to acquire a preset content tag from a preset content tag library;

[0052] A concept sentence determination module is configured to search a preset dictionary and / or network terms to determine a concept sentence associated with the preset content tag;

[0053] A sentence matching degree determination module is configured to input the preset content tag and the concept sentence into a target concept matching model to obtain the concept sentence matching degree output by the target concept matching model, wherein the concept sentence matching degree is used to characterize the probability that the concept sentence is a superordinate concept of the preset content tag;

[0054] A candidate concept tag determination module is configured to determine a candidate concept tag corresponding to the preset content tag from the concept sentence according to the matching degree of the concept sentence;

[0055] The concept tag library module is configured to construct the concept tag library according to the corresponding relationship between the preset content tags and the candidate concept tags.

[0056] In an optional implementation manner, the preset content tag determination module includes:

[0057] The content tag determination submodule is configured to perform semantic analysis on the video to be tagged and / or the associated text corresponding to the video to be tagged, and determine the video content tag from a preset content tag library according to the semantic analysis result.

[0058] In an optional embodiment, the device further comprises:

[0059] An information acquisition module is configured to acquire a sample content label, a sample concept sentence corresponding to the sample content label, and sentence annotation information of each sample concept sentence, wherein the sentence annotation information is used to indicate whether the sample concept sentence is a superordinate concept of the sample content label or not;

[0060] An information input module is configured to input the first correspondence between the sample content label and the sample concept sentence into an initial concept matching model to obtain a sample concept sentence matching degree output by the initial concept matching model, wherein the sample concept sentence matching degree is used to characterize the probability that the sample concept sentence is a superordinate concept of the sample content label;

[0061] The parameter adjustment module is configured to determine a first loss value based on the sample concept sentence matching degree and the sentence annotation information, and train the initial concept matching model based on the first loss value and a preset loss function to obtain the target concept matching model.

[0062] In a fourth aspect, an embodiment of the present application provides a device for generating a target label confidence model, the device comprising:

[0063] A confidence determination module is configured to input a sample video and a sample candidate concept label corresponding to the sample video as training data into an initial label confidence model, and obtain the confidence of the sample candidate concept label output by the initial label confidence model, wherein the confidence is used to characterize the probability that the sample candidate concept label is a superordinate concept of the sample video;

[0064] The model training module is configured to determine a second loss value based on the confidence and the label annotation information corresponding to the sample video, and train the initial label confidence model based on the second loss value and a preset loss function to obtain a target label confidence model, wherein the label annotation information is used to characterize whether the sample candidate concept label is or is not a superordinate concept of the sample video.

[0065] In an optional implementation, the confidence determination module includes:

[0066] An information acquisition submodule is configured to acquire a second correspondence between a sample video, a sample associated text of the sample video, a sample candidate concept label corresponding to the sample video, and label annotation information of each sample candidate concept label, wherein the sample associated text is a text content describing the sample video;

[0067] The information input submodule is configured to input the second corresponding relationship as training data into the initial label confidence model, so that the initial label confidence model determines the video features of the sample video, the first text features of the sample associated text, and the second text features of the sample candidate concept label, and outputs the confidence of the sample candidate concept label based on the video features, the first text features, and the second text features.

[0068] In an optional implementation, the information input submodule includes:

[0069] A first sub-network input sub-module is configured to input the sample video in the second corresponding relationship into the first sub-network, and obtain the video features of the sample video output by the first sub-network;

[0070] The second sub-network input sub-module is configured to input the sample associated text and the sample candidate concept label in the second corresponding relationship into the second sub-network, and obtain the first text feature of the sample associated text and the second text feature of the sample candidate concept label output by the second sub-network.

[0071] In an optional implementation, the first sub-network input submodule includes:

[0072] a video frame submodule, configured to input a sample video frame of the sample video into the first subnetwork, and obtain a video frame feature of each video frame output by the first subnetwork;

[0073] The video feature submodule is configured to sum and average the video frame features to obtain the video features.

[0074] In an optional implementation, the information input submodule includes:

[0075] A multimodal submodule, configured to add the video feature and the first text feature to obtain a multimodal feature, wherein the multimodal feature is used to represent a fusion feature of the video feature and the associated text feature of the sample video;

[0076] The confidence submodule is configured to output the confidence of the sample candidate concept label according to the multimodal feature and the second text feature.

[0077] In an optional implementation, the confidence submodule includes:

[0078] A dot product submodule, configured to obtain a dot product result of the multimodal feature and the second text feature;

[0079] The normalization submodule is configured to perform normalization processing on the dot product result to obtain and output the confidence of the sample candidate concept label.

[0080] In a fifth aspect, an embodiment of the present application further provides an electronic device comprising a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the above-mentioned video labeling method and / or target label confidence model generation method.

[0081] In a sixth aspect, an embodiment of the present application further provides a computer-readable storage medium, which, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to execute the above-mentioned video labeling method and / or target label confidence model generation method.

[0082] In a seventh aspect, an embodiment of the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned video labeling method and / or target label confidence model generation method.

[0083] In an embodiment of the present application, the candidate concept tags corresponding to the video to be marked can be obtained; the video to be marked, the associated text corresponding to the video to be marked, and the candidate concept tags are input into the target tag confidence model to obtain the confidence; the target concept tag is determined according to the confidence, and the video to be marked is marked. The multimodal features corresponding to the video to be marked can be determined according to the video to be marked and the associated text corresponding to the video to be marked, and then the candidate concept tags corresponding to the video to be marked can be screened according to the multimodal features corresponding to the video to be marked, and the target concept tag that can accurately represent the superordinate concept of the video to be marked is obtained, and the video to be marked is marked with the target concept tag, which improves the richness of the video tags, provides more reference basis for the video operation platform to manage videos, and provides users with richer means to find videos.

[0084] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0086] Figure 1 is a flowchart of the steps of a video marking method provided by an embodiment of the present application;

[0087] Figure 2 is a flowchart of another video marking method provided by an embodiment of the present application;

[0088] Figure 3 It is a flowchart of a method for generating a target label confidence model provided in an embodiment of the present application;

[0089] Figure 4 is a structural diagram of a video marking device provided in an embodiment of the present application;

[0090] Figure 5 It is a structural diagram of a target label confidence model generation device provided in an embodiment of the present application;

[0091] Figure 6 is a block diagram of an electronic device according to an embodiment of the present application;

[0092] Figure 7 It is a logic block diagram of an electronic device according to another embodiment of the present application. DETAILED DESCRIPTION

[0093] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0094] Figure 1 is a flowchart of a video marking method provided by an embodiment of the present application. Figure 1 As shown, the method includes:

[0095] Step 101: Obtain candidate concept tags corresponding to the video to be tagged, wherein the candidate concept tags are used to represent candidate superordinate concepts of the content of the video to be tagged.

[0096] In the embodiment of the present application, the video can be a video (such as a film and television video, a news video, etc.) released by the operator of the video platform, or a video (such as a short video, etc.) uploaded by a user to the video platform, or any other type of video. The embodiment of the present application does not impose any restrictions on the type of video. The operator of the video platform often releases new videos on the video platform, and users also release videos on the video platform. After a long time, there will be a large number of videos on the video platform. By labeling these videos with concept tags, it is easy to manage the videos, and it is also easy to recommend videos to users. At the same time, it is also convenient for users to find videos according to concept tags.

[0097] A concept tag may be a tag that represents a higher-level concept of the video content. For example, if the content of a video is related to the TV series Nirvana in Fire, the concept tag corresponding to the video may be "high-scoring political drama" to reflect that the video content Nirvana in Fire is a highly-rated political drama. The concept tag corresponding to the video may also be "popular costume drama" to reflect that the video content Nirvana in Fire is a relatively popular costume drama.

[0098] Since there may be large differences between videos with the same video content, for example, video A is a video with the novel Nirvana in Fire, while video B is a video with the TV series Nirvana in Fire, then although the contents of video A and video B both contain Nirvana in Fire, in essence, video A and video B have completely different superordinate concepts. The concept label of video A can be "high-scoring political conspiracy novel" to indicate that video A is a high-scoring political conspiracy novel, while the concept label of video B can be "high-scoring political conspiracy TV series" to indicate that video B is a high-scoring political conspiracy TV series. Therefore, when determining the concept label corresponding to a video, it is necessary to screen the candidate concept labels corresponding to the video content and select the concept label that is more in line with the actual content of the video from the candidate concept labels corresponding to the video content.

[0099] Specifically, multiple videos may be classified in advance according to the video content, and at least one video content information may be established, and then the candidate concept label corresponding to each video content information for expressing the superordinate concept of the video content information may be determined by manual annotation or other methods, and the corresponding relationship between the video content information and the candidate concept label may be established. For a video to be labeled, the video content information of the video to be labeled is first determined, and then the candidate concept label corresponding to the video content information is determined from the corresponding relationship between the video content information and the candidate concept label according to the video content information.

[0100] Step 102, the video to be marked, the associated text corresponding to the video to be marked, and the candidate concept label are input into the target label confidence model to determine the confidence of the candidate concept label, wherein the confidence is used to characterize the probability that the candidate concept label is the superordinate concept of the video to be marked. The target label confidence model is used to determine the multimodal features of the video to be marked based on the video to be marked and the associated text, determine the label features of the candidate concept label based on the associated text, and determine the confidence based on the multimodal features and the label features.

[0101] In the embodiment of the present application, the video to be marked refers to the video itself waiting to be tagged with an appropriate conceptual label. Correspondingly, the video on the video platform is usually accompanied by associated text, such as the title of the video, the description of the video, the comments of the video, etc. These text information associated with the video are usually related to the video content and are used to describe, introduce, and evaluate the video content.

[0102] Specifically, since the video to be marked and the associated text corresponding to the video to be marked can reflect the complete content of the video to be marked, it is necessary to input the video to be marked and the associated text corresponding to the video to be marked into the target label confidence model, so that the target label confidence model determines the relatively complete multimodal features corresponding to the video to be marked according to the input video to be marked and the associated text corresponding to the video to be marked. The multimodal features can not only reflect the features of the image of the video to be marked, but also reflect the features in the associated text of the video to be marked.

[0103] At the same time, it is also necessary to input the obtained candidate concept labels of the video to be labeled into the target label confidence model, so that the target label confidence model determines the label features of the candidate concept labels according to the associated text corresponding to the video to be labeled, and compares the label features with the multimodal features of the video to be labeled, thereby outputting the confidence of the candidate concept labels, wherein the confidence of the candidate concept label is used to characterize the probability that the candidate concept label is the superordinate concept of the video to be labeled, that is, if the candidate concept label can accurately reflect the superordinate concept of the video to be labeled, then the confidence of the candidate concept label is high, and if the candidate concept label cannot accurately reflect the superordinate concept of the video to be labeled, then the confidence of the candidate concept label is low.

[0104] It should be noted that, since a video to be labeled may correspond to more than one candidate concept label, each candidate concept label corresponding to the video to be labeled needs to be input into the target label confidence model so that the target label confidence model outputs the confidence of each candidate concept label corresponding to the video to be labeled.

[0105] Step 103: determine a target concept tag from the candidate concept tags according to the confidence level, and tag the to-be-tagged video according to the target concept tag.

[0106] After determining the confidence of each candidate concept label corresponding to the video to be labeled, all candidate concept labels corresponding to the video to be labeled can be sorted according to the size of the confidence, and a preset number of candidate concept labels corresponding to larger confidences are selected as target concept labels according to the confidence from large to small.

[0107] A target label confidence threshold may also be preset, and candidate concept labels whose confidence is greater than or equal to the target label confidence threshold are selected as target candidate concept labels.

[0108] In addition, the target concept label can be determined from the candidate concept labels according to the confidence of the candidate concept labels in other ways. The embodiment of the present application does not specifically limit the method of selecting the target concept label. The technical staff can flexibly adjust the selection method according to actual needs to obtain the target concept label that accurately reflects the superordinate concept of the video to be labeled from the candidate concept labels, and mark the video to be labeled with the target concept label.

[0109] In summary, a video tagging method provided in an embodiment of the present application can determine the candidate concept tags corresponding to the video to be tagged, and determine the multimodal features corresponding to the video to be tagged based on the video to be tagged and the associated text corresponding to the video to be tagged, and then screen the candidate concept tags corresponding to the video to be tagged through the multimodal features corresponding to the video to be tagged, and obtain the target concept tags that can accurately represent the superordinate concept of the video to be tagged, and tag the video to be tagged with the target concept tags, thereby improving the richness of video tags, providing more reference basis for video operation platforms to manage videos, and providing users with richer means to find videos.

[0110] Figure 2 is a flowchart of another video marking method provided by an embodiment of the present application. Figure 2 As shown, the method includes:

[0111] Step 201: Obtain a preset content tag from a preset content tag library.

[0112] Preset content tags are tags generated based on video content and are generally used to summarize video content to facilitate video classification and search.

[0113] Content tags can be created by users themselves or generated by the operator of the video platform through semantic analysis of the video. In the embodiment of the present application, content tags are any tags directly generated by the video itself, which are used to describe the video itself, for example, "Nirvana in Fire", "Nirvana in Fire TV series", "Nirvana in Fire novel", etc.

[0114] The video operator can store content tags created by users in the past, content tags obtained by semantic analysis of the video, and content tags obtained by other means to form a preset content tag library.

[0115] Since content tags can directly reflect the content of the video, they can be used directly as video content information when establishing a concept tag library. This allows the concept tag library to be directly established based on the existing data stored in the video operation system. This eliminates the need to repeatedly analyze the video content to determine the video content information, thereby improving the efficiency of establishing a concept tag library.

[0116] Specifically, all or part of the content tags in the preset content tag library can be obtained, and corresponding concept tags that can express their superordinate concepts are determined for these obtained content tags. That is, in the embodiment of the present application, the concept tags are obtained from the content tags, and the content tags are directly obtained based on the video content.

[0117] Step 202: Search a preset dictionary and / or network terms to determine a concept sentence associated with the preset content tag.

[0118] The preset dictionary may include any traditional dictionary containing descriptions of hyponymy, such as Xinhua Dictionary, Oxford Dictionary, Cihai, etc., and may also include any knowledge base containing descriptions of hyponymy, such as Da Ci Lin, OpenConcepts, Qilu, etc. The embodiment of the present application does not specifically limit the type of the preset dictionary.

[0119] Network terms may include encyclopedia content on the Internet, such as entries in Internet encyclopedias such as "Baidu Encyclopedia" and "Wikipedia". Furthermore, network terms may also include any text content with a hierarchical relationship description, such as news content and forum content on the Internet.

[0120] In an embodiment of the present application, the text corresponding to the preset content tag can be searched in the preset dictionary and / or network entry, and the conceptual sentence expressing the superordinate concept of the preset content tag can be extracted after the location of the preset content tag in the preset dictionary and / or network entry.

[0121] Specifically, a concept relationship statement can be added after the preset content tag to form a search string corresponding to the preset content tag, wherein the concept relationship statement is a statement in natural language that indicates a belonging relationship, such as "is", "is", "belongs to", "is a", etc. For example, if a preset content tag is "Langya Bang", then by adding a concept association statement after the preset content tag "Langya Bang", a search string "Langya Bang is", "Langya Bang is", "Langya Bang belongs to", "Langya Bang is a", etc. can be formed. It should be noted that the concept relationship statements are not limited to the ones shown in the above examples, and technicians can flexibly set them according to actual needs.

[0122] Furthermore, the concept association statements corresponding to all preset content tags for forming the search string may be the same or different. For example, different concept relationship statements may be set according to the different contents expressed by the preset content tags to better match the preset content tags. For example, since the preset content tag Nirvana in Fire is a TV series, the concept association statement corresponding to the preset content tag Nirvana in Fire may be "is a TV series", and the preset content tag "off-road vehicle" is a car, so the concept association statement corresponding to the preset content tag "off-road vehicle" may be "is a car".

[0123] After generating a search string according to a preset content tag, the search string is searched in a preset dictionary and / or network entry and the position of the search string in the preset dictionary and / or network entry is determined, and a limiting string after the position of the search string is extracted as a concept tag corresponding to the preset content tag.

[0124] Specifically, a semantic analysis may be performed on the string after the search string position to determine an independent concept after the search string position, and the independent concept may be extracted as a concept label of the content label. Alternatively, the string after the search string position to the first punctuation mark may be extracted, and the string may be determined as a concept label corresponding to the preset content label.

[0125] It should be noted that in the embodiments of the present application, technicians can flexibly set the size of the range of preset dictionaries and / or network terms to be searched according to actual needs to achieve a balance between completeness and speed.

[0126] Step 203: Obtain a sample content label, a sample concept sentence corresponding to the sample content label, and sentence annotation information of each sample concept sentence, wherein the sentence annotation information is used to indicate whether the sample concept sentence is a superordinate concept of the sample content label or not.

[0127] At least one preset content tag is obtained as a sample content tag, and a concept sentence corresponding to the at least one preset content tag is obtained as a sample concept sentence corresponding to the sample content tag.

[0128] Specifically, the preset content labels in the concept statement library can be sorted from large to small according to the number of corresponding candidate concept statements, and the preset content labels corresponding to the preset sample number with the highest ranking can be selected as sample content labels, which can provide a richer number of sample concept statements and improve the accuracy of the model training results. Content labels can also be selected from the concept statement library as sample content labels in other ways, such as manual selection, random selection, etc., which are not specifically limited in the embodiments of the present application.

[0129] The sentence annotation information represents whether the sample concept sentence is or is not a superordinate concept of the sample content label. It can be determined whether the sample concept sentence is a superordinate concept of the corresponding sample content label by manual annotation. For example, if the sample concept sentence A is manually verified to be a superordinate concept of the corresponding sample content label C, the sentence annotation information of the sample concept sentence A can be set to 1. If the sample concept sentence B is not manually verified to be a superordinate concept of the corresponding sample content label C, the sentence annotation information of the sample concept sentence B can be set to 0.

[0130] A training set can be formed by obtaining sample content labels, sample concept sentences corresponding to the sample content labels, and sentence annotation information of each sample concept sentence, so as to train the initial concept matching model.

[0131] Step 204, input the first correspondence between the sample content label and the sample concept sentence into the initial concept matching model to obtain the sample concept sentence matching degree output by the initial concept matching model, wherein the sample concept sentence matching degree is used to characterize the probability that the sample concept sentence is the superordinate concept of the sample content label.

[0132] The initial concept matching model may include BERT (Bidirectional Encoder Representations from Transformers), Deep (Bidirectional) Recurrent Neural Network (Deep (Bidirectional) Recurrent Neural Network), LSTM (Long Short-Term Memory), and Convolutional Neural Network (CNN) and other network models.

[0133] The initial concept matching model needs to be trained before it can be used. During training, the first correspondence between at least one sample content label and the sample concept sentence corresponding to each sample content label is input into the initial concept matching model as a training sample. The initial concept matching model outputs the sample concept sentence matching degree for each sample concept sentence input, where the sample concept sentence matching degree is used to indicate the probability that the sample concept sentence is the superordinate concept of the sample content label corresponding to the sample concept sentence.

[0134] Specifically, when the initial concept matching model includes a BERT model, each sample content label and its corresponding sample concept sentence may be composed into a string “[CLS] is a [SEP]" inputs the initial concept matching model to conform to the input format of the BERT model. Among them, It can represent the i-th sample content label of the input initial concept matching model, that is, the lower concept, It can represent a sample concept sentence corresponding to the i-th sample content label input into the initial concept matching model, that is, the superordinate concept.

[0135] Step 205, determining a first loss value according to the sample concept sentence matching degree and the sentence annotation information, and training the initial concept matching model according to the first loss value and a preset loss function to obtain the target concept matching model.

[0136] Each sample concept sentence corresponding to the sample content label corresponds to a sentence annotation information. After the initial concept matching model outputs a sample concept sentence matching degree corresponding to a sample concept sentence, the loss value between the sample concept sentence matching degree and the sentence annotation information of the sample concept sentence is calculated by a preset loss function, and the parameters in the initial concept matching model are adjusted according to the loss value. After multiple iterations of training, the training of the initial concept matching model is completed to obtain the target concept matching model. The target concept matching model can determine the probability that the superordinate concept represents the superordinate concept of the video content information based on the input video content information and the superordinate concept. The video content information may include content labels, and the superordinate concept may include candidate concept sentences.

[0137] Step 206, input the preset content tag and the concept sentence into the target concept matching model to obtain the concept sentence matching degree output by the target concept matching model, wherein the concept sentence matching degree is used to characterize the probability that the concept sentence is the superordinate concept of the preset content tag.

[0138] The target concept matching model is used to determine the probability that the preset content tag and the concept sentence have a hyponymous relationship, that is, to determine the probability that the concept sentence can represent the hypernymous concept of the preset content tag.

[0139] After determining the concept sentence of the preset content label from the preset dictionary and / or network entries, the preset content label and all corresponding concept sentences can be input into the trained target concept matching model to obtain the concept sentence matching degree corresponding to each concept sentence output by the target concept matching model. The concept sentence matching degree is used to characterize the probability that the concept sentence is the superordinate concept of the preset content label. That is to say, the smaller the concept sentence matching degree of a concept sentence is, the more the concept sentence cannot accurately represent the superordinate concept of the corresponding preset content label, and the larger the concept sentence matching degree of a concept sentence is, the more accurately the concept label can represent the superordinate concept of the corresponding preset content label.

[0140] Step 207: Determine a candidate concept tag corresponding to the preset content tag from the concept tag according to the concept tag matching degree.

[0141] After determining the concept sentence matching degree of each concept sentence corresponding to the preset content label, all concept sentences corresponding to each preset content label can be sorted according to the size of the concept sentence matching degree, and concept sentences corresponding to a preset target number of larger concept sentence matching degrees can be selected from large to small according to the concept sentence matching degrees as candidate concept labels corresponding to the preset content label.

[0142] A sentence matching threshold may also be preset, and concept sentences whose concept-sentence matching degree is greater than or equal to the sentence matching threshold are selected as candidate concept tags corresponding to the preset content tags.

[0143] In addition, other methods can be used to determine the candidate concept label corresponding to the preset content label from the concept statement based on the concept statement matching degree of the concept statement. The embodiment of the present application does not specifically limit the method of selecting the candidate concept label. The technical staff can flexibly adjust the selection method according to actual needs to obtain the concept statement that accurately reflects the superordinate concept of the preset content label from the concept statement as the candidate concept label.

[0144] Step 208: construct the concept tag library according to the correspondence between the preset content tags and the candidate concept tags.

[0145] After obtaining the candidate concept tag corresponding to at least one preset content tag, a concept tag library can be established in the memory, and the preset content tag and the candidate concept tag can be stored in the concept tag library according to the correspondence between the preset content tag and the candidate concept tag. It should be noted that there can be multiple preset content tags in the concept tag library, each preset content tag can correspond to multiple candidate concept tags, and different preset content tags can also correspond to the same candidate concept tag.

[0146] Step 209: Obtain a video content tag of the video to be marked, where the video content tag is used to represent the content of the video to be marked.

[0147] After the concept tag library is established, the videos that need to be tagged with concept tags can be used as videos to be tagged. Since the videos stored on the video platform are generally tagged with content tags, the corresponding video content tags of the videos to be tagged can be directly obtained. For videos to be tagged that are not tagged with content tags, the video content tags of the videos to be tagged can be determined through semantic analysis and other means. It should be noted that there can be multiple video content tags corresponding to each video to be tagged.

[0148] Step 209 may also include:

[0149] Sub-step 2091 , performing semantic analysis on the video to be marked and / or the associated text corresponding to the video to be marked, and determining a video content tag from a preset content tag library according to the semantic analysis result.

[0150] The video to be marked can represent the picture content of the video. By analyzing the picture content, key elements contained in the picture content can be determined, such as basketball, cars, film and television characters, etc.

[0151] The associated text corresponding to the video to be marked may include the video title, video introduction, video comment, subtitle text in the video, voice text determined by voice recognition of the audio in the video, etc. For example, for video audio, voice text can be obtained through voice recognition conversion. For subtitles in video images, subtitle text can be obtained from the video through image recognition technology. Image recognition technology may include optical character recognition (OCR) and other technologies.

[0152] After determining the key elements and / or associated texts, the associated elements and / or associated texts can be directly matched in the content tag library, and a preset content tag matching the associated elements and / or associated texts can be determined from the preset content tag library as the video content tag corresponding to the video to be marked. Semantic analysis can also be performed on the key elements and / or associated texts, and a preset content tag having a synonymous relationship with the associated elements and / or associated texts can be determined from the preset content tag library as the video content tag corresponding to the video to be marked.

[0153] Step 210: According to the video content tag, a candidate concept tag matching the video content tag is obtained from a concept tag library, wherein the concept tag library stores a correspondence between content tags and concept tags.

[0154] After determining the video content tag of the video to be tagged, a concept tag library is queried according to the video content tag of the video to be tagged, and a candidate concept tag corresponding to the video content tag of the video to be tagged is determined from the concept tag library.

[0155] Step 211: input the to-be-labeled video, the associated text corresponding to the to-be-labeled video, and the candidate concept tags into a target tag confidence model to determine the confidence of the candidate concept tags.

[0156] This step can be referred to as step 102, and will not be described in detail in this embodiment of the present application.

[0157] Step 212: determine a target concept tag from the candidate concept tags according to the confidence level, and tag the video to be tagged according to the target concept tag.

[0158] This step can be referred to as step 103, and will not be described in detail in this embodiment of the present application.

[0159] In summary, another video tagging method provided in the embodiment of the present application can determine the candidate concept tags corresponding to the video to be marked, and determine the multimodal features corresponding to the video to be marked based on the video to be marked and the associated text corresponding to the video to be marked, and then screen the candidate concept tags corresponding to the video to be marked through the multimodal features corresponding to the video to be marked, and obtain the target concept tags that can accurately represent the superordinate concept of the video to be marked, and mark the video to be marked with the target concept tags, which improves the richness of video tags, provides more reference basis for video operation platforms to manage videos, and provides users with richer means to find videos.

[0160] Figure 3 is a flowchart of a method for generating a target label confidence model provided in an embodiment of the present application, such as Figure 3 As shown, the method includes:

[0161] Step 301, obtaining a second correspondence between a sample video, a sample associated text of the sample video, a sample candidate concept label corresponding to the sample video, and label annotation information of each sample candidate concept label, wherein the sample associated text is a text content describing the sample video.

[0162] The sample video can represent the picture content of the video. By analyzing the picture content, key elements contained in the picture content can be determined, such as basketball, cars, film and television characters, etc.

[0163] The sample associated text corresponding to the sample video may include the title of the sample video, video introduction, video comments, subtitle text in the video, voice text determined by voice recognition of the audio in the sample video, etc. For example, for video audio, voice text can be obtained through voice recognition conversion. For subtitles in video images, subtitle text can be obtained from the video through image recognition technology. Image recognition technology can include technologies such as optical character recognition (OCR).

[0164] The sample candidate concept tag corresponding to the sample video can be determined from the concept tag library through the content tag of the sample video. The second corresponding relationship includes a sample video, a sample associated text of the sample video, and a sample candidate concept tag corresponding to the sample video.

[0165] Specifically, according to the number of corresponding candidate concept tags, videos with a large number of corresponding candidate concept tags can be obtained as sample videos, or sample videos can be determined from the video library by other means, such as manual selection, random selection, etc., which are not specifically limited in the embodiments of the present application.

[0166] The label annotation value represents whether the sample candidate concept label is a superordinate concept of the sample video or not. It can be determined whether the sample candidate concept label is a superordinate concept of the corresponding sample video by manual annotation. For example, if the sample candidate concept label A is a superordinate concept of the corresponding sample video C through manual verification, the label annotation value of the sample candidate concept label A can be set to 1. If the sample candidate concept label B is not a superordinate concept of the corresponding sample video C through manual verification, the label annotation value of the sample candidate concept label B can be set to 0.

[0167] A training set can be formed by obtaining sample videos, sample associated texts corresponding to the sample videos, sample candidate concept labels corresponding to the sample videos, and label annotation values ​​of each sample candidate concept label, thereby training the initial concept matching model.

[0168] Step 302, input the second correspondence as training data into the initial label confidence model, so that the initial label confidence model determines the video features of the sample video, the first text features of the sample associated text, and the second text features of the sample candidate concept label, and outputs the confidence of the sample candidate concept label based on the video features, the first text features, and the second text features.

[0169] The initial label confidence model needs to be trained before it can be used. During the training, the second correspondence including at least one sample video, the sample associated text corresponding to the sample video, and the sample candidate concept label corresponding to the sample video is input into the initial label confidence model as a training sample. The initial label confidence model can determine the video features of the sample video, the first text features of the sample associated text corresponding to the sample video, and the second text features of the sample candidate concept label corresponding to the sample video. Among them, the video features of the sample video refer to the features of the picture image content of the basic video. The first text feature can be the feature of the sample associated text content corresponding to the sample video, for example, it can be the feature of the sample video title, the feature of the sample video introduction, etc., or it can be a feature formed by the combination of multiple text contents such as the title and the introduction. The second text feature refers to the feature of each sample candidate concept label corresponding to the sample video, that is, for each sample candidate concept label corresponding to the sample video, a second text feature will be generated, and a sample video may correspond to multiple second text features.

[0170] Since the video feature is obtained from the picture content of the sample video and is used to represent the feature information in the picture content of the sample video, and the first text feature is obtained from the associated text content of the sample video and is used to represent the feature information in the associated text content of the sample video. Therefore, the video feature and the first text feature can be used to characterize the feature information of the sample video. The second text feature is the feature information of the text content of the sample candidate concept label.

[0171] The confidence of the sample candidate concept label can be further determined by using the video feature and the first text feature, two features related to the sample video, and the second text feature, a feature related to the sample candidate concept label. The confidence is used to characterize the probability that the sample candidate concept label is the superordinate concept of the corresponding sample video.

[0172] Optionally, step 302 may further include:

[0173] Sub-step 3021: input the sample video in the second corresponding relationship into the first sub-network, and obtain the video features of the sample video output by the first sub-network.

[0174] In order to improve the accuracy of the model, the initial label confidence model may include a first subnetwork and a second subnetwork, wherein the first subnetwork is used to process the video content of the sample video to generate the video features of the sample video, and the second subnetwork is used to process the text content to generate the first text features and the second text features. In this way, by generating features of different types of data through different types of subnetworks, the accuracy of the acquired features can be improved.

[0175] The first sub-network may include neural network models such as Deep residual network (ResNet), VGGNet, DenseNet (Dense Convolutional Network), and YoLo (You Only Look Once).

[0176] Optionally, sub-step 3021 may further include:

[0177] Sub-step A1: inputting a sample video frame of the sample video into the first sub-network, and obtaining a video frame feature of each video frame output by the first sub-network.

[0178] The sample video can be split into independent video frames, and all video frames in the sample video are input into the first sub-network as sample video frames corresponding to the sample video to determine the video frame features corresponding to each video frame.

[0179] Since the sample video contains a large number of video frames, in order to improve processing efficiency, some video frames can also be selected from the sample video, and the some video frames in the sample video are input into the first sub-network as sample video frames corresponding to the sample video.

[0180] Specifically, partial video frames in the sample video can be obtained according to a preset frame interval in the sample video, or a preset number of starting frames of the sample video at the starting position and / or a preset number of ending frames of the sample video at the ending position can be determined as partial video frames. Partial video frames can also be determined from the sample video in other ways. The embodiments of the present application do not particularly limit the specific method for determining partial video frames from the sample video.

[0181] Sub-step A2, summing and averaging the video frame features to obtain the video features.

[0182] After obtaining the video frame features corresponding to each sample video frame of the sample video, all the video frame features are summed up, and the summed result is divided by the number of video frame features to obtain the video features of the sample video.

[0183] It should be noted that the video features of the sample video may also be calculated using the video frame features in other ways, such as weighted averaging method and other methods.

[0184] Optionally, step 302 may further include:

[0185] Sub-step 3022: input the sample associated text and the sample candidate concept label in the second corresponding relationship into the second sub-network, and obtain the first text feature of the sample associated text and the second text feature of the sample candidate concept label output by the second sub-network.

[0186] The second sub-network may include network models such as BERT (Bidirectional Encoder Representations from Transformers), Deep (Bidirectional) Recurrent Neural Network (Deep (Bidirectional) Recurrent Neural Network), LSTM (Long Short-Term Memory), and Convolutional Neural Network (CNN).

[0187] The sample associated text can be input into the second sub-network as a whole. For example, if the sample associated text includes a title text and an introduction text B, the character string generated by combining the title text A and the introduction text B can be input into the second sub-network as a whole to obtain the first text feature corresponding to the sample associated text. Different types of text in the sample associated text can also be input into the second sub-network respectively to obtain the features of different types of text in the sample associated text respectively, and the features of different types of text are fused to obtain the first text feature corresponding to the sample associated text, wherein the fusion of different types of text features can include adding different types of text features and calculating the average feature, or fusing different types of text features in other ways, which is not specifically limited in the embodiments of the present application.

[0188] At the same time, each sample candidate concept corresponding to the sample video is input into the second sub-network respectively, and the second text feature corresponding to each sample candidate concept is obtained respectively.

[0189] Sub-step 3023, adding the video feature and the first text feature to obtain a multimodal feature, wherein the multimodal feature is used to characterize a fusion feature of the video feature and the associated text feature of the sample video.

[0190] Since the feature value is generally represented by a feature vector, the video feature can be a feature vector, and the first text feature can also be a feature vector. The video feature and the first text feature can be added to obtain a multimodal feature, which is used to characterize the fusion feature multimodal feature of the video feature of the sample video and the associated text feature corresponding to the sample video. Other calculation methods can also be used to fuse the video feature and the first text feature to obtain the multimodal feature corresponding to the sample video, which is not specifically limited in the embodiments of the present application.

[0191] Sub-step 3024: outputting the confidence of the sample candidate concept label according to the multimodal feature and the second text feature.

[0192] Optionally, sub-step 3024 may further include:

[0193] Sub-step B1, obtaining the dot product result of the multimodal feature and the second text feature.

[0194] The multimodal feature corresponding to the sample video can be a feature vector, and the second text feature of each sample candidate concept label corresponding to the sample video can also be a feature vector. Therefore, the multimodal feature corresponding to the sample video and each second text feature corresponding to the sample video can be dot-multiplied to obtain the dot-multiplication result of each second text feature corresponding to the sample video.

[0195] Sub-step B2, normalizing the dot product result to obtain and output the confidence of the sample candidate concept label.

[0196] Since the label annotation value corresponding to the sample candidate concept is used to characterize whether the sample candidate concept label is or is not the superordinate concept of the sample video, the label annotation value is generally 0 or 1, that is, the label annotation value generally ranges from 0 to 1. During the model training process, the candidate label confidence model needs to be trained through the label annotation value, specifically, the loss value between the confidence output by the candidate label confidence model and the label annotation value is determined, and then the parameters in the candidate label confidence model are adjusted according to the loss value. Therefore, it is necessary to make the loss value output by the model and the label annotation value in the same value range.

[0197] Specifically, when the label annotation value is 0 or 1, after obtaining the dot product result of each second text feature corresponding to the sample video, each dot product result can also be normalized to obtain the confidence of each sample candidate concept corresponding to the sample video, so as to facilitate the calculation of the loss value.

[0198] Step 303: determine a second loss value according to the confidence and the label annotation information corresponding to the sample video, and train the initial label confidence model according to the second loss value and a preset loss function to obtain a target label confidence model.

[0199] Through the above steps, the label annotation value and confidence corresponding to each sample candidate label of the sample video can be determined, and the loss value between the label annotation value and the confidence corresponding to the sample candidate label can be determined by a preset loss function, and the parameters in the first sub-network and / or the second sub-network in the candidate label confidence model are adjusted according to the loss value until the loss value converges, and the training of the candidate label confidence model is completed to obtain the target label confidence model. The target label confidence model is used to determine the multimodal features of the video to be labeled and the label features of the candidate concept label based on the video to be labeled and the associated text, and determine the confidence based on the multimodal features and the label features.

[0200] In summary, a method for generating a target label confidence model provided in an embodiment of the present application includes: inputting a sample video and a sample candidate concept label corresponding to the sample video as training data into an initial label confidence model, obtaining the confidence of the sample candidate concept label output by the initial label confidence model, the confidence being used to characterize the probability that the sample candidate concept label is the superordinate concept of the sample video; determining a second loss value according to the confidence and the label annotation information corresponding to the sample video, and training the initial label confidence model according to the second loss value and a preset loss function to obtain a target label confidence model. A target label confidence model for determining the confidence of a concept label can be obtained by training, so that technicians can screen the candidate concept labels corresponding to the video to be labeled according to the confidence, obtain the target concept label that can accurately represent the superordinate concept of the video to be labeled, and label the video to be labeled with the target concept label, thereby improving the richness of video labels, providing more reference basis for the video operation platform to manage videos, and providing users with richer means for finding videos.

[0201] Corresponding to the method provided by the video marking method embodiment of the present invention, see Figure 4 The present invention also provides a video marking device embodiment. In this embodiment, the device may include:

[0202] The candidate concept tag acquisition module 401 is configured to acquire a candidate concept tag corresponding to the video to be tagged, wherein the candidate concept tag is used to represent a candidate superordinate concept of the content of the video to be tagged;

[0203] The label confidence determination module 402 is configured to input the video to be labeled, the associated text corresponding to the video to be labeled, and the candidate concept label into a target label confidence model to determine the confidence of the candidate concept label, wherein the confidence is used to characterize the probability that the candidate concept label is a superordinate concept of the video to be labeled, and the target label confidence model is used to determine the multimodal features of the video to be labeled based on the video to be labeled and the associated text, determine the label features of the candidate concept label based on the associated text, and determine the confidence based on the multimodal features and the label features;

[0204] The target concept label determination module 403 is configured to determine a target concept label from the candidate concept labels according to the confidence level, and label the to-be-labeled video according to the target concept label.

[0205] In an optional embodiment, the device further comprises:

[0206] A content tag determination module is configured to obtain a video content tag of the video to be tagged, wherein the video content tag is used to represent the content of the video to be tagged;

[0207] The concept tag library module is configured to obtain a candidate concept tag matching the video content tag from a concept tag library according to the video content tag, wherein the concept tag library stores a correspondence between content tags and concept tags.

[0208] In an optional embodiment, the device further comprises:

[0209] A content tag acquisition module is configured to acquire a preset content tag from a preset content tag library;

[0210] A concept sentence determination module is configured to search a preset dictionary and / or network terms to determine a concept sentence associated with the preset content tag;

[0211] A sentence matching degree determination module is configured to input the preset content tag and the concept sentence into a target concept matching model to obtain the concept sentence matching degree output by the target concept matching model, wherein the concept sentence matching degree is used to characterize the probability that the concept sentence is a superordinate concept of the preset content tag;

[0212] A candidate concept tag determination module is configured to determine a candidate concept tag corresponding to the preset content tag from the concept sentence according to the matching degree of the concept sentence;

[0213] The concept tag library module is configured to construct the concept tag library according to the corresponding relationship between the preset content tags and the candidate concept tags.

[0214] In an optional implementation manner, the preset content tag determination module includes:

[0215] The content tag determination submodule is configured to perform semantic analysis on the video to be tagged and / or the associated text corresponding to the video to be tagged, and determine the video content tag from a preset content tag library according to the semantic analysis result.

[0216] In an optional embodiment, the device further comprises:

[0217] An information acquisition module is configured to acquire a sample content label, a sample concept sentence corresponding to the sample content label, and sentence annotation information of each sample concept sentence, wherein the sentence annotation information is used to indicate whether the sample concept sentence is a superordinate concept of the sample content label or not;

[0218] An information input module is configured to input the first correspondence between the sample content label and the sample concept sentence into an initial concept matching model to obtain a sample concept sentence matching degree output by the initial concept matching model, wherein the sample concept sentence matching degree is used to characterize the probability that the sample concept sentence is a superordinate concept of the sample content label;

[0219] The parameter adjustment module is configured to determine a first loss value based on the sample concept sentence matching degree and the sentence annotation information, and train the initial concept matching model based on the first loss value and a preset loss function to obtain the target concept matching model.

[0220] In summary, the embodiment of the present application provides a video labeling device, which can determine the candidate concept tags corresponding to the video to be labeled, and determine the multimodal features corresponding to the video to be labeled according to the video to be labeled and the associated text corresponding to the video to be labeled, and then screen the candidate concept tags corresponding to the video to be labeled through the multimodal features corresponding to the video to be labeled, and obtain the target concept tags that can accurately represent the superordinate concept of the video to be labeled, and label the video to be labeled with the target concept tags, thereby improving the richness of video tags, providing more reference basis for video operation platforms to manage videos, and providing users with richer means to find videos.

[0221] Corresponding to the method provided in the above-mentioned target label confidence model generation method embodiment of the present invention, see Figure 5 The present invention also provides an embodiment of a device for generating a target label confidence model. In this embodiment, the device may include:

[0222] The confidence determination module 501 is configured to input the sample video and the sample candidate concept label corresponding to the sample video as training data into the initial label confidence model, and obtain the confidence of the sample candidate concept label output by the initial label confidence model, wherein the confidence is used to characterize the probability that the sample candidate concept label is a superordinate concept of the sample video;

[0223] The model training module 502 is configured to determine a second loss value based on the confidence and the label annotation information corresponding to the sample video, and to train the initial label confidence model based on the second loss value and a preset loss function to obtain a target label confidence model, wherein the label annotation information is used to characterize whether the sample candidate concept label is or is not a superordinate concept of the sample video.

[0224] In an optional implementation, the confidence determination module includes:

[0225] An information acquisition submodule is configured to acquire a second correspondence between a sample video, a sample associated text of the sample video, a sample candidate concept label corresponding to the sample video, and label annotation information of each sample candidate concept label, wherein the sample associated text is a text content describing the sample video;

[0226] The information input submodule is configured to input the second corresponding relationship as training data into the initial label confidence model, so that the initial label confidence model determines the video features of the sample video, the first text features of the sample associated text, and the second text features of the sample candidate concept label, and outputs the confidence of the sample candidate concept label based on the video features, the first text features, and the second text features.

[0227] In an optional implementation, the information input submodule includes:

[0228] A first sub-network input sub-module is configured to input the sample video in the second corresponding relationship into the first sub-network, and obtain the video features of the sample video output by the first sub-network;

[0229] The second sub-network input sub-module is configured to input the sample associated text and the sample candidate concept label in the second corresponding relationship into the second sub-network, and obtain the first text feature of the sample associated text and the second text feature of the sample candidate concept label output by the second sub-network.

[0230] In an optional implementation, the first sub-network input submodule includes:

[0231] a video frame submodule, configured to input a sample video frame of the sample video into the first subnetwork, and obtain a video frame feature of each video frame output by the first subnetwork;

[0232] The video feature submodule is configured to sum and average the video frame features to obtain the video features.

[0233] In an optional implementation, the information input submodule includes:

[0234] A multimodal submodule, configured to add the video feature and the first text feature to obtain a multimodal feature, wherein the multimodal feature is used to represent a fusion feature of the video feature and the associated text feature of the sample video;

[0235] The confidence submodule is configured to output the confidence of the sample candidate concept label according to the multimodal feature and the second text feature.

[0236] In an optional implementation, the confidence submodule includes:

[0237] A dot product submodule, configured to obtain a dot product result of the multimodal feature and the second text feature;

[0238] The normalization submodule is configured to perform normalization processing on the dot product result to obtain and output the confidence of the sample candidate concept label.

[0239] In summary, the target label confidence model generation device provided by the embodiment of the present application inputs the sample video and the sample candidate concept label corresponding to the sample video as training data into the initial label confidence model, and obtains the confidence of the sample candidate concept label output by the initial label confidence model, and the confidence is used to characterize the probability that the sample candidate concept label is the superordinate concept of the sample video; the second loss value is determined according to the confidence and the label annotation information corresponding to the sample video, and the initial label confidence model is trained according to the second loss value and the preset loss function to obtain the target label confidence model. The target label confidence model for determining the confidence of the concept label can be trained, so that the technician can screen the candidate concept label corresponding to the video to be marked according to the confidence, obtain the target concept label that can accurately represent the superordinate concept of the video to be marked, and mark the video to be marked with the target concept label, which improves the richness of the video label, provides more reference basis for the video operation platform to manage videos, and also provides users with richer means to find videos.

[0240] Figure 6 is a block diagram of an electronic device 600 according to an exemplary embodiment. For example, the electronic device 600 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0241] Reference Figure 6 , the electronic device 600 may include one or more of the following components: a processing component 602 , a memory 604 , a power component 606 , a multimedia component 608 , an audio component 610 , an input / output (I / O) interface 612 , a sensor component 614 , and a communication component 616 .

[0242] The processing component 602 generally controls the overall operation of the electronic device 600, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 602 may include one or more processors 620 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 602 may include one or more modules to facilitate the interaction between the processing component 602 and other components. For example, the processing component 602 may include a multimedia module to facilitate the interaction between the multimedia component 608 and the processing component 602.

[0243] The memory 604 is used to store various types of data to support the operation of the electronic device 600. Examples of such data include instructions for any application or method operating on the electronic device 600, contact data, phone book data, messages, pictures, videos, etc. The memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0244] The power supply component 606 provides power to the various components of the electronic device 600. The power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 600.

[0245] The multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 608 includes a front camera and / or a rear camera. When the electronic device 600 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0246] The audio component 610 is used to output and / or input audio signals. For example, the audio component 610 includes a microphone (MIC), and when the electronic device 600 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is used to receive an external audio signal. The received audio signal can be further stored in the memory 604 or sent via the communication component 616. In some embodiments, the audio component 610 also includes a speaker for outputting audio signals.

[0247] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0248] The sensor assembly 614 includes one or more sensors for providing various aspects of status assessment for the electronic device 600. For example, the sensor assembly 614 can detect the open / closed state of the electronic device 600, the relative positioning of the components, such as the display and keypad of the electronic device 600, and the sensor assembly 614 can also detect the position change of the electronic device 600 or a component of the electronic device 600, the presence or absence of contact between the user and the electronic device 600, the orientation or acceleration / deceleration of the electronic device 600, and the temperature change of the electronic device 600. The sensor assembly 614 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 614 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 614 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0249] The communication component 616 is used to facilitate wired or wireless communication between the electronic device 600 and other devices. The electronic device 600 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 616 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 616 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0250] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to implement a video marking method provided in an embodiment of the present application.

[0251] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, and the instructions can be executed by a processor 620 of an electronic device 600 to perform the above method. For example, the non-transitory storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0252] Figure 7 is a block diagram of an electronic device 700 according to an exemplary embodiment. For example, the electronic device 700 may be provided as a server. Figure 7 The electronic device 700 includes a processing component 722, which further includes one or more processors, and a memory resource represented by a memory 732 for storing instructions executable by the processing component 722, such as an application. The application stored in the memory 732 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 722 is configured to execute instructions to perform a video marking method provided in an embodiment of the present application.

[0253] The electronic device 700 may also include a power supply component 726 configured to perform power management of the electronic device 700, a wired or wireless network interface 750 configured to connect the electronic device 700 to a network, and an input / output (I / O) interface 758. The electronic device 700 may operate based on an operating system stored in the memory 732, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0254] The embodiment of the present application also provides a computer program product, including a computer program, and when the computer program is executed by a processor, the video marking method is implemented.

[0255] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0256] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A video marking method, characterized in that: The method comprises: Obtaining a candidate concept tag corresponding to the video to be tagged, wherein the candidate concept tag is used to represent a candidate superordinate concept of the content of the video to be tagged, and the candidate concept tag includes a text tag; The video to be marked, the associated text corresponding to the video to be marked, and the candidate concept label are input into a target label confidence model to determine the confidence of the candidate concept label, wherein the confidence is used to characterize the probability that the candidate concept label is a superordinate concept of the content of the video to be marked, and the target label confidence model is used to determine the multimodal features of the video to be marked according to the video to be marked and the associated text, determine the label features of the candidate concept label according to the associated text, and determine the confidence according to the multimodal features and the label features, wherein the associated text represents text information associated with the video to be marked; Determining a target concept tag from the candidate concept tags according to the confidence level, and marking the to-be-marked video according to the target concept tag; The target label confidence model is used to determine the video features of the video to be labeled, the first text features of the associated text, and the second text features of the candidate concept label, and add the video features and the first text features to obtain the multimodal features, and output the confidence of the candidate concept label based on the multimodal features and the second text features.

2. The method according to claim 1, characterized in that The step of obtaining candidate concept tags corresponding to the video to be labeled includes: Obtaining a video content tag of the video to be marked, wherein the video content tag is used to represent the content of the video to be marked; According to the video content tag, a candidate concept tag matching the video content tag is obtained from a concept tag library, wherein the concept tag library stores a correspondence between content tags and concept tags.

3. The method according to claim 2, characterized in that The method further comprises: Get preset content tags from the preset content tag library; Searching a preset dictionary and / or web terms to determine a concept sentence associated with the preset content tag; Inputting the preset content tag and the concept sentence into a target concept matching model to obtain a concept sentence matching degree output by the target concept matching model, wherein the concept sentence matching degree is used to represent the probability that the concept sentence is a superordinate concept of the preset content tag; Determining a candidate concept tag corresponding to the preset content tag from the concept tag according to the concept sentence matching degree; The concept tag library is constructed according to the correspondence between the preset content tags and the candidate concept tags.

4. The method according to claim 2, characterized in that: The step of obtaining the video content label of the video to be labeled includes: A semantic analysis is performed on the video to be marked and / or the associated text corresponding to the video to be marked, and a video content tag is determined from a preset content tag library according to the semantic analysis result.

5. The method according to claim 3, characterized in that: The method further comprises: Acquire a sample content label, a sample concept sentence corresponding to the sample content label, and sentence annotation information of each of the sample concept sentences, wherein the sentence annotation information is used to indicate whether the sample concept sentence is a superordinate concept of the sample content label or not; Inputting the first correspondence between the sample content label and the sample concept sentence into the initial concept matching model to obtain the sample concept sentence matching degree output by the initial concept matching model, wherein the sample concept sentence matching degree is used to characterize the probability that the sample concept sentence is a superordinate concept of the sample content label; A first loss value is determined according to the sample concept sentence matching degree and the sentence annotation information, and the initial concept matching model is trained according to the first loss value and a preset loss function to obtain the target concept matching model.

6. A method for generating a target label confidence model, characterized in that: The method comprises: Inputting a sample video and a sample candidate concept label corresponding to the sample video as training data into an initial label confidence model, obtaining the confidence of the sample candidate concept label output by the initial label confidence model, wherein the confidence is used to characterize the probability that the sample candidate concept label is a superordinate concept of the sample video; A second loss value is determined based on the confidence and the label annotation information corresponding to the sample video, and the initial label confidence model is trained based on the second loss value and a preset loss function to obtain a target label confidence model as described in any one of claims 1 to 5, wherein the label annotation information is used to characterize whether the sample candidate concept label is or is not a superordinate concept of the sample video.

7. The method according to claim 6, characterized in that The step of inputting the sample video and the sample candidate concept label corresponding to the sample video as training data into the initial label confidence model to obtain the confidence of the sample candidate concept label output by the initial label confidence model comprises: Acquire a second correspondence between a sample video, a sample associated text of the sample video, a sample candidate concept label corresponding to the sample video, and label annotation information of each sample candidate concept label, wherein the sample associated text is a text content describing the sample video; The second corresponding relationship is input as training data into the initial label confidence model so that the initial label confidence model determines the video features of the sample video, the first text features of the sample associated text, and the second text features of the sample candidate concept label, and outputs the confidence of the sample candidate concept label based on the video features, the first text features, and the second text features.

8. The method according to claim 7, characterized in that The initial label confidence model includes a first subnetwork and a second subnetwork, and the second corresponding relationship is input into the initial label confidence model as training data so that the initial label confidence model determines the video feature of the sample video, the first text feature of the sample associated text, and the second text feature of the sample candidate concept label, including: Inputting the sample video in the second corresponding relationship into the first sub-network, and obtaining the video features of the sample video output by the first sub-network; The sample associated text and the sample candidate concept label in the second corresponding relationship are input into the second sub-network, and the first text feature of the sample associated text and the second text feature of the sample candidate concept label output by the second sub-network are obtained.

9. The method according to claim 8, characterized in that The step of inputting the sample video in the second corresponding relationship into the first sub-network and obtaining the video features of the sample video output by the first sub-network includes: Inputting a sample video frame of the sample video into the first sub-network, and obtaining a video frame feature of each video frame output by the first sub-network; The video frame features are summed and averaged to obtain the video features.

10. The method according to claim 7, characterized in that Outputting the confidence of the sample candidate concept label according to the video feature, the first text feature, and the second text feature includes: Adding the video feature and the first text feature to obtain a multimodal feature, wherein the multimodal feature is used to represent a fusion feature of the video feature and the associated text feature of the sample video; Outputting the confidence of the sample candidate concept label according to the multimodal feature and the second text feature.

11. The method according to claim 10, characterized in that Outputting the confidence of the sample candidate concept label according to the multimodal feature and the second text feature includes: Obtaining a dot product result of the multimodal feature and the second text feature; The dot product result is normalized to obtain and output the confidence of the sample candidate concept label.

12. A video marking device, characterized in that: The device comprises: A candidate concept label acquisition module is configured to acquire a candidate concept label corresponding to the video to be labeled, wherein the candidate concept label is used to represent a candidate superordinate concept of the content of the video to be labeled, and the candidate concept label includes a text label; A label confidence determination module is configured to input the video to be labeled, the associated text corresponding to the video to be labeled, and the candidate concept label into a target label confidence model to determine the confidence of the candidate concept label, wherein the confidence is used to characterize the probability that the candidate concept label is a superordinate concept of the content of the video to be labeled, and the target label confidence model is used to determine the multimodal features of the video to be labeled based on the video to be labeled and the associated text, determine the label features of the candidate concept label based on the associated text, and determine the confidence based on the multimodal features and the label features, wherein the associated text represents text information associated with the video to be labeled; a target concept label determination module, configured to determine a target concept label from the candidate concept labels according to the confidence level, and label the to-be-labeled video according to the target concept label; The target label confidence model is used to determine the video features of the video to be labeled, the first text features of the associated text, and the second text features of the candidate concept label, and add the video features and the first text features to obtain the multimodal features, and output the confidence of the candidate concept label based on the multimodal features and the second text features.

13. The device according to claim 12, characterized in that The device also includes: A content tag determination module is configured to obtain a video content tag of the video to be tagged, wherein the video content tag is used to represent the content of the video to be tagged; The concept tag library module is configured to obtain a candidate concept tag matching the video content tag from a concept tag library according to the video content tag, wherein the concept tag library stores a correspondence between content tags and concept tags.

14. The device according to claim 13, characterized in that The device also includes: A content tag acquisition module is configured to acquire a preset content tag from a preset content tag library; A concept sentence determination module is configured to search a preset dictionary and / or network terms to determine a concept sentence associated with the preset content tag; A sentence matching degree determination module is configured to input the preset content tag and the concept sentence into a target concept matching model to obtain the concept sentence matching degree output by the target concept matching model, wherein the concept sentence matching degree is used to characterize the probability that the concept sentence is a superordinate concept of the preset content tag; A candidate concept tag determination module is configured to determine a candidate concept tag corresponding to the preset content tag from the concept sentence according to the matching degree of the concept sentence; The concept tag library module is configured to construct the concept tag library according to the corresponding relationship between the preset content tags and the candidate concept tags.

15. The device according to claim 13, characterized in that The content tag determination module includes: The content tag determination submodule is configured to perform semantic analysis on the video to be tagged and / or the associated text corresponding to the video to be tagged, and determine the video content tag from a preset content tag library according to the semantic analysis result.

16. The device according to claim 14, characterized in that The device also includes: An information acquisition module is configured to acquire a sample content label, a sample concept sentence corresponding to the sample content label, and sentence annotation information of each sample concept sentence, wherein the sentence annotation information is used to indicate whether the sample concept sentence is a superordinate concept of the sample content label or not; An information input module is configured to input the first correspondence between the sample content label and the sample concept sentence into an initial concept matching model to obtain a sample concept sentence matching degree output by the initial concept matching model, wherein the sample concept sentence matching degree is used to characterize the probability that the sample concept sentence is a superordinate concept of the sample content label; The parameter adjustment module is configured to determine a first loss value based on the sample concept sentence matching degree and the sentence annotation information, and train the initial concept matching model based on the first loss value and a preset loss function to obtain the target concept matching model.

17. A target label confidence model generation device, characterized in that: The device comprises: A confidence determination module is configured to input a sample video and a sample candidate concept label corresponding to the sample video as training data into an initial label confidence model, and obtain the confidence of the sample candidate concept label output by the initial label confidence model, wherein the confidence is used to characterize the probability that the sample candidate concept label is a superordinate concept of the sample video; The model training module is configured to determine a second loss value based on the confidence and the label annotation information corresponding to the sample video, and train the initial label confidence model based on the second loss value and a preset loss function to obtain a target label confidence model as described in any one of claims 1 to 5, wherein the label annotation information is used to characterize whether the sample candidate concept label is or is not a superordinate concept of the sample video.

18. The device according to claim 17, characterized in that The confidence determination module comprises: An information acquisition submodule is configured to acquire a second correspondence between a sample video, a sample associated text of the sample video, a sample candidate concept label corresponding to the sample video, and label annotation information of each sample candidate concept label, wherein the sample associated text is a text content describing the sample video; The information input submodule is configured to input the second corresponding relationship as training data into the initial label confidence model, so that the initial label confidence model determines the video features of the sample video, the first text features of the sample associated text, and the second text features of the sample candidate concept label, and outputs the confidence of the sample candidate concept label based on the video features, the first text features, and the second text features.

19. The device according to claim 18, characterized in that The information input submodule includes: A first sub-network input sub-module is configured to input the sample video in the second corresponding relationship into the first sub-network, and obtain the video features of the sample video output by the first sub-network; The second sub-network input sub-module is configured to input the sample associated text and the sample candidate concept label in the second corresponding relationship into the second sub-network, and obtain the first text feature of the sample associated text and the second text feature of the sample candidate concept label output by the second sub-network.

20. The device according to claim 19, characterized in that The first sub-network input sub-module comprises: a video frame submodule, configured to input a sample video frame of the sample video into the first subnetwork, and obtain a video frame feature of each video frame output by the first subnetwork; The video feature submodule is configured to sum and average the video frame features to obtain the video features.

21. The device according to claim 18, characterized in that The information input submodule includes: A multimodal submodule, configured to add the video feature and the first text feature to obtain a multimodal feature, wherein the multimodal feature is used to represent a fusion feature of the video feature and the associated text feature of the sample video; The confidence submodule is configured to output the confidence of the sample candidate concept label according to the multimodal feature and the second text feature.

22. The device according to claim 21, characterized in that The confidence submodule includes: A dot product submodule, configured to obtain a dot product result of the multimodal feature and the second text feature; The normalization submodule is configured to perform normalization processing on the dot product result to obtain and output the confidence of the sample candidate concept label.

23. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 11.

24. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 11.

25. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images