Video tag generation method and device, storage medium and electronic device

Abstract

The present invention discloses a method and device for generating a video tag, a storage medium and an electronic device. The method comprises: obtaining a target video to be identified, wherein the target video carries a video description text for describing the target video; using the video description text to construct a target candidate character library of the target video; after extracting the video features of the target video and the description features of the video description text, performing N iterative calculations on the video features and the description features to obtain M candidate character sequences, wherein the candidate character objects in each candidate character sequence include each candidate character in the target candidate character library and the confidence level of the candidate character; determining the target character from each candidate character sequence based on the confidence level to obtain M target characters; and splicing the M target characters into a target video tag that matches the target video. The present invention solves the technical problem of low accuracy of video tags determined by existing methods.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a method and device for generating a video tag, a storage medium and an electronic device. Background Art

[0002] With the development of online video platforms, Internet users are increasingly accustomed to obtaining the latest online information by browsing videos. In the process of disseminating online videos on the Internet, video platforms usually match video tags for videos, and then implement video recommendations and video search functions based on video tags. For online videos, fresh and highly matched video tags can enable Internet users to discover new hot events at the first time, thereby increasing the click-through rate of new hot online videos.

[0003] The existing methods for matching labels for online videos usually perform multiple classification and clustering operations on the extracted video features during the matching process, and then determine the labels that match the video in a defined label system.

[0004] It can be seen that the existing tag determination method can only select tags that match the video from the pre-set tag system. For videos with fresh events, since the preset tag library does not include fresh tag entries corresponding to fresh events, it is impossible to match fresh tags for fresh videos. As a result, the platform cannot provide accurate recommendations or accurate search results in the process of providing video recommendations and video search functions based on tags. In other words, the existing method for determining video tags has a technical problem of low accuracy of determined tags.

[0005] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0006] The embodiments of the present invention provide a method and device for generating a video tag, a storage medium and an electronic device, so as to at least solve the technical problem that the accuracy of video tags determined by the existing methods is low.

[0007] According to one aspect of an embodiment of the present invention, a method for generating a video tag is provided, comprising: obtaining a target video to be identified, wherein the target video carries a video description text for describing the target video; constructing a target candidate character library of the target video using the video description text, wherein the target candidate character library comprises a reference character set and a description character set, wherein the reference character set comprises reference characters obtained by segmenting a plurality of historical tag texts, and the description character set comprises description characters obtained by segmenting the video description text; when video features of the target video and description features of the video description text are extracted, performing N iterative calculations on the video features and the description features to obtain M candidate character sequences, wherein the candidate character objects in each of the candidate character sequences comprise each candidate character in the target candidate character library and a confidence level matching the candidate character, wherein the confidence level is used to indicate a matching level between the candidate character and the target video, and N and M are natural numbers greater than or equal to 1; determining a target character from each of the candidate character sequences based on the confidence level to obtain M target characters; and splicing the M target characters into a target video tag matching the target video.

[0008] According to another aspect of an embodiment of the present invention, a device for generating a video tag is also provided, comprising: an acquisition unit, for acquiring a target video to be identified, wherein the target video carries a video description text for describing the target video; a construction unit, for constructing a target candidate character library of the target video using the video description text, wherein the target candidate character library comprises a reference character set and a description character set, wherein the reference character set comprises reference characters obtained by segmenting a plurality of historical tag texts, and the description character set comprises description characters obtained by segmenting the video description text; a calculation unit, for extracting the video features of the target video and the video description text; In the case of the description features of the video description text, the video features and the description features are iterated N times to obtain M candidate character sequences, wherein the candidate character objects in each of the candidate character sequences include each candidate character in the target candidate character library and the confidence of the candidate character matching, the confidence is used to indicate the matching degree between the candidate character and the target video, N, M are natural numbers greater than or equal to 1; a determination unit is used to determine the target character from each of the candidate character sequences based on the confidence to obtain the M target characters; a splicing unit is used to splice the M target characters into a target video label matching the target video.

[0009] According to another aspect of the embodiments of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned method for generating a video tag when running.

[0010] According to another aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above method for generating a video tag.

[0011] According to another aspect of an embodiment of the present invention, there is further provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the method for generating a video tag through the computer program.

[0012] In an embodiment of the present invention, a target video to be identified is obtained; a target candidate character library of the target video is constructed using a video description text, wherein the target candidate character library includes a reference character set and a description character set, the reference character set includes reference characters obtained by segmenting multiple historical tag texts, and the description character set includes description characters obtained by segmenting the video description text; after extracting the video features of the target video and the description features of the video description text, the video features and the description features are iteratively calculated N times to obtain M candidate character sequences; the target characters are determined from each candidate character sequence based on the confidence level to obtain M target characters; and the M target characters are then spliced ​​into a target video label that matches the target video. That is, in an embodiment of the present application, the video features and the description text features of the target video are iteratively calculated, and then the target characters are output word by word from the description character set composed of the reference character set and the video description text, and then the target video label is formed by the output target characters. Since the target candidate character library for outputting characters in the embodiment of the present application is combined with a description character set, and the description character set includes fresh text content related to the video, fresh characters can be output in combination with the description character set, and then fresh labels can be obtained based on the output fresh characters, thereby improving the accuracy of the output video labels and solving the technical problem of low accuracy of video labels obtained by existing video label determination methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0014] Figure 1 is a schematic diagram of a hardware environment of an optional method for generating video tags according to an embodiment of the present invention;

[0015] Figure 2 is a flow chart of an optional method for generating video tags according to an embodiment of the present invention;

[0016] Figure 3 is a schematic diagram of an optional method for generating video tags according to an embodiment of the present invention;

[0017] Figure 4 is a schematic diagram of another optional method for generating video tags according to an embodiment of the present invention;

[0018] Figure 5 is a schematic diagram of an optional method for generating video tags according to an embodiment of the present invention;

[0019] Figure 6 is a schematic diagram of another optional method for generating video tags according to an embodiment of the present invention;

[0020] Figure 7 is a flowchart of another optional method for generating video tags according to an embodiment of the present invention;

[0021] Figure 8 is a structural schematic diagram of an optional device for generating video tags according to an embodiment of the present invention;

[0022] Fig. 9 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] According to one aspect of an embodiment of the present invention, a method for generating a video tag is provided. As an optional implementation, the method for generating a video tag can be applied to, but is not limited to, Figure 1 In the video tag generation system composed of the server 102 and the terminal device 104 shown in FIG. Figure 1As shown, the server 102 is connected to the terminal device 104 through the network 110. The above network may include but is not limited to: wired network, wireless network, wherein the wired network includes: local area network, metropolitan area network and wide area network, and the wireless network includes: Bluetooth, WIFI and other networks that realize wireless communication. The above terminal device may include but is not limited to at least one of the following: mobile phone (such as Android mobile phone, iOS mobile phone, etc.), laptop computer, tablet computer, PDA, MID (Mobile Internet Devices, mobile Internet device), PAD, desktop computer, smart TV, car-mounted device, etc. A client may be installed on the above terminal device, such as a video sharing client, etc. A display, a processor and a memory are also provided on the above terminal device. The display can be used to display the program interface of the video uploading application program and display the video content of the uploading server. The processor can be used to pre-process the video file to be uploaded before transmission, for example, compress the acquired video file; the memory is used to store the video to be uploaded. It is understandable that after the target video to be uploaded is acquired in the terminal device 104, the terminal device 104 can send the target video to the server 102 through the network 110. When the server 102 receives the target video, it generates a video tag prompt tag matching the target video according to the video uploaded by the terminal device 104. The terminal device 104 can receive the video tag returned by the server 102 through the network 110. The server 102 can be a single server, or a server cluster composed of multiple servers, or a cloud server. The server includes a database and a processing engine. Among them, the database may include a historical tag vocabulary for matching tags for videos and a pre-trained tag generation model; the processing engine is used to generate a target video tag based on the acquired target video.

[0026] According to one aspect of an embodiment of the present invention, the above-mentioned video tag generation system can also perform the following steps: the terminal device 104 executes step S102 to obtain the target video to be identified; then executes step S104, the terminal device 104 sends the target video to the server 102 through the network 110; the server 102 executes steps S106 to S114 to obtain the target video to be identified and use the video description text to build a target candidate character library for the target video, wherein the target candidate character library includes a reference character set and a description character set, the reference character set includes reference characters obtained by segmenting multiple historical tag texts, and the description character set includes description characters obtained by segmenting the video description text; after extracting the target video In the case of the video features of the video and the description features of the video description text, the video features and the description features are iterated and calculated N times to obtain M candidate character sequences, wherein the candidate character objects in each candidate character sequence include each candidate character in the target candidate character library and the confidence matching with the candidate character, the confidence is used to indicate the matching degree between the candidate character and the target video, and N and M are natural numbers greater than or equal to 1; based on the confidence, the target character is determined from each candidate character sequence to obtain M target characters; the M target characters are spliced ​​into a target video label matching the target video; then step S116 is executed, and the server 102 sends the target video label to the terminal device 104 through the network 110. It can be understood that in the case where the terminal device 104 is a device with sufficient computing processing capabilities, the above steps S106 to S114 can also be performed in the terminal device 104.

[0027] In an embodiment of the present invention, a target video to be identified is obtained; a target candidate character library of the target video is constructed using a video description text, wherein the target candidate character library includes a reference character set and a description character set, the reference character set includes reference characters obtained by segmenting multiple historical tag texts, and the description character set includes description characters obtained by segmenting the video description text; after extracting the video features of the target video and the description features of the video description text, the video features and the description features are iteratively calculated N times to obtain M candidate character sequences; the target characters are determined from each candidate character sequence based on the confidence level to obtain M target characters; and the M target characters are then spliced ​​into a target video label that matches the target video. That is, in an embodiment of the present application, the video features and the description text features of the target video are iteratively calculated, and then the target characters are output word by word from the description character set composed of the reference character set and the video description text, and then the target video label is formed by the output target characters. Since the target candidate character library for outputting characters in the embodiment of the present application is combined with a description character set, and the description character set includes fresh text content related to the video, fresh characters can be output in combination with the description character set, and then fresh labels can be obtained based on the output fresh characters, thereby improving the accuracy of the output video labels and solving the technical problem of low accuracy of video labels obtained by existing video label determination methods.

[0028] The above is only an example and is not limited in this embodiment.

[0029] As an optional implementation, Figure 2 As shown, the above-mentioned method for generating video tags includes the following steps:

[0030] S202, obtaining a target video to be identified, wherein the target video carries a video description text for describing the target video;

[0031] It should be noted that the above video description information may include, but is not limited to, one or more text information used to describe the content of the target video, such as the video title information of the target video, the subtitle information of the target video, the tag information of the target video, the text information appearing in the screen of the target video, etc. The specific text information is not limited here.

[0032] S204, constructing a target candidate character library for the target video using the video description text;

[0033] The target candidate character library includes a reference character set and a description character set. The reference character set includes reference characters obtained by segmenting multiple historical tag texts, and the description character set includes description characters obtained by segmenting video description texts.

[0034] The above method will be illustrated with a specific example. Suppose that in the operation of generating historical tags, it has been determined that the following tags can be used to classify the video: "Game", "Love", "TV drama", and based on the historically generated tags, the reference character set can be determined as "You, Xi, Ai, Qing, Dian, Shi, Ju";

[0035] Suppose the description text of the target video to be recognized obtained is a subtitle in the target video "The game of City Heroes is really fun. Come and play quickly, everyone!", and based on the above description text, the description character set can be determined as "Cheng, Shi, Ying, Xiong, You, Xi, Zhen, Hao, Wan, Da, Jia, Kuai, Lai, Wan, Ba".

[0036] Combining the above reference character set and the above description character set, the target candidate character library "You, Xi, Ai, Qing, Dian, Shi, Ju, Cheng, Shi, Ying, Xiong, You, Xi, Zhen, Hao, Wan, Da, Jia, Kuai, Lai, Wan, Ba" can be obtained.

[0037] The above character library includes the characters "You, Xi" in the description character set and the characters "You, Xi" in the reference character set. In an optional way, the duplicate characters in the above character library can also be de-duplicated to obtain the updated target candidate character library "You, Xi, Ai, Qing, Dian, Shi, Ju, Cheng, Shi, Ying, Xiong, Zhen, Hao, Wan, Da, Jia, Kuai, Lai, Wan, Ba".

[0038] It can be understood that the above method for constructing the target candidate character library is only an example. In the specific implementation, the above method for determining the target candidate character library can also be other similar methods, and the specific method for constructing the above target candidate character library is not limited here.

[0039] S206, in the case of extracting the video features of the target video and the description features of the video description text, perform N iterative calculations on the video features and the description features to obtain M candidate character sequences, where each candidate character object in each candidate character sequence includes each candidate character in the target candidate character library and the confidence level matching the candidate character;

[0040] It can be understood that the above confidence level is used to indicate the matching degree between the candidate character and the target video, and N and M are natural numbers greater than or equal to 1.

[0041] It can be understood that the above video features can be visual features for characterizing the style of the video frame, or video content features for characterizing the content of the video frame. Here, the specific content characterized by the above video features is not limited; the above-described description features are text features corresponding to the video description text, which can be title text features, subtitle text features, or picture text features included in the video frame. Here, the specific content of the above video description text features is also not limited.

[0042] It can be understood that in the above iterative calculation process, the above video features and description text features can be converted into corresponding vectors, which is convenient for iterative calculation. At the same time, the above iterative calculation indicates that the input value of the current iterative calculation is related to the output value of the previous iterative calculation. That is to say, in the above embodiments of the present application, there is a correlation between the subsequent results output by the iterative calculation and the previous results of the iterative calculation.

[0043] S208. Determine target characters from each candidate character sequence based on the confidence level to obtain M target characters;

[0044] The above candidate character sequences are described below. It can be understood that the above candidate character sequences are composed of the confidence levels matched by each character in the target candidate character library. Continuing with the above candidate character library "You, Xi, Ai, Qing, Dian, Shi, Ju, Cheng, Shi, Ying, Xiong, Zhen, Hao, Wan, Da, Jia, Kuai, Lai, Wan, Ba" as an example for illustration.

[0045] Assume that in the case of the first iterative calculation of the video features and description features of the above video, the obtained first candidate character sequence is: "You (98%), Xi (90%), Ai (10%), Qing (10%), Dian (5%), Shi (8%), Ju (8%), Cheng (58%), Shi (60%), Ying (88%), Xiong (70%), Zhen (60%), Hao (70%), Wan (60%), Da (28%), Jia (5%), Kuai (18%), Lai (8%), Wan (68%), Ba (1%)";

[0046] Assume that the rule for outputting characters is to output the character with the highest confidence level. Then, the first character output for the current sequence can be determined as "You" according to the above candidate sequence;

[0047] Suppose the video features and description features of the above video are used for a second iteration calculation in combination with the results of the previous iteration. Suppose the second candidate character sequence obtained is: "You (70%), Xi (98%), Ai (10%), Qing (10%), Dian (5%), Shi (8%), Ju (8%), Cheng (58%), Shi (60%), Ying (88%), Xiong (70%), Zhen (60%), Hao (70%), Wan (60%), Da (28%), Jia (5%), Kuai (18%), Lai (8%), Wan (68%), Ba (1)".

[0048] Suppose the rule for outputting characters is to output the character with the highest confidence. Thus, the second character output for the current sequence can be determined as "Xi" based on the above candidate sequence.

[0049] After multiple iterations according to the above method, suppose the finally output M characters are "You", "Xi", "Cheng", "Shi", "Ying", "Xiong", "Hao", "You", "Xi".

[0050] S210, Concatenate the M target characters into a target video label that matches the target video.

[0051] Continuing to illustrate this step with the above embodiment, when the obtained M target characters are "You", "Xi", "Cheng", "Shi", "Ying", "Xiong", "Hao", "You", "Xi", the above characters can be concatenated in the order of output to obtain three labels: "Youxi", "Chengshi Yingxiong", "Haoyouxi".

[0052] In an alternative way, during the process of outputting the above characters, an output identifier can also be used to indicate the way of concatenating the target video label. For example, when the obtained M target characters are "You", "Xi", "CLS", "Cheng", "Shi", "Ying", "Xiong", "CLS", "Hao", "You", "Xi", the "Youxi" concatenated from the characters "You" and "Xi" can be used as the first target label according to the interval identifier "CLS" between "You", "Xi" and "Cheng", "Shi", "Ying", "Xiong", the "Chengshi Yingxiong" concatenated from "Cheng", "Shi", "Ying", "Xiong" can be used as the second target label, and the "Haoyouxi" concatenated from "Hao", "You", "Xi" after the interval identifier "CLS" can be used as the third target label.

[0053] In another alternative way, when the obtained M target characters are the disordered characters "游 (You)", "游 (You)", "戏 (Xi)", "城 (Cheng)", "英 (Ying)", "市 (Shi)", "雄 (Xiong)", "好 (Hao)", "戏 (Xi)", according to semantic analysis, it can be determined that "游 (You)" and "戏 (Xi)" are concatenated into the first label "游戏 (Game)", "游 (You)", "好 (Hao)", and "戏 (Xi)" are concatenated into the second label "好游戏 (Good Game)", and "城 (Cheng)", "英 (Ying)", "市 (Shi)", and "雄 (Xiong)" are concatenated into the third label "城市英雄 (City Heroes)". It can be seen that when the promotion video of a new game "City Heroes" is launched, through the above implementation manners of the present application, even if the original historical label does not contain "城市英雄 (City Heroes)", based on the fresh characters provided in the description character library, through iterative calculation operations, the label "城市英雄 (City Heroes)" can be obtained, thereby achieving the technical effect of automatically generating fresh labels and improving the efficiency and accuracy of label generation. At the same time, since in the above implementation manners of the present application, labels are generated based on the word-by-word output method, the label "好游戏 (Good Game)" can be generated even if the description text and the original historical label library do not include the label "好游戏 (Good Game)". As a result, when a user searches for "好游戏 (Good Game)", the above video can also be retrieved, thereby increasing the click-through rate and viewing popularity of the above video.

[0054] The following Figure 3 illustrates the above method in conjunction with another specific implementation manner.

[0055] As Figure 3 shown, a title image of a video is shown. The title image includes the text content "今天你馋了吗 (Are you craving something today)", and at the same time, the title of the video is "追剧必备的小零食来一波,快@他做给你吃吧 (Here comes a wave of snacks essential for TV drama watching. Hurry up and @ him to make them for you to eat)", and the video includes the subtitle content "这个教程非常简单 (This tutorial is very simple)".

[0056] As Figure 3 shown in the video image, the above relevant text can be used as the video description text information of the video, that is, it includes the text content "今天你馋了吗 (Are you craving something today)" in the video image, as well as the text content of the video title "追剧必备的小零食来一波,快@他做给你吃吧 (Here comes a wave of snacks essential for TV drama watching. Hurry up and @ him to make them for you to eat)" and the subtitle content "这个教程非常简单 (This tutorial is very simple)".

[0057] Next, extract the video features of the video. The video features can be the image features of the video cover image as Figure 3 shown; the video category feature "美食 (Food)" of the video can also be extracted as another video feature; extract the description features of the video description text information, that is, extract the corresponding feature information of the above video description text information. As a specific form, the above features can all be represented by vectors or matrices.

[0058] Furthermore, according to the visual features, category features and description features extracted above, the confidence corresponding to each character in the candidate character library determined by the reference character library and the description character library is determined. Assuming that the historical label text preset for generating reference characters does not include the label "food tutorial", in the traditional label extraction method, only labels can be determined from the preset historical label library, and thus the label "food tutorial" which has a high relevance to the video cannot be obtained, that is, there is a technical problem of low accuracy of the extracted labels.

[0059] In this embodiment, the current candidate character library includes the reference character library obtained by the historical tag segmentation, and the current video description text character library, that is, the text content in the above video image "Are you greedy today?", the text content of the video title "Here are some snacks that are essential for watching TV series, @ him to make them for you" and the subtitle content "This tutorial is very simple" and other text character content, and the confidence corresponding to each tag text of the candidate tag library composed of the historical tag library and the above text content is obtained by the above method. It can be seen that since the text description content includes the key text "tutorial", and the classification feature of the video also reflects the key feature "food", the "food tutorial" tag can be output.

[0060] Assuming that the relevant features of the above video are iteratively calculated, and the output characters are "oil, fried, food, product, crispy, skin, flower, raw, beautiful, food, teaching, program, snack, food", the target label can be determined as the final output label. Figure 3 Shown are "fried foods", "crispy peanuts", "food tutorials" and "snacks".

[0061] It can be seen that through the above method of the present application, tags that are not included in the historical tag library can be output based on the content of the video description text, thereby achieving the technical effect of improving the accuracy of the output video tags.

[0062] In an embodiment of the present invention, a target video to be identified is obtained; a target candidate character library of the target video is constructed using a video description text, wherein the target candidate character library includes a reference character set and a description character set, the reference character set includes reference characters obtained by segmenting multiple historical tag texts, and the description character set includes description characters obtained by segmenting the video description text; after extracting the video features of the target video and the description features of the video description text, the video features and the description features are iteratively calculated N times to obtain M candidate character sequences; the target characters are determined from each candidate character sequence based on the confidence level to obtain M target characters; and the M target characters are then spliced ​​into a target video label that matches the target video. That is, in an embodiment of the present application, the video features and the description text features of the target video are iteratively calculated, and then the target characters are output word by word from the description character set composed of the reference character set and the video description text, and then the target video label is formed by the output target characters. Since the target candidate character library for outputting characters in the embodiment of the present application is combined with a description character set, and the description character set includes fresh text content related to the video, fresh characters can be output in combination with the description character set, and then fresh labels can be obtained based on the output fresh characters, thereby improving the accuracy of the output video labels and solving the technical problem of low accuracy of video labels obtained by existing video label determination methods.

[0063] As an optional implementation, the above-mentioned N-times iterative calculation of the video features and the description features to obtain M candidate character sequences includes:

[0064] S1, in the process of performing the i-th iterative calculation on the video features and the description features, in the vector conversion network, based on the visual features, the description features and the iterative reference vector obtained by the i-1th iterative calculation, the i-th intermediate latent vector set is calculated, wherein the i-th intermediate latent vector set includes the i-th visual latent vector matching the visual features, the i-th description latent vector matching the description features, and the iterative reference vector obtained by the i-th iterative calculation, and i is a natural number greater than or equal to 1 and less than or equal to N;

[0065] S2, based on the iterative reference vector calculated in the i-th iteration, determining the confidence of each character in the target candidate character library, and determining the j-th candidate character sequence according to the confidence, where j is a natural number greater than or equal to 0 and less than or equal to i;

[0066] S3, when the iterative reference vector obtained in the iterative calculation of the i-th iteration does not meet the end condition, perform the (i+1)-th iterative calculation;

[0067] S4, when the iteration reference vector calculated in the i-th iteration reaches the end condition, the iteration reference vectors calculated in the previous i iterations are determined as N iteration reference vectors.

[0068] It should be noted that the above-mentioned vector conversion network can be a Transformer model. The typical structure of Transformer includes an encoder and a decoder. The two structures are usually collectively referred to as Transformer modules. The difference between the two structures is that the decoder can only see the information of the current position and the previous position during the calculation process. The models that can be used in this embodiment include the BERT model (Bidirectional Encoder Representation from Transformers), the GPT model (Generative Pre-Training), the UniLM model (Unified Language Model) and the VPUniLM model. As a preferred method, the VPUniLM model can be used as the above-mentioned vector conversion network in this embodiment.

[0069] It is understandable that the visual features of the target video may be image features extracted from each frame of the target video, or image features extracted from the style of the target video. The form of the specific visual features used to characterize the content of the target video is not limited here.

[0070] It should be noted that in the above model, while obtaining the visual features and descriptive features in the video features, it is also necessary to obtain the iterative reference vector obtained by the previous iterative calculation, so as to use the above visual features, descriptive features and the iterative reference vector obtained by the previous iterative calculation as the input of the current iterative calculation, and then calculate the visual latent vector matching the visual features, the descriptive latent vector matching the descriptive features, and the current iterative reference vector. It can be understood that in this embodiment, since each iterative calculation process needs to be combined with the iterative reference latent vector obtained by the previous iterative calculation as the input, the result of each iterative calculation is related to the iterative result obtained by the historical iterative calculation, or in other words, each iterative calculation result indicates the characteristics of the historical iterative calculation results, and then the correlation between the output iterative calculation results can be determined by the order of iterative output.

[0071] The above-mentioned counting letters i and j are explained below. It can be understood that for the i-th iterative calculation, the j-th candidate character sequence for generating the target character can be determined, and j can be less than or equal to i. That is to say, the number of iterative reference vectors obtained by iterative calculation can be greater than or equal to the number of candidate character sequences. In a specific implementation, the iterative reference vector obtained for the i-th time can be a vector used to indicate a separator. For example, the iterative reference vector obtained by the first iterative calculation is used to determine the first candidate character sequence; the iterative reference vector obtained by the second iterative calculation is used to determine the second candidate character sequence; the iterative reference vector obtained by the third iterative calculation is used to determine the first separator; the iterative reference vector obtained by the fourth iterative calculation is used to determine the third candidate character sequence; the iterative reference vector obtained by the fifth iterative calculation is used to determine the fourth candidate character sequence; the iterative reference vector obtained by the sixth iterative calculation indicates the end separator. Furthermore, corresponding to the above six iterative reference calculations, four candidate character sequences can be determined, and further, four target characters can be determined based on the four candidate character sequences, and two target labels can be obtained based on the above four target character combinations.

[0072] In another embodiment, the counting letters i and j can be equal, that is, the number of iterative reference vectors obtained by iterative calculation can be equal to the number of candidate character sequences. For example, five candidate character sequences are determined corresponding to five iterative reference vectors, and then five target characters are determined, and one or more target labels are further obtained by splicing the five target characters.

[0073] Through the above method of the present application, in the process of performing the i-th iterative calculation on the video features and the description features, the i-th intermediate latent vector set is calculated in the vector conversion network based on the visual features, the description features and the iterative reference vector obtained by the i-1th iterative calculation in the video features; based on the iterative reference vector obtained by the i-th iterative calculation, the confidence of each character in the target candidate character library is determined, and the j-th candidate character sequence is determined according to the confidence, where j is a natural number greater than or equal to 0 and less than or equal to i; when the iterative reference vector obtained by the i-th iterative calculation does not meet the end condition, the i+1-th iterative calculation is performed; when the iterative reference vector obtained by the i-th iterative calculation meets the end condition, the iterative reference vector obtained by the previous i-th iterative calculation is determined as N iterative reference vectors, and then the corresponding candidate character sequence is determined based on the iterative reference vector. In the above real-time method of the present application, the intermediate latent vector is generated in an iterative manner, and the candidate character sequence is determined based on the iterative reference vector in the intermediate latent vector, thereby ensuring the chronological relationship between the candidate character sequences. The output target characters and the spliced ​​target labels are determined by using a candidate character sequence with a chronological association relationship, thereby achieving the technical effect of outputting video labels with high accuracy and solving the technical problem of low label accuracy obtained by existing label generation methods.

[0074] As an optional implementation, the above-mentioned determination of the confidence of each match of each character in the target candidate character library based on the iterative reference vector calculated at the i-th iteration includes: in a label generation network connected to the vector conversion network, determining a first confidence of each match of each reference character in the reference character set and a second confidence of each match of each description character in the description character set based on the iterative reference vector calculated at the i-th iteration.

[0075] It is understandable that in this embodiment, in the process of calculating the confidence by iterating the reference vector, the process of calculating the confidence of each reference character in the reference character set is different from the process of calculating the confidence of each description character in the description character set. For example, different calculation methods and different calculation parameters can be used to distinguish the method of calculating the first confidence of the reference character set and the method of calculating the second confidence of the description character set.

[0076] In an optional manner, a higher weight parameter can be configured for the initial first confidence corresponding to the calculated reference character set to obtain the first confidence; a lower weight parameter can be configured for the initial second confidence corresponding to the calculated description character set to obtain the second confidence, thereby improving the confidence of each character in the reference character set, and thereby making the probability that the output character is a character in the reference character set higher, thereby achieving the technical effect of constraining the output character type and rules through the reference character set.

[0077] Through the above-mentioned embodiments of the present application, in a label generation network connected to a vector conversion network, based on the iterative reference vector obtained by the i-th iteration calculation, a first confidence of each reference character in the reference character set and a second confidence of each description character in the description character set are determined to distinguish the confidence calculation methods corresponding to the reference character set and the description character set, and then the probability of outputting characters in the reference character set and the description character set can be adjusted as needed, thereby achieving the technical effect of constraining the output character type and rules through the reference character set.

[0078] As an optional implementation, the above-mentioned determination of the first confidence of the respective matching of each reference character in the reference character set and the second confidence of the respective matching of each description character in the description character set based on the iterative reference vector calculated in the i-th iteration includes:

[0079] S1, determining a first weight coefficient and a first bias parameter that match each reference character in a label generation network, a second weight coefficient and a second bias parameter that match each description character, and a reference weight and a reference bias parameter;

[0080] S2, determining a first confidence level of each reference character based on the iterative reference vector, the first weight coefficient and the first bias parameter obtained by the iterative calculation of the i-th time;

[0081] S3, obtaining a first intermediate result calculated based on the i-th description latent vector, the second weight coefficient and the second bias parameter, and a second intermediate result calculated based on the iterative reference vector, the reference weight and the reference bias parameter calculated based on the i-th iteration;

[0082] S4: Determine a second confidence level of each description character based on the first intermediate result and the second intermediate result.

[0083] The above implementation is described in detail below. It should be noted that, in this embodiment, the video description text is the video title.

[0084] First, assume that the above description latent vector is The nth latent vector corresponding to the nth character is output after one iteration of the Transformer model to indicate and describe the text feature (the title in this embodiment); assuming that the reference iteration latent vector is It should be noted that in this embodiment, multiple inputs and multiple inputs are required during the iterative calculation process. Represents the latent vector of the last position of the output calculated at the tth iteration.

[0085] Next, according to the training results of the model, the first weight coefficient and the first bias parameter that match each reference character in the label generation network, the second weight coefficient and the second bias parameter that match each description character, and the reference weight and reference bias parameters are determined, that is, the following 6 parameters are determined:

[0086] The first weight coefficient: Represents the weight parameter of the i-th character in the reference character set;

[0087] First bias parameter:

[0088] Second weight coefficient: W title , W dec ;

[0089] Second bias parameter: b title and b dec is the bias parameter.

[0090] Based on the parameters obtained from the above training, assuming Represents the hidden vector output of the model at the title position (i.e., describing the hidden vector). In the t-th step decoding stage, Represents the hidden vector output of the last position of the step output (i.e., the reference iterative hidden vector), represents the score of the i-th character in the reference character set at step t, then Calculated by the following formula:

[0091]

[0092] In the formula, Represents the weight parameter of the i-th character in the reference character set, is the offset parameter of the i-th character in the reference character set.

[0093] In order to select characters from the title text, a pointer network module is added to predict the score of selecting words from the title. Assume that the latent vector corresponding to the nth character in the title is represents the score of the nth character at step t, Calculated by the following formula:

[0094]

[0095] Where W title and W dec is the weight parameter, b title and b dec is the bias parameter.

[0096] It should be noted that the first intermediate result is The second intermediate result is the The first matching degree of each reference character is The second matching degree of each description character is

[0097] Through the above-mentioned implementation mode of the present application, by determining the first weight coefficient and the first bias parameter that match each reference character in the label generation network, the second weight coefficient and the second bias parameter that match each description character, and the reference weight and the reference bias parameter; based on the iterative reference vector, the first weight coefficient and the first bias parameter calculated by the i-th iteration, the first confidence of each reference character is determined; the first intermediate result calculated based on the i-th description latent vector, the second weight coefficient and the second bias parameter, and the second intermediate result calculated based on the iterative reference vector, the reference weight and the reference bias parameter calculated by the i-th iteration are obtained; based on the first intermediate result and the second intermediate result, the second confidence of each description character is determined, and the second matching degree of each text segmentation is determined, so as to accurately determine the score of each candidate character in the reference character set and the score of each description character in the description character set, and then the score of each candidate character in the current iterative calculation step can be accurately calculated, thereby improving the accuracy of label generation.

[0098] As an optional implementation, the above-mentioned iterative reference vector obtained based on the i-th iteration calculation determines the confidence of each character in the target candidate character library, and determines the j-th candidate character sequence according to the confidence, including: performing weighted sum calculation on the multiple first confidences and multiple second confidences obtained through the fully connected layer in the label generation network to obtain the confidence of each character in the target candidate character library to generate the j-th candidate character sequence.

[0099] It can be understood that the first matching degree of each reference character obtained by the t-th iteration calculation is The second matching degree corresponding to each description character is In one implementation, different weights may be configured for the first matching degree and the second matching degree, and the confidence corresponding to each tag is determined only based on the result of the weighted sum.

[0100] In another embodiment, in the multiple tag extraction steps, the first matching degree of each reference character calculated in the t-th iteration is obtained. The second matching degree of each description character is Assign the same weight to the first matching degree and the second matching degree, and then And the corresponding characters are combined to directly obtain the candidate character sequence.

[0101] Through the above-mentioned implementation mode of the present application, a weighted sum calculation is performed on the multiple first confidences and multiple second confidences obtained through the fully connected layer in the label generation network to obtain the confidence of each character in the target candidate character library to generate the jth candidate character sequence, thereby determining multiple first matching degrees and second matching degrees through multiple iterative calculation steps, and determining multiple candidate sequences based on the multiple first matching degrees and second matching degrees, as well as target characters corresponding to the candidate sequences, thereby improving the accuracy of generated labels.

[0102] As an optional implementation manner, before obtaining the target video to be identified, the above method further includes:

[0103] S1, obtaining a sample video and a sample label matching the sample video, wherein the sample video carries a sample video description text;

[0104] S2, constructing a sample candidate character library of the sample video using the sample video description text, wherein the sample candidate character library includes a reference character set and a sample description character set, the reference character set includes reference characters obtained by segmenting multiple historical tags, and the sample description character set includes description characters obtained by segmenting the sample video description text;

[0105] S3, using sample videos, sample labels and sample candidate word libraries to train the initialized label generation network until the training convergence condition is reached.

[0106] The following is an explanation of how to implement the training phase. In the training phase, the labels corresponding to the known videos, such as a video containing two labels "Game" and "City Heroes", are connected with the symbol ";" as the true value, and the visual features, category embedding vector, title word embedding vector and true value word embedding vector of the video are input together. In the prediction phase, the visual features, category embedding vector, title word embedding vector and the currently predicted word embedding vector are input. The model uses the encoder operation to calculate the latent vector of the visual features and the original title word embedding vector position, and uses the decoder operation to calculate the latent vector of the true value position. The latent vector is subjected to a layer of full connection operation to calculate the probability distribution of the dictionary. Here, the loss function type is cross entropy, n represents the length of the predicted text, and V represents the dictionary.

[0107]

[0108] When the loss function shown in the above formula reaches the convergence condition, the following parameter training is determined to be completed: First weight coefficient: Represents the weight parameter of the i-th character in the reference character set;

[0109] First bias parameter:

[0110] Second weight coefficient: W title , W dec ;

[0111] Second bias parameter: b title , b dec .

[0112] The following describes the training process of another UniLM (Unified Language Model) model.

[0113] Through the above method of the present application, a sample video and a sample label matching the sample video are obtained, wherein the sample video carries a sample video description text; a sample candidate character library of the sample video is constructed using the sample video description text, wherein the sample candidate character library includes a reference character set and a sample description character set, wherein the reference character set includes reference characters obtained by segmenting multiple historical labels, and the sample description character set includes description characters obtained by segmenting the sample video description text; a method of training an initialized label generation network using the sample video, the sample label and the sample candidate character library until the training convergence condition is reached, thereby obtaining the above label generation network in a supervised manner, improving the reliability of the target label obtained by the above label generation network, and solving the technical problem of low label accuracy obtained by the existing label generation method.

[0114] As an optional manner, in the above process of performing the i-th iterative calculation on the video features and the description features, the i-th intermediate latent vector set is calculated in the vector conversion network based on the visual features, the description features in the video features and the iterative reference vector calculated in the i-1th iterative calculation, including:

[0115] When i is equal to 1, the visual features and the descriptive features are converted into the word embedding vector corresponding to the target video; the category embedding vector, the paragraph embedding vector and the position embedding vector corresponding to the target video are extracted, wherein the category embedding vector is used to indicate the category information of the target video, the paragraph embedding vector includes a first paragraph sub-vector obtained based on the visual features and a second paragraph sub-vector obtained based on the descriptive features, and the position embedding vector is used to indicate the frame sequence of each video frame in the target video, and the order of each text segmentation in the video description text; the word embedding vector, the category embedding vector, the paragraph embedding vector and the position embedding vector are used as the input vector of the vector conversion network to calculate the i-th intermediate latent vector set;

[0116] When i is greater than 1, the word embedding vector, category embedding vector, paragraph embedding vector, position embedding vector, and the iterative reference vector calculated at the i-1th iteration are used as the input vector of the vector conversion network to calculate the i-th intermediate latent vector set.

[0117] As an optional implementation, the i-th intermediate latent vector set is calculated in the vector conversion network based on the visual features, the description features in the video features, and the iterative reference vector calculated in the i-1th iteration, including:

[0118] S1, determining an i-th visual latent vector and an i-th descriptive latent vector based on visual features and descriptive features, wherein the i-th visual latent vector and the i-th descriptive latent vector are used to indicate the contextual relationship of characters in the target video label;

[0119] S2, determining an iterative reference vector calculated at an i-th iteration based on the visual features, the descriptive features and the iterative reference vector calculated at the i-1-th iteration.

[0120] The category embedding vector of the above-mentioned target video is a feature used to indicate the approximate category of the target video. For example, if the target video is a video about a sports meeting, the category feature of the video can be used to characterize the video as "sports", and the corresponding category embedding vector can be (0, 0, 0, 0, 1); for another example, if the target video is a video of a song mv, the category feature of the video can be used to characterize the video as "song", and the corresponding category embedding vector can be (0, 0, 0, 0, 2); for another example, if the target video is a documentary "Chinese Food", the category feature of the video can be used to characterize the video as "food" and "documentary", and the corresponding category embedding vector can be (0, 0, 0, 0, 3). The above is only an exemplary description, and does not limit the specific content corresponding form of the category embedding vector that specifically characterizes the target video in this embodiment.

[0121] The following combination Figure 4 The above method is described.

[0122] like Figure 4 As shown, before iterative calculation, the vector combination input into the above vector conversion network includes: position embedding vector, paragraph embedding vector, word embedding vector and category embedding vector.

[0123] With respect to the category features of the video, in this embodiment, each video category may correspond to a vector group, so that the video features are represented by the vector group. Figure 4 In the model diagram shown, the converted category embedding vector is (5, 5, 5, 5, 5, 5).

[0124] Next, the visual features and descriptive features are converted into word embedding vectors. First, the target video is framed at a frame rate of 1fps, and the extracted video frames are input into the CLIP model to obtain the visual feature vector, corresponding to Figure 4 The video vector (F, F, F) in . And convert the video description text into a vector, corresponding to Figure 4 The title vector in (I, love, you) is taken and added to the connection vector CLS to form the word embedding vector (F, F, F, CLS, I, love, you).

[0125] At the same time Figure 4 As shown, the paragraph embedding vector and position embedding vector of the video are also obtained. Among them, the paragraph embedding vector is (0, 0, 0, 1, 1, 1, 1). It can be understood that the paragraph embedding vector is used to represent the video vector part and the title vector part in the word embedding vector, among which (0, 0, 0) corresponds to the video vector (F, F, F), and (1, 1, 1) corresponds to the title vector (I, love, you). The position embedding vector (0, 1, 2, 0, 1, 2, 3) is used to represent the order of the vectors in the input model. For example, the (0, 1, 2) part is used to identify the frame order of the video frame of the video vector (F, F, F); the (1, 2, 3) part is used to identify the order of the three text segmentations of the title vector (I, love, you).

[0126] like Figure 4 As shown in Figure 2, after the above position embedding vector, paragraph embedding vector, word embedding vector, and category embedding vector are input into the Transformer module for the first time, the intermediate latent vector set is obtained. (Not shown in the figure ).

[0127] It can be understood that by inputting the above intermediate latent vector set into the pointer network module, the scores of each character in the reference character set and the description character set can be obtained.

[0128] Through the above method of the present application, when i is equal to 1, the visual features and the descriptive features are converted into the word embedding vector corresponding to the target video; the category embedding vector, the paragraph embedding vector and the position embedding vector corresponding to the target video are extracted, wherein the category embedding vector is used to indicate the category information of the target video, the paragraph embedding vector includes a first paragraph sub-vector obtained based on the visual features and a second paragraph sub-vector obtained based on the descriptive features, and the position embedding vector is used to indicate the frame sequence of each video frame in the target video, and the order of each text segmentation in the video description text; the word embedding vector, the category embedding vector, the paragraph embedding vector and the position embedding vector are used as the input vector of the vector conversion network to calculate the i-th intermediate latent vector set, so as to obtain the confidence value of each candidate character through the pointer network module and the Transformer module, thereby solving the technical problem of low label accuracy obtained by the existing label generation method.

[0129] As an optional implementation, determining the target character from each candidate character sequence based on confidence includes: according to the contextual relationship of the characters in the target video tag, using the target character determined in the jth candidate character sequence as the jth target character in the target video tag.

[0130] It is understandable that in the model prediction stage, the above method will be used to obtain and Concatenate them together and select the character with the highest score, which is the tth candidate character sequence obtained in the current step, and output the tth target character.

[0131] Continue to combine Figure 4 The above method is described.

[0132] like Figure 4 As shown, the process of the first iterative calculation is as follows:

[0133] Will Figure 4 After the position embedding vector, paragraph embedding vector, word embedding vector, and category embedding vector shown in the figure are input into the Transformer module for the first time, the intermediate latent vector set output is obtained. as well as (not shown in the figure).

[0134] based on And the training W title , W dec , b title and b dec , the scores of the i-th character in the reference character set and the n-th character in the description character set in the iterative calculation process of step 1 are calculated by the following formulas:

[0135]

[0136]

[0137] Among them, represents the score of the i-th character in the reference character set at the first step, represents the score of the n-th character in the description character set at the first step. Concatenate and , and select the word with the highest score, which is the output word of the first step. As Figure 4 shown, the output word of the first step is "love";

[0138] The parsing steps of the second step are as follows:

[0139] Input the above position embedding vector, paragraph embedding vector, word embedding vector, and category embedding vector, as well as the obtained in the first step into the Transformer module to obtain an intermediate hidden vector set and

[0140] Based on and the W title 、W dec 、b title and b dec trained, calculate the scores of the i-th character in the reference character set and the n-th character in the description character set respectively in the second-step iterative calculation process through the following formula:

[0141]

[0142]

[0143] Among them, represents the score of the i-th character in the reference character set at the second step, represents the score of the n-th character in the description character set at the second step. Concatenate and , and select the word with the highest score, which is the output word of the second step. As Figure 4 shown, the output word of the first step is "feeling";

[0144] The ending steps are as follows:

[0145] Input the above position embedding vector, paragraph embedding vector, word embedding vector, and category embedding vector, as well as the obtained in the first step into the Transformer module to obtain an intermediate hidden vector set as well as in, When the preset end mark vector is matched, the iteration step is determined to be ended and no new characters will be generated.

[0146] Then, the output word "love" of the first step and the output word "emotion" of the second step are concatenated, and the target label of the target video titled "I love you" is obtained as "love".

[0147] Through the above-mentioned implementation mode of the present application, through iterative calculation and word-by-word output, since the target candidate character library used to output characters in the embodiment of the present application is combined with a description character set, and the description character set includes fresh text content related to the video, it is possible to output fresh characters in combination with the description character set, and then obtain fresh labels based on the output fresh characters, thereby improving the accuracy of the output video labels and solving the technical problem of low accuracy of video labels obtained by existing video label determination methods.

[0148] As an optional implementation, after splicing the M target characters into a target video label that matches the target video, the method further includes: adjusting the order of the M target characters in the target video label based on the contextual semantic relationship to obtain an updated target video label.

[0149] In an optional manner, assuming that multiple iterations are performed according to the above method, and assuming that the M characters initially outputted are "city", "city", "hero", "hero", "game", "game", "game", "good" in order, then according to the semantic analysis of the above characters, it can be determined that the order of the two characters "hero" and "hero" is adjusted to obtain the "hero" entry; at the same time, the order of the three characters "game", "game", and "good" can also be adjusted to obtain the "good game" entry; and then the final order of the updated M characters is determined to be: "city", "city", "hero", "hero", "game", "game", "good", "game", and "game". And based on the character order, the target label is determined: "city hero", "game", and "good game".

[0150] The following is an explanation of the order of outputting the characters. Figure 5 As shown in the figure, the attention matrix can be used in the above Transformer module for character input and output. As shown in the figure, each row of the matrix represents the output, and each column represents the input, and the attention matrix represents the association between the output and the input. Assuming that the white squares represent 0 and the black squares represent 1, then the first row indicates that the output character (x1) can only be associated with the start mark <s>The second line indicates that the output character (x2) can only be followed by the start mark <s>It is related to (x1), and so on. That is to say, by simply introducing a lower triangular form of Mask in the Attention matrix of the Transformer and staggering the input and output by one position for training, a unidirectional language model can be achieved, that is, characters are output in a predetermined order.

[0151] Optionally, a scrambled language model is described.

[0152] Similar to a language model, a scrambled language model also performs conditional probability decomposition, but the decomposition order is random, as shown in the following formula:

[0153] p(x1,x2,x3,…x n )=p(x1)p(x2|x2)p(x3|x1,x2)…p(x n |x1,x2,…x n-1 )

[0154] =p(x3)p(x1|x3)p(x2|x3,x1)…p(x n |x3,x1,…x n-1 )

[0155] =…

[0156] =p(x n-1 )p(x1|x n-1 )p(x n |x n-1 ,x1)…p(x2|x n-1 ,x1,…x3)

[0157] According to the above formula, for characters x1, x2…x n , any "appearance order" is possible. In principle, each order corresponds to a model, so in principle there are n! language models. For a model based on Transformer, all these orders can be implemented in the same model.

[0158] The following is combined with Figure 6 to illustrate the implementation method. Taking the generation of "Welcome to Beijing" as an example, assume that the required character order for generation is: " <s>→Welcome→Beijing→You→Welcome→North→ <e>”, then through Figure 6 the Attention matrix in the above-mentioned MSAK of the above embodiment, the character output in the above order can be realized. As Figure 6 shown, there is only one black square in the 4th row, indicating that "Ying" can only be followed by the start marker <s>is related, and the second row has two black squares, indicating that "Jing" can only be associated with the starting tag <s>Related to "welcome", and so on. Intuitively, this is like "shuffling" the Mask in the lower triangular form of the unidirectional language model.

[0159] That is to say, implementing a language model with a specific order is equivalent to shuffling the original lower triangular Mask in a certain way. Because the above-mentioned Attention provides such an n×n Attention matrix, the Transformer module in this embodiment can have enough degrees of freedom to Mask the above character matrix in different ways, so as to achieve the effect of diverse character output orders.

[0160] The following combines Figure 7 The following flow is shown to illustrate the complete process of the video tag generation method provided by this application:

[0161] S702, construct a target candidate character library;

[0162] Taking the video description text as the video title as an example to illustrate this embodiment. Suppose the video title is "I love you", and the reference character library includes the characters "dear" and "love". That is, combining the above video title and the characters in the above reference character library to obtain the target candidate character library: "I", "love", "you", "dear", "love".

[0163] S704, extract video vectors;

[0164] Specifically, as Figure 4 shown, the vector combinations input into the above vector conversion network include: position embedding vectors, paragraph embedding vectors, word embedding vectors, and category embedding vectors.

[0165] For the category features of the video, in this embodiment, each video category can be corresponded to a vector group to represent the video features with the vector group. In Figure 4 the model schematic diagram shown, the converted category embedding vector is (5, 5, 5, 5, 5, 5, 5).

[0166] Then, convert the visual features and description features into word embedding vectors. First, extract frames from the target video, and the extraction frame rate is 1fps. Input the extracted video frames into the CLIP model to obtain visual feature vectors, corresponding to Figure 4 the video vectors (F, F, F) in. And convert the video description text into a vector, corresponding to Figure 4 the title vector (I, love, you) in, and add the connection vector CLS, that is, form the word embedding vector (F, F, F, CLS, I, love, you).

[0167] At the same time, as Figure 4 As shown, the paragraph embedding vector and position embedding vector of the video are also obtained. Among them, the paragraph embedding vector is (0, 0, 0, 1, 1, 1, 1). It can be understood that the paragraph embedding vector is used to represent the video vector part and the title vector part in the word embedding vector, among which (0, 0, 0) corresponds to the video vector (F, F, F), and (1, 1, 1) corresponds to the title vector (I, love, you). The position embedding vector (0, 1, 2, 0, 1, 2, 3) is used to represent the order of the vectors in the input model. For example, the (0, 1, 2) part is used to identify the frame order of the video frame of the video vector (F, F, F); the (1, 2, 3) part is used to identify the order of the three text segmentations of the title vector (I, love, you).

[0168] S706, inputting the extracted video vector into the Transformer model to obtain an output intermediate vector;

[0169] Specifically, Figure 4 After the position embedding vector, paragraph embedding vector, word embedding vector, and category embedding vector shown in the figure are input into the Transformer module for the first time, the intermediate latent vector output is obtained. as well as (not shown in the figure).

[0170] S708, determining whether the intermediate vector includes the end vector;

[0171] i.e. judgment Is it the end vector? If the vector is not ended, execute the subsequent steps.

[0172] S710, performing analytical calculation on the intermediate vector based on the trained parameter set to obtain the score of each candidate character in the tth step;

[0173] S712, determining whether the current candidate character is the candidate character with the highest score;

[0174] S714, determining the candidate character with the highest score as the output character of this step;

[0175] Specifically, based on And the training W title , W dec , b title and b dec , the scores of the i-th word in the history tag vocabulary and the n-th word in the title vocabulary in the first step of parsing are calculated by the following formulas:

[0176]

[0177]

[0178] Among them, represents the score of the i-th word in the historical tag vocabulary at the first step, represents the score of the n-th word in the video title at the first step. Combine and together, and select the word with the highest score, which is the output word of the first step. Suppose the first candidate character sequence obtained is: "I" (70%), "love" (90%), "you" (76%), "dear" (40%), "affection" (80%), and then determine that the character with the highest confidence in the above first candidate character sequence is "love", and then as Figure 4 shown, the output character of the first step is "love".

[0179] Since is not the end vector, then execute the second-step iterative calculation steps as follows:

[0180] Input the above position embedding vector, paragraph embedding vector, word embedding vector, category embedding vector, and the obtained in the first step into the Transformer module to get the intermediate hidden vector output and

[0181] Based on the judgment which is also not the end vector, based on and the W title 、W dec 、b title and b dec trained, calculate the score of the i-th word in the historical tag vocabulary and the score of the n-th word in the title vocabulary respectively in the second-step parsing process through the following formula:

[0182]

[0183]

[0184] Among them, represents the score of the i-th word in the historical tag vocabulary at the second step, represents the score of the n-th word in the video title at the second step. Combine and Concatenate them and select the word with the highest score, which is the output word of the second step. Suppose the second candidate character sequence obtained is: "I" (60%), "love" (20%), "you" (60%), "dear" (40%), "affection" (80%). Then determine that the character with the highest confidence in the above first candidate character sequence is "affection", as Figure 4 shown, the output word of the second step is "affection";

[0185] Since it is not the end vector, then execute the third step of the parsing process as follows:

[0186] Input the above position embedding vector, paragraph embedding vector, word embedding vector, and category embedding vector, as well as the obtained in the first step, into the Transformer module to get the intermediate hidden vector output and Among them, matches the preset end flag vector, that is, it is determined that the parsing process ends and no more tags are generated.

[0187] S716, output the output characters determined in each step, and concatenate them to generate video tags.

[0188] Finally, concatenate the output word "love" of the first step and the output word "affection" of the second step to get the tag of the target video with the title "I love you" as "love and affection".

[0189] In the above-mentioned embodiment of the present invention, a target video to be identified is obtained; a target candidate character library of the target video is constructed using the video description text, wherein the target candidate character library includes a reference character set and a description character set, the reference character set includes reference characters obtained by segmenting multiple historical tag texts, and the description character set includes description characters obtained by segmenting the video description text; when the video features of the target video and the description features of the video description text are extracted, the video features and the description features are iteratively calculated N times to obtain M candidate character sequences; the target characters are determined from each candidate character sequence based on the confidence level to obtain M target characters; and then the M target characters are spliced ​​into a target video label that matches the target video. That is to say, in the embodiment of the present application, the video features and the description text features of the target video are iteratively calculated, and then the target characters are output word by word from the description character set composed of the reference character set and the video description text, and then the target video label is formed by the output target characters. Since the target candidate character library for outputting characters in the embodiment of the present application is combined with a description character set, and the description character set includes fresh text content related to the video, fresh characters can be output in combination with the description character set, and then fresh labels can be obtained based on the output fresh characters, thereby improving the accuracy of the output video labels and solving the technical problem of low accuracy of video labels obtained by existing video label determination methods.

[0190] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0191] According to another aspect of the embodiments of the present invention, a device for generating a video tag for implementing the above-mentioned method for generating a video tag is also provided. Figure 8 As shown, the device comprises:

[0192] An acquisition unit 802 is used to acquire a target video to be identified, wherein the target video carries a video description text for describing the target video;

[0193] A construction unit 804 is used to construct a target candidate character library of a target video using the video description text, wherein the target candidate character library includes a reference character set and a description character set, the reference character set includes reference characters obtained by segmenting a plurality of historical tag texts, and the description character set includes description characters obtained by segmenting the video description text;

[0194] The calculation unit 806 is used to perform N iterative calculations on the video features and the description features after extracting the video features of the target video and the description features of the video description text, so as to obtain M candidate character sequences, wherein the candidate character objects in each candidate character sequence include each candidate character in the target candidate character library and the confidence degree of the candidate character, the confidence degree is used to indicate the matching degree between the candidate character and the target video, and N and M are natural numbers greater than or equal to 1;

[0195] A determination unit 808, configured to determine a target character from each candidate character sequence based on the confidence level, to obtain M target characters;

[0196] The splicing unit 810 is used to splice the M target characters into a target video label that matches the target video.

[0197] Optionally, in this embodiment, the embodiments to be implemented by the above-mentioned various unit modules can refer to the above-mentioned various method embodiments, which will not be repeated here.

[0198] According to another aspect of the embodiments of the present invention, an electronic device for implementing the above-mentioned method for generating video tags is also provided. The electronic device may be Fig. 9 The terminal device or server shown in the figure. This embodiment is described by taking the electronic device as a terminal device as an example. Fig. 9 As shown, the electronic device includes a memory 902 and a processor 904. The memory 902 stores a computer program, and the processor 904 is configured to execute the steps in any of the above method embodiments through the computer program.

[0199] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0200] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0201] S1, obtaining a target video to be identified, wherein the target video carries a video description text for describing the target video;

[0202] S2, constructing a target candidate character library of the target video using the video description text, wherein the target candidate character library includes a reference character set and a description character set, the reference character set includes reference characters obtained by segmenting multiple historical tag texts, and the description character set includes description characters obtained by segmenting the video description text;

[0203] S3, when the video features of the target video and the description features of the video description text are extracted, the video features and the description features are iterated N times to obtain M candidate character sequences, wherein the candidate character objects in each candidate character sequence include each candidate character in the target candidate character library and the confidence level of the candidate character, the confidence level is used to indicate the matching degree between the candidate character and the target video, and N and M are natural numbers greater than or equal to 1;

[0204] S4, determining a target character from each candidate character sequence based on the confidence level, and obtaining M target characters;

[0205] S5, concatenate the M target characters into a target video label that matches the target video.

[0206] Alternatively, a person skilled in the art may understand that: Fig. 9 The structure shown is for illustration only, and the electronic device may also be a vehicle-mounted terminal, a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, and other terminal devices. Fig. 9 The structure of the electronic device is not limited. Fig. 9 More or fewer components (such as network interfaces, etc.) as shown in, or with Fig. 9 Different configurations are shown.

[0207] Among them, the memory 902 can be used to store software programs and modules, such as the program instructions / modules corresponding to the method and device for generating video tags in the embodiments of the present invention. The processor 904 executes various functional applications and data processing by running the software programs and modules stored in the memory 902, that is, realizing the above-mentioned method for generating video tags. The memory 902 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 902 may further include a memory remotely arranged relative to the processor 904, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 902 may be specifically used, but not limited to, to store information such as various elements in the observation perspective screen, generation information of video tags, and so on. As an example, such as Fig. 9 As shown, the memory 902 may include, but is not limited to, the acquisition unit 802, the construction unit 804, the calculation unit 806, the determination unit 808, and the splicing unit 810 in the video tag generation device. In addition, it may also include, but is not limited to, other module units in the video tag generation device, which will not be repeated in this example.

[0208] Optionally, the transmission device 906 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wired network and a wireless network. In one example, the transmission device 906 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 906 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0209] In addition, the electronic device mentioned above further includes: a display 908, and a connection bus 910, which is used to connect various module components in the electronic device mentioned above.

[0210] In other embodiments, the terminal device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes through network communication. Among them, the nodes may form a peer-to-peer (P2P, Peer To Peer) network, and any form of computing device, such as a server, terminal and other electronic devices, may become a node in the blockchain system by joining the peer-to-peer network.

[0211] According to one aspect of the present application, a computer program product is provided, the computer program product comprising a computer program / instruction, the computer program / instruction comprising a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit, various functions provided by the embodiments of the present application are executed.

[0212] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0213] According to one aspect of the present application, a computer-readable storage medium is provided, and a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned method for generating video tags.

[0214] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0215] S1, obtaining a target video to be identified, wherein the target video carries a video description text for describing the target video;

[0216] S2, constructing a target candidate character library of the target video using the video description text, wherein the target candidate character library includes a reference character set and a description character set, the reference character set includes reference characters obtained by segmenting multiple historical tag texts, and the description character set includes description characters obtained by segmenting the video description text;

[0217] S3, when the video features of the target video and the description features of the video description text are extracted, the video features and the description features are iterated N times to obtain M candidate character sequences, wherein the candidate character objects in each candidate character sequence include each candidate character in the target candidate character library and the confidence level of the candidate character, the confidence level is used to indicate the matching degree between the candidate character and the target video, and N and M are natural numbers greater than or equal to 1;

[0218] S4, determining a target character from each candidate character sequence based on the confidence level, and obtaining M target characters;

[0219] S5, concatenate the M target characters into a target video label that matches the target video.

[0220] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0221] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on such understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which can be personal computers, servers or network devices, etc.) to perform all or part of the steps of the above methods of various embodiments of the present invention.

[0222] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0223] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0224] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0225] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0226] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.< / s> < / s> < / e> < / s> < / s> < / s>

Claims

1. A method for generating a video tag, characterized in that: include: Acquire a target video to be identified, wherein the target video carries a video description text for describing the target video; Constructing a target candidate character library of the target video using the video description text, wherein the target candidate character library includes a reference character set and a description character set, the reference character set includes reference characters obtained by segmenting a plurality of historical tag texts, and the description character set includes description characters obtained by segmenting the video description text; In the case where the video features of the target video and the descriptive features of the video description text are extracted, N iterative calculations are performed on the video features and the descriptive features, wherein in the i-th iterative calculation process, an i-th intermediate latent vector set is calculated in the vector conversion network based on the visual features in the video features, the descriptive features, and the iterative reference vector obtained by the i-1-th iterative calculation, wherein the i-th intermediate latent vector set includes the i-th visual latent vector matching the visual features, the i-th descriptive latent vector matching the descriptive features, and the iterative reference vector obtained by the i-th iterative calculation, where i is greater than is a natural number that is equal to 1 and less than or equal to N; in a label generation network connected to the vector conversion network, based on the iterative reference vector calculated in the i-th iteration, determining a first confidence level of each reference character in the reference character set and a second confidence level of each description character in the description character set, and determining a j-th candidate character sequence according to the first confidence level and the second confidence level, wherein j is a natural number that is greater than or equal to 0 and less than or equal to i, the first confidence level and the second confidence level are used to indicate a degree of match between the candidate character and the target video, and N is a natural number that is greater than or equal to 1; Determine target characters respectively from the plurality of candidate character sequences; The plurality of target characters are concatenated into a target video tag matching the target video.

2. The method according to claim 1, characterized in that The performing N iterative calculations on the video features and the description features comprises: When the iterative reference vector obtained by the iterative calculation of the i-th iteration does not meet the termination condition, performing the (i+1)-th iterative calculation; When the iteration reference vector calculated in the i-th iteration reaches the end condition, the iteration reference vectors calculated in the previous i iterations are determined as N iteration reference vectors.

3. The method according to claim 1, characterized in that The determining, based on the iterative reference vector obtained by the iterative calculation of the i-th iteration, of a first confidence level of a match of each reference character in the reference character set and a second confidence level of a match of each description character in the description character set comprises: Determine a first weight coefficient and a first bias parameter that matches each of the reference characters, a second weight coefficient and a second bias parameter that matches each of the description characters, and a reference weight and a reference bias parameter in the label generation network; Determining the first confidence level of each of the reference characters based on the iterative reference vector, the first weight coefficient, and the first bias parameter obtained by the iterative calculation of the i-th time; Obtaining a first intermediate result calculated based on the i-th descriptive latent vector, the second weight coefficient, and the second bias parameter, and a second intermediate result calculated based on the iterative reference vector calculated by the i-th iteration, the reference weight, and the reference bias parameter; The second confidence level of each of the description characters is determined based on the first intermediate result and the second intermediate result.

4. The method according to claim 1, characterized in that Determining the jth candidate character sequence according to the first confidence level and the second confidence level comprises: The weighted sum of the obtained multiple first confidences and multiple second confidences is calculated through the fully connected layer in the label generation network to obtain the confidence of each character in the target candidate character library to generate the jth candidate character sequence.

5. The method according to claim 1, characterized in that Before obtaining the target video to be identified, it also includes: Obtaining a sample video and a sample tag matching the sample video, wherein the sample video carries a sample video description text; Constructing a sample candidate character library of the sample video using the sample video description text, wherein the sample candidate character library includes a reference character set and a sample description character set, the reference character set includes reference characters obtained by segmenting a plurality of historical tags, and the sample description character set includes description characters obtained by segmenting the sample video description text; The initialized label generation network is trained using the sample video, the sample label and the sample candidate character library until a training convergence condition is reached.

6. The method according to claim 1, characterized in that In the process of performing the i-th iterative calculation on the video features and the description features, the i-th intermediate latent vector set is calculated in the vector conversion network based on the visual features in the video features, the description features and the iterative reference vector obtained by the i-1-th iterative calculation, including: In the case where i is equal to 1, the visual features and the descriptive features are converted into word embedding vectors corresponding to the target video; the category embedding vector, the paragraph embedding vector and the position embedding vector corresponding to the target video are extracted, wherein the category embedding vector is used to indicate the category information of the target video, the paragraph embedding vector includes a first paragraph sub-vector obtained based on the visual features and a second paragraph sub-vector obtained based on the descriptive features, and the position embedding vector is used to indicate the frame sequence of each video frame in the target video, and the order of each text segmentation in the video description text; the word embedding vector, the category embedding vector, the paragraph embedding vector and the position embedding vector are used as input vectors of the vector conversion network to calculate the i-th intermediate latent vector set; When i is greater than 1, the word embedding vector, the category embedding vector, the paragraph embedding vector, the position embedding vector, and the iterative reference vector calculated in the i-1th iteration are used as input vectors of the vector conversion network to calculate the i-th intermediate latent vector set.

7. The method according to claim 1, characterized in that The step of calculating the i-th intermediate latent vector set in the vector conversion network based on the visual features in the video features, the description features, and the iterative reference vector calculated in the i-1th iteration includes: Determining the i-th visual latent vector and the i-th descriptive latent vector based on the visual features and the descriptive features, wherein the i-th visual latent vector and the i-th descriptive latent vector are used to indicate a contextual relationship of characters in the target video tag; The iterative reference vector obtained by the i-th iterative calculation is determined based on the visual features, the descriptive features and the iterative reference vector obtained by the i-1-th iterative calculation.

8. The method according to claim 7, characterized in that Determining the target character from each of the candidate character sequences based on the confidence level comprises: According to the contextual relationship of the characters in the target video tag, the target character determined in the j-th candidate character sequence is used as the j-th target character in the target video tag.

9. The method according to claim 8, characterized in that After the target characters are spliced ​​into a target video tag matching the target video, the method further includes: The order of the plurality of target characters in the target video tag is adjusted based on the contextual semantic relationship to obtain the updated target video tag.

10. A device for generating a video tag, characterized in that: include: An acquisition unit, used for acquiring a target video to be identified, wherein the target video carries a video description text for describing the target video; A construction unit, configured to construct a target candidate character library of the target video using the video description text, wherein the target candidate character library includes a reference character set and a description character set, the reference character set includes reference characters obtained by segmenting a plurality of historical tag texts, and the description character set includes description characters obtained by segmenting the video description text; A calculation unit is used to perform N iterative calculations on the video features and the descriptive features after extracting the video features of the target video and the descriptive features of the video description text. In the process of the i-th iterative calculation, in the vector conversion network, based on the visual features in the video features, the descriptive features and the iterative reference vector obtained by the i-1-th iterative calculation, calculate the i-th intermediate latent vector set, wherein the i-th intermediate latent vector set includes the i-th visual latent vector matching the visual features, the i-th descriptive latent vector matching the descriptive features, and the iterative reference vector obtained by the i-th iterative calculation, where i is A natural number greater than or equal to 1 and less than or equal to N; in a label generation network connected to the vector conversion network, based on the iterative reference vector calculated in the i-th iteration, determine the first confidence of each reference character in the reference character set and the second confidence of each description character in the description character set, and determine the j-th candidate character sequence according to the first confidence and the second confidence, wherein j is a natural number greater than or equal to 0 and less than or equal to i, the first confidence and the second confidence are used to indicate the degree of match between the candidate character and the target video, and N is a natural number greater than or equal to 1; A determination unit, used to determine target characters from the plurality of candidate character sequences respectively; The splicing unit is used to splice the multiple target characters into a target video label matching the target video.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method described in any one of claims 1 to 9 when executed.

12. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 9 through the computer program.

Images