Video tag processing method and apparatus, and computer device and storage medium

Abstract

A video tag processing method, comprising: determining a video to be processed, and performing extraction on the video to obtain modality information of at least two modalities (202); on the basis of the modality information of the at least two modalities, respectively performing cross retrieval according to feature dimensions of the at least two modalities, so as to obtain respective multi-modal retrieval tags of the modality information of the at least two modalities (204); on the basis of the respective multi-modal retrieval tags of the modality information of the at least two modalities, determining at least one candidate tag for the video (206); and on the basis of the at least one candidate tag and the modality information of the at least two modalities, performing tag prediction, so as to obtain a video tag for the video (208).

Description

Video tag processing methods, apparatus, computer equipment and storage media

[0001] Related applications

[0002] This application claims priority to Chinese patent application filed on April 26, 2024, with application number 2024105113762, entitled "Video Tag Processing Method, Apparatus, Computer Equipment and Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of computer technology, and in particular to a video tag processing method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0004] With the increasing richness and diversity of video content, users have a growing need for efficient and accurate browsing and retrieval of video information. To meet this demand, video tagging tasks, which analyze video content to assign tags to videos, have emerged. Identifying corresponding tags for video content helps users quickly understand the content, enabling accurate searches and personalized recommendations, and also provides strong support for content management and ad targeting. However, the accuracy of current tag recognition for various types of video content is limited. Summary of the Invention

[0005] According to various embodiments provided in this application, a video tag processing method, apparatus, computer device, computer-readable storage medium, and computer program product are provided.

[0006] In a first aspect, this application provides a video tag processing method, executed by a computer device, the method comprising:

[0007] Identify the video to be processed, and extract modal information of at least two modalities from the video;

[0008] Based on modal information of at least two modalities, cross-retrieval is performed according to the feature dimensions of at least two modalities to obtain multimodal retrieval tags for each of the at least two modalities.

[0009] Based on the multimodal retrieval tags of at least two modalities, determine at least one candidate tag for the video; and

[0010] Video tags for a video are obtained by performing tag prediction based on modal information of at least one candidate tag and at least two modalities.

[0011] Secondly, this application also provides a video tag processing apparatus. The apparatus includes:

[0012] The modal information extraction module is used to determine the video to be processed and extract modal information of at least two modalities from the video.

[0013] The cross-retrieval module is used to perform cross-retrieval based on modal information of at least two modalities, according to the feature dimensions of at least two modalities, to obtain multimodal retrieval tags for each of the modal information of at least two modalities.

[0014] A candidate tag determination module is used to determine at least one candidate tag for a video based on multimodal retrieval tags obtained from modal information of at least two modalities; and

[0015] The tag prediction module is used to predict tags based on at least one candidate tag and modal information of at least two modalities to obtain video tags for the video.

[0016] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing computer-readable instructions, and the processor executing the computer-readable instructions to implement the steps of the above-described video tag processing method.

[0017] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores computer-readable instructions thereon, which, when executed by a processor, implement the steps of the above-described video tag processing method.

[0018] Fifthly, this application also provides a computer program product. The computer program product includes computer-readable instructions that, when executed by a processor, implement the steps of the above-described video tag processing method.

[0019] Details of one or more embodiments of this application are set forth in the following drawings and description. Other features and advantages of this application will become apparent from the specification, drawings, and claims. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the disclosed drawings without creative effort.

[0021] Figure 1 is an application environment diagram of a video tag processing method in one embodiment;

[0022] Figure 2 is a flowchart illustrating a video tag processing method in one embodiment;

[0023] Figure 3A is a schematic diagram of the cross-retrieval process in one embodiment;

[0024] Figure 3B is a flowchart illustrating a video tag processing method based on three modalities in one embodiment;

[0025] Figure 4 is a schematic diagram of the process of determining labels using a label prediction model in one embodiment;

[0026] Figure 5 is a flowchart illustrating the cross-retrieval based on three modalities in one embodiment;

[0027] Figure 6 is a schematic diagram of the process of cross-retrieval based on a multimodal feature label library in one embodiment;

[0028] Figure 7 is a schematic diagram of the process of cross-retrieval based on two modal information in one embodiment;

[0029] Figure 8 is a flowchart illustrating the process of determining a label based on multimodal information in one embodiment;

[0030] Figure 9 is a flowchart illustrating the video tag processing method in another embodiment;

[0031] Figure 10 is a schematic diagram of an application scenario of the video tag processing method in one embodiment;

[0032] Figure 11 is a schematic diagram of feature matching of text images in one embodiment;

[0033] Figure 12 is a schematic diagram of the cross-retrieval process in one embodiment;

[0034] Figure 13 is a structural block diagram of a video tag processing device in one embodiment;

[0035] Figure 14 is an internal structure diagram of a computer device in one embodiment. Detailed Implementation

[0036] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0038] The video tagging method provided in this application embodiment can be applied to the application environment shown in Figure 1. The terminal 102 communicates with the server 104 via a network. A data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated into the server 104, or placed in the cloud or on another server. The terminal 102 can send the video to be processed to the server 104. After receiving the video, the server 104 performs cross-retrieval based on modal information extracted from the video in at least two modalities, according to the feature dimensions of at least two modalities. Based on the multimodal retrieval tags of the obtained modal information in at least two modalities, it determines at least one candidate tag for the video, and performs tag prediction based on the at least one candidate tag and the modal information in at least two modalities to obtain the video tag for the video. The server 104 can then feed back the obtained video tag for the video to the terminal 102. In addition, in some embodiments, the video tagging method can also be implemented by the server 104 or the terminal 102 separately. For example, the terminal 102 can directly perform video tagging on the video to be processed, or the server 104 can obtain the video to be processed from the data storage system and perform video tagging on the video to be processed.

[0039] The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.

[0040] In an exemplary embodiment, as shown in FIG2, a video tagging processing method is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is described using the server shown in FIG1 as an example, and includes the following steps 202 to 208. Wherein:

[0041] Step 202: Determine the video to be processed and extract modal information of at least two modalities from the video.

[0042] The videos to be processed are those that require specific tags. Video tags refer to keywords related to the video. By associating videos with relevant tags, accurate classification, retrieval, and push notifications can be performed. Modal refers to different types of information, specifically including but not limited to visual modalities, audio modalities, and text modalities. Visual modalities include visible content in the video, such as images, colors, and actions; these visual elements provide a direct visual experience and are an important component of video information delivery. Audio modalities include audible sounds in the video, such as speech, music, and background noise; audio modalities add auditory elements to the video, enhancing immersion and emotional resonance. Text modalities are text resources that describe the video content, such as subtitles, titles, and tags; text modalities provide additional information to the video, helping to better understand the content. Modal information can specifically include various types of information, such as visual modal information corresponding to visual modalities, audio modal information corresponding to audio modalities, and text modal information corresponding to text modalities.

[0043] Specifically, when the computer device executing the video tagging processing method includes a server, the server can determine the video that needs tagging and extract modal information of different modalities from the video. In specific implementations, the server can extract two or three modalities of information from the video. For example, the server can extract visual modal information corresponding to the visual modality, which may include video frames in the video; the server can extract audio modal information corresponding to the audio modality, which may include speech and music in the video; the server can also extract text modal information corresponding to the text modality, which may include subtitles, titles, and other information in the video; furthermore, the server can perform character recognition on the video frames to obtain character recognition results, which can be used as text modal information of the video. In some applications, the server can also perform speech recognition on the audio included in the video to obtain semantic recognition results, which can be used as text modal information of the video.

[0044] Step 204: Based on the modal information of at least two modalities, perform cross-retrieval according to the feature dimensions of at least two modalities to obtain the multimodal retrieval tags for each of the at least two modalities.

[0045] Cross-referencing involves performing multimodal tag retrieval on modal information for each modality according to the feature dimensions of each modality. In other words, for each modality, tag retrieval is performed separately according to the feature dimensions of each modality. Each modality can correspond to its own feature dimension, such as text modality corresponding to text feature dimension, visual modality corresponding to visual feature dimension, and audio modality corresponding to visual feature dimension. For example, as shown in Figure 3A, modal information of three modes is extracted from the video, including modal information 1 corresponding to mode A, modal information 2 corresponding to mode B, and modal information 3 corresponding to mode C. During cross-referencing, for modal information 1, tag retrieval can be performed according to the feature dimensions of mode A, mode B, and mode C respectively; similarly, for modal information 2 and modal information 3, tag retrieval can be performed according to the feature dimensions of mode A, mode B, and mode C respectively, yielding multimodal retrieval tag 1 corresponding to modal information 1, multimodal retrieval tag 2 corresponding to modal information 2, and multimodal retrieval tag 3 corresponding to modal information 3. This allows for multimodal feature dimension tag retrieval for the same modal information. Multimodal retrieval tags are the retrieval results obtained by performing multimodal feature dimension tag retrieval for the same modal information; specifically, they can include video tags retrieved for a particular modal information under various modal feature dimensions. For example, when performing tag retrieval for modal information 1 according to the feature dimensions of modality A, modality B, and modality C, we can obtain retrieval tag 1 under the feature dimension of modality A, retrieval tag 2 under the feature dimension of modality B, and retrieval tag 3 under the feature dimension of modality C. Based on retrieval tag 1, retrieval tag 2, and retrieval tag 3, we can obtain the multimodal retrieval tags for modal information 1.

[0046] Optionally, for each modality's modal information, the server can perform cross-retrieval according to multiple modal feature dimensions. These multiple modalities can specifically include various modalities to which the modal information extracted from the video belongs. For each modal information, tag retrieval can be performed on each modal feature dimension to retrieve tags matching that modal information, thus obtaining tag retrieval results under each modal feature dimension. Based on the tag retrieval results under various modal feature dimensions, the multimodal retrieval tags for that modal information can be obtained. The server traverses each modal information in at least two modalities to obtain the multimodal retrieval tags for each of the at least two modalities. In specific applications, each modal feature and its corresponding tag can be pre-constructed for each modal feature dimension. When performing cross-retrieval based on modal information, the modal information can be matched with each modal feature separately, and the tag corresponding to the matching modal feature is taken as the tag retrieval result for that modal information under the corresponding modal feature dimension. After traversing various modal feature dimensions for cross-retrieval, the multimodal retrieval tags corresponding to the modal information can be obtained.

[0047] Step 206: Determine at least one candidate tag for the video based on the multimodal retrieval tags of the modal information of at least two modalities.

[0048] Candidate tags are used to initially label the video, serving as preliminary tags for the video. Candidate tags are obtained based on the multimodal retrieval tags for each modal information. For example, the server can synthesize the multimodal retrieval tags for each modal information to obtain at least one candidate tag for the video. In some implementations, the server can perform statistical analysis on the multimodal retrieval tags for each modal information, such as frequency analysis for each tag, and determine at least one candidate tag based on the statistical results. For instance, after performing frequency analysis on each tag, the server can select the K most frequent tags as candidate tags for the video, where the value of K can be set according to actual needs. In some embodiments, different weights can be set for the multimodal retrieval tags corresponding to each modal information, thereby determining the candidate tags for the video based on a weighted average of the multimodal retrieval tags for each modal information.

[0049] Step 208: Perform label prediction based on at least one candidate label and modal information of at least two modalities to obtain video labels for the video.

[0050] In this context, video tags are the final tags determined for the video to be processed. Video tags are predicted based on candidate tags and modal information from various modalities. For example, the server can combine candidate tags and modal information from at least two modalities for tag prediction, such as inputting the candidate tags and modal information from at least two modalities into a pre-trained tag prediction model to obtain video tags for the video. In some embodiments, the server can also determine the features of each of the candidate tags and the modal information from at least two modalities separately, and fuse these features to obtain tag prediction features. The server can then perform tag prediction based on these fused tag prediction features, such as inputting the tag prediction features into a pre-trained tag prediction model to obtain video tags for the video.

[0051] In a specific application, as shown in Figure 3B, for the video to be processed, the server can extract visual modal information corresponding to the visual modality, text modal information corresponding to the text modality, and audio modal information corresponding to the audio modality. The server can perform cross-referencing based on the various modal information according to the multimodal feature dimensions, which include visual, text, and audio modalities. Specifically, for visual modal information, cross-referencing can be performed according to the feature dimensions of visual, text, and audio modalities to obtain multimodal retrieval label 1 for visual modal information. Multimodal retrieval label 1 includes retrieval results 11 corresponding to the visual modality, 12 corresponding to the text modality, and 13 corresponding to the audio modality. Retrieval result 11 is a retrieval result obtained by tagging visual modal information within the visual modal feature dimensions; retrieval result 12 is a retrieval result obtained by tagging visual modal information within the text modal feature dimensions; and retrieval result 13 is a retrieval result obtained by tagging visual modal information within the audio modal feature dimensions. Similarly, the multimodal retrieval tag 2 obtained from text modal information includes visual modal corresponding retrieval result 21, text modal corresponding retrieval result 22, and audio modal corresponding retrieval result 23. The multimodal retrieval tag 3 obtained from audio modal information includes visual modal corresponding retrieval result 31, text modal corresponding retrieval result 32, and audio modal corresponding retrieval result 33. The server can comprehensively determine at least one candidate tag for the video based on multimodal retrieval tag 1, multimodal retrieval tag 2, and multimodal retrieval tag 3. The server performs tag prediction based on the candidate tags, visual modal information, text modal information, and audio modal information to obtain video tags.

[0052] In a specific application, as shown in Figure 4, when the server performs label prediction based on candidate labels, visual modal information, text modal information, and audio modal information, the server can input the candidate labels, visual modal information, text modal information, and audio modal information into a pre-trained label prediction model, which will then predict the labels and output video labels. In some embodiments, the server can also first construct model input features based on candidate labels, visual modal information, text modal information, and audio modal information, and then input the constructed model input features into the label prediction model to perform label prediction and obtain the video labels. In some embodiments, when performing label prediction processing based on candidate labels and modal information, the server can perform label prediction processing based on candidate labels and all types of modal information; the server can also determine the modal information belonging to the target modality from at least two types of modal information. The target modality can be at least a subset of modal types selected from at least two modalities. The server can perform label prediction processing based on candidate labels and target modal information to flexibly adjust the amount of modal information data and improve the efficiency of label prediction processing.

[0053] In the aforementioned video tagging method, for the video to be processed, based on modal information extracted from at least two modalities, cross-retrieval is performed according to the feature dimensions of at least two modalities. At least one candidate tag for the video is determined based on the multimodal retrieval tags obtained from the at least two modalities. Tag prediction is then performed based on the at least one candidate tag and the modal information from at least two modalities to obtain the video tag. By performing cross-retrieval based on the modal information from at least two modalities of the video, determining at least one candidate tag based on the multimodal retrieval tags obtained from the cross-retrieval, and combining the candidate tag with the modal information from at least two modalities for tag prediction, the introduction of candidate tags enhances the information targeted by the tag prediction, thereby improving the accuracy of video tag determination.

[0054] In an exemplary embodiment, based on modal information of at least two modalities, cross-retrieval is performed according to the feature dimensions of at least two modalities to obtain multimodal retrieval tags for each of the at least two modalities. This includes: determining a multimodal feature tag library, which includes reference modal features of a reference video in at least two modalities and reference tags pre-determined for each reference modal feature; and performing feature matching between the modal information of at least two modalities and the reference modal features of the reference video in at least two modalities to obtain feature matching results, and retrieving multimodal retrieval tags for each of the at least two modalities from the multimodal feature tag library based on the feature matching results.

[0055] The multimodal feature label library stores various reference modal features and their corresponding reference labels. This library can include reference modal features of a reference video in at least two modalities, along with their respective reference labels. Reference videos are videos with pre-defined video labels. Reference modal features can be extracted from the reference video for each modality, such as visual reference modal features for visual modalities, text reference modal features for text modalities, or audio reference modal features for audio modalities. Each reference modal feature can be associated with a corresponding reference label, which is a pre-defined and added video label for the reference video. Feature matching results are obtained by matching the modal information of the video with the reference modal features of the reference video. Based on the feature matching results, the relevance between the video to be processed and the reference video can be determined. If the relevance between the video to be processed and the reference video is high, the reference labels carried by the reference video are more relevant to the labels of the video to be processed, and the reference labels carried by the reference video can be used as the retrieval labels for the video to be processed.

[0056] Optionally, the server can determine a pre-built multimodal feature tag library. In specific implementations, the server can use historical videos as reference videos and tag them, i.e., determine the reference tags corresponding to the reference videos. Furthermore, the server can extract reference modal features for different modalities from the reference videos. Specifically, feature extraction can be performed based on the modal information corresponding to different modalities in the reference videos, thereby obtaining reference modal features for different modalities. The server can establish associations between the reference tags of the reference videos and various reference modal features, thereby supporting retrieval based on the corresponding reference tags using various reference modal features. The server constructs a multimodal feature tag library based on the reference modal features of the reference videos in at least two modalities and the reference tags corresponding to each reference modal feature. For modal information of at least two modalities extracted from the video, the server determines the feature matching result between each modal information and the reference modal features of the reference videos in at least two modalities. The feature matching result is obtained by performing feature matching between the modal information and the reference modal features of the reference videos in at least two modalities. Based on the feature matching results, the server retrieves modality information for at least two modalities and their respective multimodal search tags from the multimodal feature tag library.

[0057] In a specific application, as shown in Figure 5, after the server extracts visual modal information, text modal information, and audio modal information from the video, it can perform cross-retrieval of these three modal information in a multimodal feature label library. Specifically, each type of visual modal information can be matched with various reference modal features in the multimodal feature label library to obtain feature matching results. The multimodal feature label library can include various reference modal features and their corresponding reference labels. The reference modal features can include visual reference modal features, text reference modal features, and audio reference modal features, and each reference modal feature has a corresponding reference label. For the obtained feature matching results, the server can determine the feature matching result that represents a match, determine the reference modal feature corresponding to the match result, and determine the reference label corresponding to the reference modal feature from the multimodal feature label library. The server can use the determined reference label as the multimodal retrieval label for the corresponding modal information. For visual modal information, text modal information, and audio modal information, multimodal retrieval label 1, multimodal retrieval label 2, and multimodal retrieval label 3 can be obtained respectively.

[0058] In this embodiment, the server retrieves the corresponding multimodal retrieval tag from the multimodal feature tag library based on the feature matching results between each modal information and each reference modal feature in the multimodal feature tag library. This allows for cross-retrieval of multimodal feature dimensions using the multimodal feature tag library, ensuring the accuracy of cross-retrieval and thus helping to ensure the accuracy of video tag determination.

[0059] In an exemplary embodiment, as shown in Figure 6, modal information of at least two modalities is respectively matched with reference modal features of the reference video in at least two modalities to obtain feature matching results. Based on the feature matching results, multimodal retrieval tags for each of the at least two modalities are retrieved from the multimodal feature tag library, including:

[0060] Step 602: Perform feature matching between each modal information in at least two modalities and the reference modal features of the reference video in at least two modalities to obtain the feature matching result for each modal information.

[0061] The feature matching result is obtained by matching modal information with reference modal features in a multimodal feature label library. For example, the server can traverse each modal information from at least two modalities to perform cross-referencing on each modal information. Specifically, for each modal information, the server can perform feature matching between the targeted modal information and each reference modal feature in the multimodal feature label library to obtain the feature matching result for the targeted modal information. In a concrete implementation, the server can extract modal information features from the targeted modal information and perform feature matching based on the extracted modal information features and each reference modal feature, such as performing feature similarity calculations to obtain the corresponding feature matching result. After traversing various modal information, feature matching results for various modal information can be obtained.

[0062] Step 604: From the feature matching results of each modality information, determine the target matching result for each modality information. The target matching result represents the feature matching result that matches the modality information.

[0063] In this context, the target matching result represents the feature matching result that indicates a match, meaning that the modal information matches the corresponding reference modal feature, and the correlation between the two is high. Optionally, for each type of modal information's feature matching result, the server can determine the target matching result that represents a match. In specific applications, the feature matching result may include feature similarity. The server can then judge each feature matching result based on a similarity threshold to determine whether the feature matching result represents a match. For example, feature matching results with a similarity exceeding the similarity threshold can be determined as target matching results that represent a match. By judging the feature matching results for each type of modal information separately, the target matching result for each type of modal information can be obtained.

[0064] Step 606: Determine the reference modal features corresponding to the target matching results of each modal information, and retrieve the multimodal retrieval tags of each modal information from the multimodal feature tag library based on the reference modal features corresponding to the target matching results.

[0065] Specifically, for each target matching result of each type of modality information, the server can determine the reference modality feature corresponding to the target matching result. The reference modality feature is obtained by feature matching with the modality information to get the reference modality feature of the target matching result. The server then retrieves the corresponding reference label from the multimodal feature label library. The server can use the retrieved reference label as the label retrieval result for that modality information. After traversing all target matching results of that type of modality information, the server can obtain the multimodal retrieval label for that modality information based on the various label retrieval results. After traversing various types of modality information, the server can obtain the multimodal retrieval labels for each type of modality information.

[0066] In this embodiment, the server performs feature matching on each modal information with each reference modal feature in the multimodal feature tag library. Based on the reference modal feature corresponding to the target matching result, the server retrieves the multimodal retrieval tag of the modal information from the multimodal feature tag library. After traversing each modal information, the server can obtain the multimodal retrieval tags of each modal information. Thus, based on feature matching, the server can accurately retrieve the multimodal retrieval tags of each modal information, which is beneficial to improving the accuracy of determining video tags based on multimodal retrieval tags.

[0067] In an exemplary embodiment, at least two modalities include a visual modality and a text modality; the modal information of the at least two modalities includes visual modal information and text modal information of the video; the multimodal feature tag library includes visual reference modal features of the reference video in the visual modality and visual reference tags corresponding to the visual reference modal features, as well as text reference modal features of the reference video in the text modality and text reference tags corresponding to the text reference modal features; feature matching is performed between the modal information of the at least two modalities and the reference modal features of the reference video in the at least two modalities to obtain feature matching results, and modal information of the at least two modalities is retrieved from the multimodal feature tag library based on the feature matching results. Each of the multimodal retrieval tags includes: extracting features based on visual modality information to obtain visual features, extracting features based on text modality information to obtain text features; performing visual feature matching between the visual features and visual reference modality features and text reference modality features respectively to obtain visual feature matching results, and obtaining multimodal retrieval tags for visual modality information from visual reference tags and text reference tags based on the visual feature matching results; and performing text feature matching between the text features and visual reference modality features and text reference modality features respectively to obtain text feature matching results, and obtaining multimodal retrieval tags for text modality information from visual reference tags and text reference tags based on the text feature matching results.

[0068] The multimodal feature library includes at least two modalities: a visual modality and a text modality. The modal information extracted from the video to be processed includes visual modal information corresponding to the visual modality and text modal information corresponding to the text modality. Visual features are video features extracted based on visual modal information, and text features are video features extracted based on text modal information. The multimodal feature label library includes visual reference modal features and their corresponding visual reference labels. Visual reference modal features are video features of the reference video in the visual modality. The multimodal feature label library also includes text reference modal features and their corresponding text reference labels. Text reference modal features are video features of the reference video in the text modality.

[0069] For example, in scenarios where the extracted modal information includes both visual and textual modal information, the modalities involved include both visual and textual modalities. The server can perform feature extraction based on the visual and textual modal information separately. For instance, a multimodal feature extraction model can be used to extract features from the visual and textual modal information separately to obtain visual features and textual features. In some embodiments, the server can also use different feature extraction models to extract features from the visual and textual modal information separately. For example, a visual feature extraction model can be used to extract visual features from the visual modal information, and a textual feature extraction model can be used to extract textual features from the textual modal information.

[0070] For visual features, the server can perform visual feature matching with visual reference modal features and text reference modal features in the multimodal feature label library, respectively. Specifically, it can calculate the feature similarity between the visual features and the visual reference modal features and text reference modal features separately to obtain the visual feature matching results. Based on the visual feature matching results, the server can determine the multimodal retrieval label for the visual modal information from the visual reference labels and text reference labels. Specifically, it can obtain the multimodal retrieval label based on the reference label corresponding to the visual feature matching result that matches the representation. For text features, similarly, the server can perform text feature matching with visual reference modal features and text reference modal features in the multimodal feature label library, respectively. Specifically, it can calculate the feature similarity between the text features and the visual reference modal features and text reference modal features separately to obtain the text feature matching results. Based on the text feature matching results, the server can determine the multimodal retrieval label for the text modal information from the visual reference labels and text reference labels. Specifically, it can obtain the multimodal retrieval label based on the reference label corresponding to the text feature matching result that matches the representation.

[0071] In a specific application, as shown in Figure 7, the video to be processed extracts visual modality information and text modality information separately. The server can perform feature extraction on the visual modality information and text modality information respectively to obtain visual features and text features. The server can perform visual feature matching in a multimodal feature label library. Specifically, it can perform visual feature matching with visual reference modality features and text reference modality features in the multimodal feature label library, and determine the multimodal retrieval label 1 of the visual modality information based on the obtained visual feature matching result. For text features, the server can perform text feature matching with visual reference modality features and text reference modality features in the multimodal feature label library, and determine the multimodal retrieval label 2 of the text modality information based on the obtained text feature matching result.

[0072] In this embodiment, in the application scenarios of text modality and visual modality, the server extracts visual features and text features based on visual modality information and text modality information, respectively. Then, it performs feature matching between the visual features and text features and the visual reference modality features and text reference modality features in the multimodal feature tag library, and determines the multimodal retrieval tags based on the corresponding feature matching results. This can accurately determine the multimodal retrieval tags for visual modality information and text modality information, which helps to ensure the accuracy of determining video tags.

[0073] In an exemplary embodiment, visual features are matched with visual reference modal features and text reference modal features respectively to obtain visual feature matching results. Multimodal retrieval tags for visual modal information are then obtained from visual reference tags and text reference tags based on the visual feature matching results. This includes: matching visual features with visual reference modal features to obtain a first visual feature matching result; determining the first visual feature matching result whose representation matches, determining the visual reference modal feature corresponding to the first visual feature matching result whose representation matches, and determining a first visual modal retrieval tag from visual reference tags based on the visual reference modal feature corresponding to the first visual feature matching result whose representation matches; matching visual features with text reference modal features to obtain a second visual feature matching result; determining the second visual feature matching result whose representation matches, determining the text reference modal feature corresponding to the second visual feature matching result whose representation matches, and determining a first text modal retrieval tag from text reference tags based on the text reference modal feature corresponding to the second visual feature matching result whose representation matches; and obtaining multimodal retrieval tags for visual modal information based on the first visual modal retrieval tag and the first text modal retrieval tag.

[0074] For example, for visual features, the server performs visual feature matching with each visual reference modal feature in the multimodal feature label library, specifically by calculating feature similarity, to obtain a first visual feature matching result. The server can determine the feature matching situation represented by each first visual feature matching result. The server determines the first visual feature matching result that represents a matching result, and further determines the visual reference modal feature corresponding to the first visual feature matching result that represents a matching result. That is, the visual reference modal feature matches the visual feature. Based on the visual reference modal feature, the server can determine the reference label corresponding to the visual reference modal feature from each visual reference label, and use the determined reference label as the first visual modal retrieval label. The server performs visual feature matching with each text reference modal feature in the multimodal feature label library, specifically by calculating feature similarity, to obtain a second visual feature matching result. The server determines the matching result of the second visual feature that matches the representation, and determines the text reference modality feature corresponding to the matching result of the second visual feature. That is, the text reference modality feature matches the visual feature. The server can determine the reference label corresponding to the text reference modality feature from each text reference label based on the text reference modality feature, and use the determined reference label as the first text modality retrieval label. The server obtains the multimodal retrieval label of visual modality information based on the first visual modality retrieval label and the first text modality retrieval label. Specifically, the first visual modality retrieval label and the first text modality retrieval label can be combined to obtain the multimodal retrieval label of visual modality information.

[0075] In this embodiment, the server performs visual feature matching with visual reference modal features and text reference modal features respectively, and determines the corresponding visual modal retrieval tags and text modal retrieval tags based on the visual feature matching results. Combining the visual modal retrieval tags and text modal retrieval tags can obtain multimodal retrieval tags for visual modal information. Tag retrieval based on the feature dimensions of visual and text modal features can ensure the accuracy of multimodal retrieval tags, thereby improving the accuracy of video tag determination.

[0076] In an exemplary embodiment, text features are matched with visual reference modal features and text reference modal features respectively to obtain text feature matching results. Multimodal retrieval tags for text modal information are then obtained from visual reference tags and text reference tags based on the text feature matching results. This includes: matching text features with visual reference modal features to obtain a first text feature matching result; determining the first text feature matching result with matching representations, determining the visual reference modal feature corresponding to the first text feature matching result with matching representations, and determining a second visual modal retrieval tag from visual reference tags based on the visual reference modal feature corresponding to the first text feature matching result with matching representations; matching text features with text reference modal features to obtain a second text feature matching result; determining the second text feature matching result with matching representations, determining the text reference modal feature corresponding to the second text feature matching result with matching representations, and determining a second text modal retrieval tag from text reference tags based on the text reference modal feature corresponding to the second text feature matching result with matching representations; and obtaining multimodal retrieval tags for text modal information based on the second visual modal retrieval tag and the second text modal retrieval tag.

[0077] Specifically, for text features, the server performs text feature matching with each visual reference modality feature in the multimodal feature label library, specifically calculating feature similarity to obtain a first text feature matching result. The server can determine the feature matching status represented by each first text feature matching result, identify the matching first text feature matching result, and determine the corresponding visual reference modality feature. Since this visual reference modality feature matches the text feature, the server can determine the corresponding reference label from each visual reference label based on this visual reference modality feature, and use the determined reference label as the second visual modality retrieval label. The server performs visual feature matching with each text reference modality feature in the multimodal feature label library, specifically calculating feature similarity to obtain a second text feature matching result. The server determines the matching result of the second text feature that matches the representation, and determines the text reference modality feature corresponding to the matching result of the second text feature. That is, the text reference modality feature matches the text feature. Based on the text reference modality feature, the server can determine the reference label corresponding to the text reference modality feature from each text reference label, and use the determined reference label as the second text modality retrieval label. The server obtains the multimodal retrieval label of the text modality information based on the second visual modality retrieval label and the second text modality retrieval label. Specifically, the second visual modality retrieval label and the second text modality retrieval label can be combined to obtain the multimodal retrieval label of the text modality information.

[0078] In this embodiment, the server performs text feature matching with both visual reference modal features and text reference modal features, and determines the corresponding visual modal retrieval tags and text modal retrieval tags based on the obtained text feature matching results. By combining the visual modal retrieval tags and text modal retrieval tags, multimodal retrieval tags for text modal information can be obtained. Tag retrieval based on the feature dimensions of visual and text modal features can ensure the accuracy of multimodal retrieval tags, thereby improving the accuracy of video tag determination.

[0079] In an exemplary embodiment, the steps of extracting features based on visual modality information to obtain visual features and extracting features based on text modality information to obtain text features are implemented through a multimodal feature extraction model. The training steps of the multimodal feature extraction model include: acquiring sample pairs, which include visual content and descriptive text for the visual content; extracting features for the visual content and descriptive text respectively through an initial multimodal feature extraction model to be trained to obtain sample pair visual features for the visual content and sample pair text features for the descriptive text; and updating the initial multimodal feature extraction model based on the sample pair visual features and sample pair text features to obtain the multimodal feature extraction model.

[0080] The multimodal feature extraction model supports feature extraction from information across multiple modalities. This model can be pre-trained based on various artificial neural network algorithms or deep learning algorithms, depending on actual needs. For example, the server can pre-train the multimodal feature extraction model using at least one of the following algorithms: Convolutional Neural Networks (CNN), CLIP (Contrastive Language-Image Pre-training), Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM), Generative Adversarial Networks (GAN), Autoencoder (AE), or Deep Belief Network (DBN).

[0081] Sample pairs are the sample data used to train the multimodal feature extraction model. Sample pairs can include visual content and descriptive text describing the visual content. Visual content can include image content or video content, and descriptive text describes the visual content. Sample pairs can include positive and negative pairs. Positive pairs are those where the visual content and descriptive text match, while negative pairs are those where the visual content and descriptive text do not match. The initial multimodal feature extraction model is the original model that needs to be trained.

[0082] Optionally, the server can acquire sample pairs and extract features from them using an initial multimodal feature extraction model to be trained. Specifically, the initial multimodal feature extraction model extracts features from visual content to obtain visual features of the sample pairs, and extracts features from descriptive text to obtain text features of the sample pairs. The server can update the initial multimodal feature extraction model based on the visual and text features output by the model and continue training. Specifically, the server can update the model parameters based on the matching degree between the visual and text features of the sample pairs and continue training with the next sample pair until the training termination condition is met, resulting in a trained multimodal feature extraction model. The trained multimodal feature extraction model extracts features from the input multimodal information and outputs features for the corresponding modality. The training termination condition can be flexibly set according to actual needs, such as reaching convergence, meeting model output accuracy, or reaching a specified number of training iterations.

[0083] In this embodiment, the server trains a multimodal feature extraction model using sample pairs including visual content and descriptive text for the visual content. The trained multimodal feature extraction model extracts features from the input multimodal information and outputs features under the corresponding modality. This strengthens the correlation between text modality features and visual modality features, ensures the accuracy of feature extraction by the multimodal feature extraction model, and thus helps improve the accuracy of video tag determination.

[0084] In an exemplary embodiment, feature extraction is performed on visual content and descriptive text using an initial multimodal feature extraction model to be trained, to obtain sample-pair visual features of visual content and sample-pair text features of descriptive text. This includes: when the visual content included in the sample pair belongs to video content, determining at least one video frame in the video content, performing visual feature extraction on at least one video frame using the initial multimodal feature extraction model to be trained, and obtaining sample-pair visual features of video content based on the visual features of at least one video frame; and determining the descriptive text included in the sample pair, performing text feature extraction on the descriptive text included in the sample pair using the initial multimodal feature extraction model to be trained, to obtain sample-pair text features of descriptive text.

[0085] The visual content included in the sample pair can be either video content or image content. That is, the sample pair can include image text pairs and video text pairs. The image text pair can include image content and descriptive text for the image content, and the video text pair can include video content and descriptive text for the video content.

[0086] For example, when the visual content included in the sample pair belongs to video content, i.e., when the sample pair belongs to video-text pairs, to ensure that the feature dimensions of the visual features remain consistent, the server can extract visual features from at least one video frame in the video content using an initial multimodal feature extraction model to obtain the visual features corresponding to at least one video frame. In specific applications, the server can sample video frames for the video content to obtain at least one video frame. For each sampled video frame, the server performs feature extraction separately using the initial multimodal feature extraction model to obtain the visual features of each video frame. The server obtains the visual features of the sample pair of video content based on the visual features of at least one video frame. In specific applications, when the number of at least one video frame is 1, the server can directly determine the visual features of the sample pair based on the visual features of a single video frame, such as directly using the visual features of a single video frame as the visual features of the sample pair; when the number of at least one video frame is greater than or equal to 2, the server can fuse the visual features of each video frame, such as by performing average pooling fusion, to obtain the visual features of the sample pair with the same feature dimensions as a single image. For the descriptive text in the video text pair, the server extracts text features from the descriptive text using an initial multimodal feature extraction model to obtain sample text features of the descriptive text.

[0087] In this embodiment, for sample pairs including video content, the server extracts visual features from at least one video frame in the video content and obtains the visual features of the sample pair based on the visual features of at least one video frame. By using the initial multimodal feature extraction model to extract text features from the descriptive text, it can be ensured that the visual features of the sample pairs of video content and the visual features of the sample pairs of image content have the same feature dimension. This unifies the model training based on video content and image content and ensures that the multimodal feature extraction model can accurately extract features of various modalities.

[0088] In an exemplary embodiment, at least two modalities include a visual modality and a text modality; the modal information of the at least two modalities includes visual modal information and text modal information; and tag prediction is performed based on at least one candidate tag and the modal information of the at least two modalities to obtain video tags for the video, including: concatenating the text modal information and at least one candidate tag to obtain concatenated text; and determining the visual features corresponding to the visual modal information, and performing tag prediction based on the concatenated text and the visual features corresponding to the visual modal information to obtain video tags for the video.

[0089] The modal information extracted from the video includes visual modal information corresponding to the visual modality and text modal information corresponding to the text modality. Candidate tags are in text form, and the concatenated text is obtained by combining the text modal information and the candidate tags.

[0090] Specifically, the server can concatenate text modal information and determined candidate tags to obtain concatenated text for the video. In practice, the server can concatenate the text modal information and candidate tags sequentially according to a preset concatenation order. The server then predicts tags based on the visual features corresponding to the concatenated text and visual modal information to obtain the video tags for the video. Visual features are obtained through feature extraction from the visual modal information, such as by using a pre-trained multimodal feature extraction model. In some embodiments, a pre-trained tag prediction model can be used to predict tags based on the concatenated text and visual features. Specifically, the concatenated text and visual features can be input into the tag prediction model for tag prediction, and the tag prediction model outputs the video tags for the video.

[0091] In this embodiment, the server can concatenate text modal information and candidate tags, and then perform tag prediction based on the visual features corresponding to the concatenated text and visual modal information. By combining candidate tags and modal information of at least two modalities, the information targeted by tag prediction is enhanced, thereby improving the accuracy of video tag determination.

[0092] In an exemplary embodiment, video tags for a video are obtained by predicting tags based on visual features corresponding to the concatenated text and visual modality information. This includes: performing feature alignment processing on the visual features corresponding to the visual modality information using a feature alignment model to obtain visual alignment features; and performing tag prediction based on the concatenated text and visual alignment features using a tag prediction model to obtain video tags for the video.

[0093] The feature alignment model is used to align visual features corresponding to visual modal information, mapping these visual features to features suitable for processing by the label prediction model. The feature alignment model can be pre-trained as needed. Visually aligned features are obtained by aligning visual features corresponding to visual modal information using the feature alignment model. These visually aligned features are suitable for input into the label prediction model, such as matching the feature dimension of the label prediction model with the dimension of its input features. The label prediction model is used to predict labels based on the input concatenated text and visual features. The label prediction model can be pre-trained based on various artificial neural network algorithms or deep learning algorithms as needed. For example, the server can pre-train the label prediction model based on at least one of the following algorithms: convolutional neural network, recurrent neural network, long short-term memory network, generative adversarial network, autoencoder, large language model, or deep belief network.

[0094] For example, the server can determine a pre-trained feature alignment model and perform feature alignment processing on the visual features corresponding to the visual modality information using the feature alignment model. Specifically, the visual features corresponding to the visual modality information can be input into the feature alignment model so that the feature alignment model can perform alignment mapping on the visual features to obtain visual alignment features. The server inputs the concatenated text and visual alignment features into a pre-trained label prediction model so that the label prediction model can perform label prediction and output video labels for the video.

[0095] In this embodiment, the server performs feature alignment processing on the visual features corresponding to the visual modality information using a feature alignment model, and then performs label prediction based on the spliced ​​text and visual alignment features using a label prediction model. By mapping the visual features, the normal label prediction processing of the label prediction model is ensured, thereby guaranteeing the accuracy of video label determination.

[0096] In an exemplary embodiment, the training steps of the feature alignment model and the label prediction model include: acquiring first sample visual features and sample description information for the first sample video, wherein the first sample visual features are extracted from the visual modal information of the first sample video; training an intermediate feature alignment model based on the first sample visual features and sample description information of the first sample video; acquiring second sample visual features, sample text modal information, and sample labels for the second sample video; wherein the second sample visual features are extracted from the visual modal information of the second sample video; performing feature alignment processing on the second sample visual features through the intermediate feature alignment model to obtain second sample aligned features; performing label prediction based on the sample text modal information and the second sample aligned features through the initial label prediction model to be trained to obtain predicted labels for the second sample video; and updating the intermediate feature alignment model and the initial label prediction model according to the predicted labels and sample labels to obtain the trained feature alignment model and label prediction model.

[0097] The intermediate feature alignment model is a feature alignment model initially trained based on the first sample visual features and sample description information of the first sample video. The first sample visual features are obtained by extracting visual features from the visual modal information of the first sample video, and the sample description information of the first sample video is used to describe the video content of the first sample video. The second sample video is used for joint training of the feature alignment model and the label prediction model; the second sample visual features are extracted based on the visual modal information of the second sample video. The sample text modal information is modal information extracted from the second sample video according to the text modality, specifically including but not limited to the title, subtitles, speech recognition results, character recognition results, etc., of the second sample video. The sample labels are tags pre-added to the second sample video.

[0098] Optionally, the server can acquire the first sample visual features and sample description information of the first sample video, and perform preliminary training based on the first sample visual features and sample description information to obtain an intermediate feature alignment model. The server acquires the second sample visual features, sample text modal information, and sample labels of the second sample video. The server inputs the second sample visual features into the pre-trained intermediate feature alignment model, which performs feature alignment processing and outputs the second sample aligned features. The server inputs the sample text modal information and the second sample aligned features into the initial label prediction model to be trained, which performs label prediction and outputs the predicted label for the second sample video. The server updates the intermediate feature alignment model and the initial label prediction model based on the predicted label and sample label, and then continues joint training with the next second sample video until the model training conditions are met, at which point training ends, resulting in a trained feature alignment model and label prediction model. In some embodiments, the server can determine the label difference between the predicted label and the sample label, and update the model parameters of the intermediate feature alignment model and the initial label prediction model respectively according to the label difference, and continue joint training until the training ends to obtain the trained feature alignment model and label prediction model.

[0099] In this embodiment, the server performs preliminary training on the intermediate feature alignment model and joint training on the intermediate feature alignment model and the initial label prediction model. This ensures the training effect of the feature alignment model and the label prediction model, and ensures the accuracy of video label determination based on the feature alignment model and the label prediction model.

[0100] In an exemplary embodiment, an intermediate feature alignment model is trained based on the first sample visual features of a first sample video and sample description information for the first sample video. This includes: performing feature alignment processing on the first sample visual features using an initial feature alignment model to be trained to obtain first sample aligned features; performing video description on the first sample aligned features using an initial label prediction model to be trained to obtain model description information for the first sample video; and updating the initial feature alignment model based on the model description information and sample description information, and continuing training until training is completed to obtain an intermediate feature alignment model.

[0101] The first sample video serves as the sample data for initial training of the feature alignment model. The first sample visual features are obtained by extracting visual features from the visual modal information of the first sample video, specifically by extracting features from the video frames within the first sample video. Sample description information is used to describe the first sample video, specifically describing its video content. The initial feature alignment model is the original feature alignment model to be trained, and the initial label prediction model is the original label prediction model to be trained.

[0102] For example, the server can determine a first sample video for initial training of the feature alignment model and obtain sample description information of the first sample video based on the first sample visual features. The server inputs the first sample visual features into the initial feature alignment model to be trained, so that the initial feature alignment model performs feature alignment processing and outputs the first sample aligned features. The server inputs the first sample aligned features into the initial label prediction model to be trained, so that the initial label prediction model performs video description and outputs model description information for the first sample video. The server updates the initial feature alignment model based on the model description information and the sample description information and continues training until training ends, obtaining an intermediate feature alignment model. In some embodiments, the server can determine the description information difference between the model description information and the sample description information, and update the model parameters in the initial feature alignment model based on the description information difference, and continue training based on the next first sample video until the model training conditions are met, at which point training ends, obtaining an intermediate feature alignment model.

[0103] In this embodiment, the feature alignment model is initially trained using the visual feature description information of the first sample video. This ensures that the feature alignment model can accurately align and map the input visual features, thereby ensuring the normal label prediction processing of the label prediction model.

[0104] In an exemplary embodiment, determining at least one candidate tag for a video based on the multimodal retrieval tags of at least two modal information includes: performing statistics on the multimodal retrieval tags of at least two modal information to obtain tag statistics results; and determining at least one candidate tag for the video based on the tag statistics results.

[0105] The tag statistics results are obtained by statistically analyzing the multimodal search tags. Specifically, the tag statistics results can include the frequency statistics of the multimodal search tags, that is, the tag statistics results record the frequency of each tag in the multimodal search tags.

[0106] Optionally, the server can perform statistics on the multimodal retrieval tags for each modal information, specifically frequency statistics on the tags, to obtain tag statistical results. Based on the tag statistical results, the server can determine at least one candidate tag for the video from the multimodal retrieval tags for each modal information. For example, when the tag statistical results include frequency statistics, the server can determine the K tags with the highest frequency statistics as candidate tags for the video. In some embodiments, when performing statistics on the multimodal retrieval tags for each modal information, different statistical weights can be set for different modal information according to actual needs, thereby combining the statistical weights to perform weighted statistics on the multimodal retrieval tags for each modal information to obtain tag statistical results.

[0107] In this embodiment, the server performs statistics on the multimodal retrieval tags for each type of modal information and determines candidate tags based on the tag statistics results. This can determine candidate tags by comprehensively considering the multimodal retrieval tags corresponding to the multimodal information, ensuring the accuracy of the candidate tags. When introducing candidate tags for tag prediction, the accuracy of video tag determination can be improved.

[0108] This application also provides an application scenario in which the above-described video tagging method is applied. Specifically, the video tagging method is applied in this scenario as follows:

[0109] Video tagging is a crucial part of video content feature recognition. By automatically generating tags for massive amounts of UGC (User-generated Content) videos, machines can provide downstream content distribution channels, such as recommendation systems and content operations, with video content features at different granularities. This improves content distribution efficiency while significantly reducing the cost of manual content review. Due to the diversity of UGC video content, the number of tags in the tag library used in business scenarios can reach hundreds of thousands or even millions, making it challenging to assign appropriate content tags to each video.

[0110] In existing technologies, label recognition methods can be broadly categorized into two types from a model implementation perspective: 1) Classification-based methods: The common approach is to label a batch of videos with their corresponding content tags, construct a training set to train a classification model, and then use the model to label new videos. The model typically fuses multimodal features and adds a fully connected layer (each neuron corresponds to a specific tag) on ​​top of the fused features to predict the existence of each tag; 2) Retrieval-based methods: Retrieval usually employs a v2v2t (video-to-video-to-tag) approach. Based on a trained encoder, features are extracted from the video. The manually labeled data is processed into a <video features, tag list> format and written into a retrieval database. For videos requiring tagging, features are first extracted using the same encoder. Videos with similar content are searched in the retrieval database, and the tag lists corresponding to the top-K (first K) similar videos are used as candidate tags. The final result is obtained through an aggregation module (e.g., voting).

[0111] Whether for classification or retrieval, the core algorithm module is the extraction and fusion of multimodal features. Current methods for tagging video sets basically reuse the multimodal content understanding framework used in short video scenarios: multiple modal information contained in the original video medium, such as video frame sequences, text information (video titles, OCR text, etc.), and ASR extracted from audio signals, are input into a multimodal model for feature extraction and fusion, finally predicting video content tags. The structure of the multimodal model can be that each modality has a separate encoder, extracting modal features from each and then fusing them; or the tokens (markers) from video frames after passing through a visual feature encoder can be fused and predicted together with text tokens through a transformer; or features from other modalities can be tokenized and input together with text into an LLM (Large Language Model) to generate prediction results.

[0112] The Transformer is a network architecture model suitable for processing sequential data. It employs a full attention structure instead of traditional recurrent neural networks (RNNs) or convolutional neural networks (CNNs). Its core idea is to model the relationship between each element in the input sequence and other elements as an attention weight matrix, and to compute the representation of each element through a self-attention mechanism. This structure gives the Transformer a strong ability to capture long-range dependencies in sequences, resulting in excellent performance in NLP tasks. The Transformer model has wide applications in NLP, including machine translation, text summarization, and question answering systems. It was the first to introduce the self-attention mechanism and surpasses traditional sequence-to-sequence (Seq2Seq) models, such as LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit), in many aspects. The success of Transformers has also spurred many variant models based on self-attention mechanisms, such as BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer), which have achieved significant progress in NLP tasks. A token refers to a word or sub-word in text and is the basic unit used by the model when processing text.

[0113] LLM (Language Learning Model) learns the statistical patterns and semantic information of language by training on large amounts of text data, thereby enabling it to predict the next word or sentence. LLM has wide applications in Natural Language Processing (NLP), such as machine translation, speech recognition, and text generation. LLM can employ different training strategies and model structures, such as pre-training and fine-tuning. Pre-training refers to training the model on a large scale under unsupervised or self-supervised conditions, enabling it to have a general understanding and expressive ability for language data. Fine-tuning, on the other hand, optimizes the model for specific application scenarios or tasks through supervised learning based on the pre-trained model. LLM has achieved significant results in language understanding, generation, and translation tasks, and its performance and application scope continue to expand with the continuous increase in model size and training data.

[0114] As shown in Figure 8, regardless of the structure of the multimodal model, the video label recognition process can be abstracted as follows: extracting multimodal information from the video, including video frames, titles, Optical Character Recognition (OCR) results, Automatic Speech Recognition (ASR) results, etc., and then using the multimodal model to predict labels based on the multimodal information of the video to obtain the label results. Current multimodal large-scale model methods mainly implement question answering for general vision or NLP tasks. Among these, OCR is a technology for detecting and recognizing text content from images, while ASR is a technology for converting human speech into text.

[0115] Existing technologies have several drawbacks: classification-based methods can only support a closed set of labels defined during training. If labels need to be added or removed, the entire model needs to be retrained, which brings time and manpower costs in terms of data annotation and model training. Retrieval-based methods can flexibly support new labels, which can be achieved by refreshing the retrieval database without updating the model. However, the performance of such methods is usually limited by the effect of multimodal features. On the other hand, they cannot cover labels that do not exist in the retrieval database.

[0116] MLLM (Multimodal Large Language Model) methods are typically designed as general-purpose models that simultaneously support multiple perceptual and cognitive tasks in a question-and-answer format. MLLM, building upon LLM, integrates media data from other modalities, such as images, videos, and audio, enabling the model to process information from different modalities simultaneously, better understand and express semantics, and thus improve the effectiveness and accuracy of applications. These methods can be directly applied to tag recognition tasks, but for tag generation tasks, without providing any candidate tags, directly training MLLM to generate tags from scratch yields significantly less effective results than being guided by some candidate tags. TagGPT (Tag Generation with GPT) employs a similar approach. It first uses chatGPT (Chat Generative Pre-trained Transformer) to generate and post-process a candidate tag library. The generated results are matched with the candidate tag library, controlling the generated content to some extent. However, TagGPT uses object detection, OCR, and other methods to recognize video content and convert it all to text. Subsequent model processing in plain text form theoretically results in the loss of a significant amount of information.

[0117] To address the above issues, this embodiment provides a video tagging method based on a multimodal large-scale model with retrieval enhancement. This method, grounded in the MLLM framework, uses a simple cross-retrieval module to obtain candidate tags, which are then input into the large-scale model to enhance input features and guide the generation task, significantly improving the quality of the generated tags. Furthermore, the two input features required for cross-retrieval—visual features derived from the MLLM alignment module and requiring no additional computation, and the feature encoder required for text feature calculation derived from the CLIP (Contrastive Language-Image Pre-training) training of the visual encoder and also requiring no separate training—result in lightweight overall computation. As shown in Figure 9, for video frames, visual features are encoded using the visual feature encoder in the feature extraction model, and then aligned using the alignment module before being input into the multimodal large-scale language model (MLLM). For video text, the text is input into the MLLM, and the video text may include the video title, optical character recognition results, automatic speech recognition results, etc. On the other hand, the video text obtains text features through the text feature encoder in the feature extraction model. Based on the visual features and text features, cross-retrieval is performed to obtain some candidate tags. The candidate tags and question prompt words are input into the multimodal large language model. The multimodal large language model generates video-specific tag results based on the visual features processed by the alignment module, the video text, and the candidate tags.

[0118] The video tagging method provided in this embodiment can be applied in content distribution scenarios. As shown in Figure 10, a video enters the content processing chain from the content production stage, obtains corresponding content features through human-machine collaboration, and then enters the downstream content distribution stage. The video tagging method provided in this embodiment belongs to the machine tagging stage in video content processing. Specifically, in the video content processing process, video tagging can be performed based on a collaborative method of machine tagging and manual tagging, that is, adding corresponding video tags to the video.

[0119] The video tagging method provided in this embodiment can address various video tag recognition tasks, involving a retrieval-enhanced multimodal large-scale model approach. This method defines tag recognition as a text generation task, obtaining tag results end-to-end through multimodal large-scale language model decoding. Furthermore, it incorporates a lightweight retrieval enhancement module, pre-obtaining some candidate tag enhancement features to guide the generation task, significantly improving the tag generation performance.

[0120] For the visual feature encoder and text feature encoder included in the feature extraction model, this embodiment uses CLIP to train the visual feature encoder and text feature encoder. The CLIP algorithm is a multimodal learning technique that aims to train a model capable of understanding and processing image and text data. CLIP achieves this goal by simultaneously training text and image models and utilizing contrastive learning to improve model performance. In the CLIP algorithm, the text and image models are trained with a large number of images and related descriptions. The model learns to map text and image representations to the same semantic space, so that text and images with similar meanings have similar representations in that space. For example, the model can associate the representation of the word "dog" with the representation of a picture of a dog. Contrastive learning plays a crucial role in the CLIP algorithm. During training, the model needs to distinguish between positive examples (i.e., correct descriptions that match the image) and negative examples (i.e., descriptions that do not match the image). The model achieves this goal by maximizing the similarity between positive examples and minimizing the similarity between negative examples, a method that enables the model to learn more robust and generalized representations.

[0121] As shown in Figure 11, for an image and its descriptive text, a visual feature encoder extracts visual features I from the image, and a text feature encoder extracts text features T from the descriptive text. A graph-text feature pair (I1) is then constructed between visual features I and text features T. i ·T i For each image-text feature pair, the CLIP model aims to maximize the similarity between them while minimizing the similarity to other unrelated image and text pairs. Specifically, the CLIP algorithm uses the InfoNCE (Information Noise Contrastive Estimation) loss function to measure the similarity between image-text feature pairs (I...). i ·T i The similarity between two positive sample images (x and x). i and text y i Given K negative sample images and text, the InfoNCE loss function is defined as follows:

[0122] Among them, L CLIP For model loss, τ is a temperature hyperparameter used to control the distribution of similarity; exp() is the power operation with the natural constant as the base; v i and t i Representing the image x i and text y i eigenvectors.

[0123] This embodiment uses two types of data for training: image-text pairs and video-text pairs. Business data naturally provides data such as image-description text and video-video title pairs, eliminating the need for manual annotation. The visual encoder uses ViT (Vision Transformer), and the text encoder uses BERT. Furthermore, the encoder structure can be replaced with other network structures.

[0124] During training, this embodiment modifies the CLIP training process to adapt to two different media formats: images and videos. For image data, it remains consistent with CLIP; however, for video data, it adopts a sampling method similar to TSN (Temporal Segment Networks). This involves first extracting frames from a video at 1 frame per second, then dividing it into N equal segments. In training mode, one frame is randomly selected from each segment; in testing mode, the middle frame from each segment is selected, resulting in a total of N frames. These N frames are then encoded using ViT and subjected to average pooling to obtain feature dimensions consistent with those of the images. This unifies the training methods for image-text pairs and view-text pairs. After training, a visual feature encoder (denoted as E) is obtained. v It will be used by MLLM to extract visual features, and will also produce a byproduct, a text feature encoder (denoted as E). t This will be used in the cross-referencing module to extract text features.

[0125] For the cross-retrieval module, the retrieval method is based on the video-to-video-to-tag approach, thus requiring a multimodal feature fusion video embedding (embedded feature, hereinafter referred to as EMB) to search for similar videos. This embodiment further follows the design principle of minimizing additional computational modules, avoiding the use of video EMBs to search for similar videos. Instead, it performs a cross-search based on the visual and textual EMBs of the videos. The visual and textual EMBs trained using CLIP are mapped to the same semantic space, ensuring that texts and images with similar meanings have similar representations in this space. Therefore, if video a and video b have similar content, the distances between the four EMBs (visual and textual EMBs) of videos a and b are close. Thus, a video's visual EMB (or textual EMB) can be used to search for similar visual or textual EMBs to find similar videos, and the corresponding tag results of these similar videos are then used as candidates.

[0126] Cross-referencing can be implemented using the following steps:

[0127] Step 1: Offline processing stage. The training set <videos, human review label list> is processed into <visual features, human review label list> and <text features, human review label list>. Two retrieval databases are established, including a visual feature label retrieval database and a text feature label retrieval database. The retrieval databases can be implemented using frameworks such as FAISS (Facebook AI Similarity Search).

[0128] Step 2: In the online processing stage, visual features v and text features t are extracted from the video to be processed. Then, searches are performed on two retrieval databases to retrieve a list of tags corresponding to videos with similarity > θ, where θ is a hyperparameter. This yields a 4-way recall (visual feature v searches for similar visual features and text features, i.e., v-to-v and v-to-t; similarly, text feature t performs t-to-v and t-to-t) tag results.

[0129] Step 3: In the online processing stage, the tag results from the above four recall channels are merged, their frequencies are counted, and the top-K tags (where K is a hyperparameter) are taken as the final result. It should be noted that the tag results here only serve as a guiding signal input for MLLM. The value of K can be appropriately increased to improve the recall rate, allowing MLLM to select the correct tags.

[0130] As shown in Figure 12, the video is processed by both a visual feature encoder and a text feature encoder to extract visual features v and text features t. Cross-referencing is then performed based on visual features v and text features t, specifically targeting visual feature retrieval libraries and text feature retrieval libraries respectively. Specifically, visual feature v searches for similar visual and text features, i.e., v-to-v (v2v) and v-to-t (v2t); similarly, text feature t undergoes t-to-v (t2v) and t-to-t (t2t) searches. Labeling is then performed on the obtained four-way cross-referencing results, with the top-K labels selected as candidate labels. In some embodiments, weights can be introduced. Specifically, an offline validation set can be defined. The four retrieval paths (v2v, t2v, t2t, v2t) can also yield labels individually. Weights are set based on the performance of each path on the validation set; paths with high precision and recall receive greater weight, while others receive only 1 / 4 weight, i.e., no weighting is applied.

[0131] For retrieval-enhanced MLLM, training the multimodal model can be implemented using MLLM. The MLLM model receives two types of input: visual feature tokens and text. The visual feature tokens are obtained through processing by a visual feature encoder, which is trained using the CLIP method. To align with the LLM input, an alignment module P is appended to the visual token. This module contains a learnable parameter matrix, primarily responsible for mapping the visual features to the LLM input dimension and converting them into tokens suitable for the LLM, denoted as [video_emb]. The text includes the video title, description text, ASR, etc. Additionally, some candidate tags recalled by the cross-retrieval module serve as guiding signals; these are also in plain text form, represented as "Video partial candidate tags are: xxx, xxx, xxx, ...", concatenated with the video text, denoted as [video_txt].

[0132] The LLM can be an open-source Chinese LLM, such as chinese-llama or BLOOM. The input prompt template is:

[0133] "###Human:[video_emb][video_txt][instruct]###Assistant:"

[0134] LLM generates the label results obtained from inference based on this input.

[0135] Among them, "###Human:" and "###Assistant:" are commonly used training templates to help the machine understand that "Human" is followed by instructions, and "Assistant" means to start answering questions. [video_emb], [video_txt], and [instruct] are the video visual token, video text token, and instruction text token, respectively.

[0136] Furthermore, the training of MLLM is divided into two stages: stage one is pre-training for the feature alignment module, and stage two is fine-tuning training for the feature alignment module and MLLM instructions.

[0137] In Phase 1, the pre-training of the feature alignment module fixes all parameters of the visual feature encoder and LLM, training only the alignment module P. The main goal is to align visual tokens to the LLM. The training data for this phase uses <video, video description text> pairs. Here, we continue to use the <short video, video title> pairs from the business scenario to generate a brief description of the video content to train the target model, using the following prompts:

[0138] "###Human:[video_emb][alignment directive]###Assistant:"

[0139] The alignment instructions can vary, and may include at least one of the following statements: "Briefly describe the following video." "Provide a brief description of the given video clip." "Concisely explain the provided video clip." "Summarize the visual content of the following video." "Provide a brief and clear explanation of the following video clip." "Briefly explain the meaning of the provided video." "Briefly describe the key features of the clip." "Briefly describe the content of the presented video." "Provide a clear and concise summary of the following video." "Write an informative summary of the following video clip." "Present the provided video in a concise narrative style."

[0140] For Phase Two, which involves fine-tuning the instructions for the feature alignment module and MLLM, the parameters of the visual feature encoder are fixed, while the parameters of the alignment module and LLM are unrestricted during training. The training objective of this phase is to construct corresponding instructions based on <video, tag> data from real-world business scenarios, and to fine-tune these instructions using the generated tag list text as the target: "###Human:[video_emb][video_txt][instruct]###Assistant:"

[0141] Similarly, like the alignment instructions in Phase 1, `instruct` can also be designed with synonyms to improve training performance. Examples include: "What tags can be used to summarize the above videos?", "Based on the content of the above videos, output some key content tags.", and "Based on the content of the above videos, generate the main content tags."

[0142] In this embodiment, the text feature encoder, a byproduct of CLIP, is only used by the cross-retrieval module. Its parameters remain unchanged from those of the visual feature encoder and it is not used in training. Meanwhile, based on the business scenario, the background music in short videos is often the same popular background music, which does not have good discriminative power. Furthermore, the important spoken information has already been extracted and merged into the video text information through ASR. Therefore, the model designed in this embodiment does not use the original audio signal as one of the inputs.

[0143] Further, model-based inference processing is performed. After model training, labels can be predicted for new videos. N frames of video images are extracted, and N features are obtained through a visual feature encoder. These features are then transformed by an alignment module to obtain the input visual token. These N features are averaged and pooled, and then cross-referenced with the text information (including title, ASR text, etc.) after text feature encoding to obtain a subset of candidate labels. These candidate labels are then concatenated with the text information and question instructions to obtain the text token. The visual and text tokens are then input into the LLM for decoding to generate the label results.

[0144] In other embodiments, the visual feature encoder and text feature encoder can be replaced with networks of other structures; the alignment module, which aligns visual features to the input of the LLM, can also be replaced with other structures, such as q-former; the main process of this embodiment can also be changed by replacing the LLM framework with other generative models; the retrieval module can also be modified to other similar implementations.

[0145] The video tagging method provided in this embodiment is based on the general MLLM framework and proposes a retrieval-enhanced MLLM. The implementation method is simple and cleverly utilizes the intermediate results of MLLM to design a cross-retrieval module, recalling a portion of candidate tags as guiding signals to be input into the large model, enhancing the input features, and guiding the model to generate better tag results.

[0146] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0147] Based on the same inventive concept, this application also provides a video tag processing apparatus for implementing the video tag processing method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more video tag processing apparatus embodiments provided below can be found in the limitations of the video tag processing method described above, and will not be repeated here.

[0148] In an exemplary embodiment, as shown in FIG13, a video tag processing device 1300 is provided, including: a modal information extraction module 1302, a cross-referencing module 1304, a candidate tag determination module 1306, and a tag prediction module 1308, wherein:

[0149] The modal information extraction module 1302 is used to determine the video to be processed and extract modal information of at least two modalities from the video;

[0150] The cross-retrieval module 1304 is used to perform cross-retrieval based on modal information of at least two modalities, according to the feature dimensions of at least two modalities, to obtain multimodal retrieval tags for each of the modal information of at least two modalities.

[0151] The candidate tag determination module 1306 is used to determine at least one candidate tag for the video based on the multimodal retrieval tags of at least two modalities; and

[0152] The tag prediction module 1308 is used to predict tags based on at least one candidate tag and modal information of at least two modalities to obtain video tags for the video.

[0153] In one embodiment, the cross-retrieval module 1304 is further configured to determine a multimodal feature tag library, which includes reference modal features of the reference video in at least two modalities and reference tags pre-determined for each reference modal feature; and to perform feature matching between the modal information of at least two modalities and the reference modal features of the reference video in at least two modalities to obtain feature matching results, and to retrieve the multimodal retrieval tags of the modal information of at least two modalities from the multimodal feature tag library based on the feature matching results.

[0154] In one embodiment, the cross-retrieval module 1304 is further configured to perform feature matching on each modal information in at least two modalities with the reference modal features of the reference video in at least two modalities to obtain a feature matching result for each modal information; determine the target matching result for each modal information from the feature matching result for each modal information, wherein the target matching result represents the matching feature matching result; and determine the reference modal features corresponding to the target matching result for each modal information, and retrieve the multimodal retrieval tags for each modal information from the multimodal feature tag library based on the reference modal features corresponding to the target matching result.

[0155] In one embodiment, at least two modalities include a visual modality and a text modality; the modal information of at least two modalities includes visual modal information of the video and text modal information of the video; the multimodal feature tag library includes visual reference modal features of the reference video in the visual modality and visual reference tags corresponding to the visual reference modal features, and text reference modal features of the reference video in the text modality and text reference tags corresponding to the text reference modal features; the cross-retrieval module 1304 is further configured to perform feature extraction based on visual modal information to obtain visual features, and perform feature extraction based on text modal information to obtain text features; perform visual feature matching between the visual features and visual reference modal features and text reference modal features respectively to obtain visual feature matching results, and obtain multimodal retrieval tags for visual modal information from visual reference tags and text reference tags according to the visual feature matching results; and perform text feature matching between the text features and visual reference modal features and text reference modal features respectively to obtain text feature matching results, and obtain multimodal retrieval tags for text modal information from visual reference tags and text reference tags according to the text feature matching results.

[0156] In one embodiment, the cross-retrieval module 1304 is further configured to: perform visual feature matching with visual reference modal features to obtain a first visual feature matching result; determine the first visual feature matching result whose representation matches, determine the visual reference modal feature corresponding to the first visual feature matching result whose representation matches, and determine a first visual modal retrieval label from visual reference labels based on the visual reference modal feature corresponding to the first visual feature matching result whose representation matches; perform visual feature matching with text reference modal features to obtain a second visual feature matching result; determine the second visual feature matching result whose representation matches, determine the text reference modal feature corresponding to the second visual feature matching result whose representation matches, and determine a first text modal retrieval label from text reference labels based on the text reference modal feature corresponding to the second visual feature matching result whose representation matches; and obtain a multimodal retrieval label for visual modal information based on the first visual modal retrieval label and the first text modal retrieval label.

[0157] In one embodiment, the cross-retrieval module 1304 is further configured to perform text feature matching with visual reference modal features to obtain a first text feature matching result; determine the first text feature matching result whose representation matches, determine the visual reference modal feature corresponding to the first text feature matching result whose representation matches, and determine a second visual modal retrieval label from visual reference labels based on the visual reference modal feature corresponding to the first text feature matching result whose representation matches; perform text feature matching with text reference modal features to obtain a second text feature matching result; determine the second text feature matching result whose representation matches, determine the text reference modal feature corresponding to the second text feature matching result whose representation matches, and determine a second text modal retrieval label from text reference labels based on the text reference modal feature corresponding to the second text feature matching result whose representation matches; and obtain a multimodal retrieval label for text modal information based on the second visual modal retrieval label and the second text modal retrieval label.

[0158] In one embodiment, the steps of extracting features based on visual modality information to obtain visual features and extracting features based on text modality information to obtain text features are implemented through a multimodal feature extraction model. The model also includes a feature extraction model training module for acquiring sample pairs, which include visual content and descriptive text for the visual content; extracting features from the visual content and descriptive text using an initial multimodal feature extraction model to be trained, obtaining sample pair visual features for the visual content and sample pair text features for the descriptive text; and updating the initial multimodal feature extraction model based on the sample pair visual features and sample pair text features to obtain the multimodal feature extraction model.

[0159] In one embodiment, the feature extraction model training module is further configured to: when the visual content included in the sample pair belongs to video content, determine at least one video frame in the video content, extract visual features for the at least one video frame using the initial multimodal feature extraction model to be trained, and obtain the visual features of the sample pair of video content based on the visual features of the at least one video frame; and determine the descriptive text included in the sample pair, extract text features for the descriptive text included in the sample pair using the initial multimodal feature extraction model to be trained, and obtain the text features of the sample pair of descriptive text.

[0160] In one embodiment, at least two modalities include a visual modality and a text modality; the modal information of at least two modalities includes visual modal information and text modal information; the label prediction module 1308 is further configured to concatenate the text modal information and at least one candidate label to obtain concatenated text; and determine the visual features corresponding to the visual modal information, and perform label prediction based on the concatenated text and the visual features corresponding to the visual modal information to obtain video labels for the video.

[0161] In one embodiment, the label prediction module 1308 is further configured to perform feature alignment processing on the visual features corresponding to the visual modality information through a feature alignment model to obtain visual alignment features; and to perform label prediction based on the spliced ​​text and visual alignment features through a label prediction model to obtain video labels for the video.

[0162] In one embodiment, the system further includes a joint training module for acquiring first sample visual features and sample description information for the first sample video, wherein the first sample visual features are extracted from the visual modal information of the first sample video; training an intermediate feature alignment model based on the first sample visual features and sample description information of the first sample video; acquiring second sample visual features, sample text modal information, and sample labels for the second sample video; wherein the second sample visual features are extracted from the visual modal information of the second sample video; performing feature alignment processing on the second sample visual features through the intermediate feature alignment model to obtain second sample aligned features; performing label prediction based on the sample text modal information and second sample aligned features through an initial label prediction model to be trained to obtain predicted labels for the second sample video; and updating the intermediate feature alignment model and the initial label prediction model according to the predicted labels and sample labels to obtain the trained feature alignment model and label prediction model.

[0163] In one embodiment, the joint training module is further configured to perform feature alignment processing on the visual features of the first sample using the initial feature alignment model to be trained, to obtain the first sample alignment features; perform video description on the first sample alignment features using the initial label prediction model to be trained, to obtain model description information for the first sample video; and update the initial feature alignment model based on the model description information and the sample description information and continue training until the training ends, to obtain an intermediate feature alignment model.

[0164] In one embodiment, the candidate tag determination module 1306 is further configured to perform statistics on the multimodal retrieval tags for each of the modal information of at least two modalities to obtain tag statistics results; and determine at least one candidate tag for the video based on the tag statistics results.

[0165] Each module in the aforementioned video tag processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0166] In an exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as shown in Figure 14. The computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is connected to the system bus via the I / O interfaces. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer-readable instructions, and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the non-volatile storage medium. The database of the computer device stores video tag processing data. The I / O interfaces of the computer device are used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communicating with external terminals via a network connection. When the computer-readable instructions are executed by the processor, a video tag processing method is implemented.

[0167] Those skilled in the art will understand that the structure shown in Figure 14 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or may combine certain components, or may have different component arrangements.

[0168] In one embodiment, a computer device is also provided, including a memory and a processor, the memory storing computer-readable instructions, the processor executing the computer-readable instructions to implement the steps in the above method embodiments.

[0169] In one embodiment, a computer-readable storage medium is provided storing computer-readable instructions that, when executed by a processor, implement the steps in the above method embodiments.

[0170] In one embodiment, a computer program product is provided, including computer-readable instructions that, when executed by a processor, implement the steps in the above method embodiments.

[0171] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0172] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a non-volatile computer-readable storage medium. When executed, these computer-readable instructions can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0173] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0174] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A video tagging method, executed by a computer device, the method comprising: The video to be processed is identified, and modal information of at least two modalities is extracted from the video. Based on the modal information of the at least two modalities, cross-retrieval is performed according to the feature dimensions of the at least two modalities to obtain the multimodal retrieval tags for the modal information of the at least two modalities respectively; Based on the multimodal retrieval tags of the modal information of the at least two modalities, at least one candidate tag is determined for the video; and Based on the modal information of the at least one candidate label and the at least two modalities, label prediction is performed to obtain video labels for the video.

2. The method according to claim 1, wherein the step of performing cross-retrieval based on the modal information of the at least two modalities according to the feature dimensions of the at least two modalities to obtain multimodal retrieval tags for the modal information of the at least two modalities includes: A multimodal feature tag library is determined, wherein the multimodal feature tag library includes reference modal features of the reference video in the at least two modalities and reference tags that are pre-determined for each of the reference modal features; and The modal information of the at least two modalities is respectively matched with the reference modal features of the reference video in the at least two modalities to obtain the feature matching results. Based on the feature matching results, the multimodal retrieval tags of the modal information of the at least two modalities are retrieved from the multimodal feature tag library.

3. The method according to claim 2, wherein the step of performing feature matching between the modal information of the at least two modalities and the reference modal features of the reference video in the at least two modalities to obtain feature matching results, and retrieving multimodal retrieval tags for the modal information of the at least two modalities from the multimodal feature tag library based on the feature matching results, includes: Each modal information in the modal information of the at least two modalities is matched with the reference modal features of the reference video in the at least two modalities to obtain the feature matching result of each modal information; From the feature matching results of each modality information, determine the target matching result for each modality information, wherein the target matching result represents the feature matching result that matches the modality information. and Determine the reference modal features corresponding to the target matching results of each modal information, and retrieve the multimodal retrieval tags for each modal information from the multimodal feature tag library based on the reference modal features corresponding to the target matching results.

4. The method according to claim 2, wherein the at least two modalities include a visual modality and a text modality; the modal information of the at least two modalities includes the visual modal information of the video and the text modal information of the video; the multimodal feature tag library includes visual reference modal features of the reference video in the visual modality and visual reference tags corresponding to the visual reference modal features, and text reference modal features of the reference video in the text modality and text reference tags corresponding to the text reference modal features; The step of performing feature matching between the modal information of the at least two modalities and the reference modal features of the reference video in the at least two modalities to obtain feature matching results, and retrieving multimodal retrieval tags for the modal information of the at least two modalities from the multimodal feature tag library based on the feature matching results, includes: Visual features are obtained by extracting features based on the visual modality information, and text features are obtained by extracting features based on the text modality information. The visual features are matched with the visual reference modal features and the text reference modal features respectively to obtain visual feature matching results. Based on the visual feature matching results, multimodal retrieval tags for the visual modal information are obtained from the visual reference tags and the text reference tags. The text features are matched with the visual reference modal features and the text reference modal features respectively to obtain text feature matching results. Based on the text feature matching results, the multimodal retrieval labels of the text modal information are obtained from the visual reference labels and the text reference labels.

5. The method according to claim 4, wherein the step of performing visual feature matching with the visual reference modal feature and the text reference modal feature respectively to obtain a visual feature matching result, and obtaining a multimodal retrieval label for the visual modal information from the visual reference label and the text reference label based on the visual feature matching result, comprises: The visual features are matched with the visual reference modality features to obtain a first visual feature matching result. Determine the first visual feature matching result that matches the representation, determine the visual reference modal feature corresponding to the first visual feature matching result that matches the representation, and determine the first visual modal retrieval label from the visual reference label based on the visual reference modal feature corresponding to the first visual feature matching result that matches the representation; The visual features are matched with the text reference modality features to obtain a second visual feature matching result. Determine the matching result of the second visual feature that matches the representation, determine the text reference modality feature corresponding to the matching result of the second visual feature that matches the representation, and determine the first text modality retrieval tag from the text reference tag based on the text reference modality feature corresponding to the matching result of the second visual feature that matches the representation; and Based on the first visual modality retrieval tag and the first text modality retrieval tag, the multimodal retrieval tag of the visual modality information is obtained.

6. The method according to claim 4, wherein the step of performing text feature matching between the text features and the visual reference modality features and the text reference modality features respectively to obtain text feature matching results, and obtaining multimodal retrieval labels for the text modality information from the visual reference labels and the text reference labels based on the text feature matching results, comprises: The text features are matched with the visual reference modality features to obtain the first text feature matching result; Determine a first text feature matching result that matches the representation, determine the visual reference modal feature corresponding to the first text feature matching result that matches the representation, and determine a second visual modal retrieval label from the visual reference label based on the visual reference modal feature corresponding to the first text feature matching result that matches the representation; The text features are matched with the text reference modality features to obtain a second text feature matching result; Determine the second text feature matching result that matches the representation, determine the text reference modality feature corresponding to the second text feature matching result that matches the representation, and determine the second text modality retrieval tag from the text reference tag based on the text reference modality feature corresponding to the second text feature matching result that matches the representation; and Based on the second visual modality retrieval tag and the second text modality retrieval tag, the multimodal retrieval tag of the text modality information is obtained.

7. The method according to claim 4, wherein the steps of extracting features based on the visual modality information to obtain visual features and extracting features based on the text modality information to obtain text features are implemented by a multimodal feature extraction model, wherein the training steps of the multimodal feature extraction model include: Obtain sample pairs, wherein the sample pairs include visual content and descriptive text for the visual content; The initial multimodal feature extraction model to be trained is used to extract features from the visual content and the descriptive text respectively, resulting in sample pairs of visual features of the visual content and sample pairs of text features of the descriptive text; and Based on the visual features of the sample pairs and the text features of the sample pairs, the initial multimodal feature extraction model is updated to obtain the multimodal feature extraction model.

8. The method according to claim 7, wherein the step of extracting features from the visual content and the descriptive text using an initial multimodal feature extraction model to be trained, to obtain sample-pair visual features of the visual content and sample-pair text features of the descriptive text, comprises: When the visual content included in the sample pair belongs to video content, at least one video frame in the video content is identified. Visual features are extracted from the at least one video frame using an initial multimodal feature extraction model to be trained, and the visual features of the sample pair of the video content are obtained based on the visual features of the at least one video frame. The descriptive text included in the sample pair is determined, and the text features of the descriptive text included in the sample pair are extracted using the initial multimodal feature extraction model to be trained, so as to obtain the sample pair text features of the descriptive text.

9. The method according to any one of claims 1 to 8, wherein the at least two modalities include a visual modality and a text modality; and the modal information of the at least two modalities includes visual modal information and text modal information; The step of predicting video tags for the video based on the modal information of the at least one candidate tag and the at least two modalities includes: The text modal information and the at least one candidate tag are concatenated to obtain the concatenated text; and The visual features corresponding to the visual modality information are determined, and the video tags for the video are obtained based on the concatenated text and the visual features corresponding to the visual modality information.

10. The method according to claim 9, wherein the step of predicting tags based on visual features corresponding to the concatenated text and the visual modality information to obtain video tags for the video includes: The visual features corresponding to the visual modality information are aligned using a feature alignment model to obtain visual aligned features. and The video tags for the video are obtained by using a tag prediction model to predict tags based on the spliced ​​text and the visual alignment features.

11. The method according to claim 10, wherein the training steps of the feature alignment model and the label prediction model include: The first sample visual features and sample description information for the first sample video are obtained. The first sample visual features are extracted from the visual modal information of the first sample video. An intermediate feature alignment model is trained based on the first sample visual features of the first sample video and the sample description information for the first sample video. The second sample visual features, sample text modal information, and sample labels of the second sample video are obtained. The second sample visual features are extracted from the visual modal information of the second sample video. The visual features of the second sample are aligned using the intermediate feature alignment model to obtain the aligned features of the second sample. The initial label prediction model to be trained performs label prediction based on the sample text modal information and the second sample alignment features to obtain the predicted label for the second sample video. and The intermediate feature alignment model and the initial label prediction model are updated based on the predicted label and the sample label to obtain the trained feature alignment model and label prediction model.

12. The method according to claim 11, wherein training an intermediate feature alignment model based on the first sample visual features of the first sample video and sample description information for the first sample video comprises: The visual features of the first sample are aligned using the initial feature alignment model to be trained, and the aligned features of the first sample are obtained. The initial label prediction model to be trained is used to perform video description on the alignment features of the first sample, thereby obtaining model description information for the first sample video; and The initial feature alignment model is updated based on the model description information and the sample description information, and then training continues until the training ends to obtain an intermediate feature alignment model.

13. The method according to any one of claims 1 to 12, wherein determining at least one candidate tag for the video based on the multimodal retrieval tags of the modal information of the at least two modalities comprises: Statistical analysis is performed on the multimodal retrieval tags for each of the at least two modalities to obtain tag statistical results; and At least one candidate tag is determined for the video based on the tag statistics results.

14. A video tagging device, the device comprising: The modal information extraction module is used to determine the video to be processed and extract modal information of at least two modalities from the video. The cross-retrieval module is used to perform cross-retrieval based on the modal information of the at least two modalities, according to the feature dimensions of the at least two modalities, to obtain the multimodal retrieval tags of the modal information of the at least two modalities respectively; The candidate tag determination module is used to determine at least one candidate tag for the video based on the multimodal retrieval tags of the modal information of the at least two modalities. and The tag prediction module is used to predict tags based on the at least one candidate tag and the modal information of the at least two modalities to obtain video tags for the video.

15. A computer device comprising a memory and a processor, the memory storing computer-readable instructions, the processor executing the computer-readable instructions to implement the steps of the method of any one of claims 1 to 13.

16. A computer-readable storage medium having stored thereon computer-readable instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 13.

17. A computer program product comprising computer-readable instructions that, when executed by a processor, implement the steps of the method of any one of claims 1 to 13.

Images