Video tag identification method, device, equipment, medium and product

Abstract

The embodiment of the present application discloses a video tag recognition method, device, equipment, medium and product, the method comprising: obtaining a video to be recognized, and extracting multimodal features of the video to be recognized to obtain a first multimodal feature, the first multimodal feature comprising multiple modal features corresponding to multiple modalities; retrieving multiple reference videos similar to the video to be recognized according to the first multimodal feature, each reference video carrying a video tag; constructing contextual learning information according to the second multimodal features corresponding to each reference video and the video tags carried by each reference video, and the first multimodal feature of the video to be recognized; and identifying the video tag of the video to be recognized according to the contextual learning information. The technical solution of the embodiment of the present application can quickly and accurately identify the video tag of the video.

Description

Technical Field

[0001] The present application relates to the field of video tag technology, and in particular to a video tag recognition method, a video tag recognition device, an electronic device, a computer-readable storage medium, and a computer product. Background Art

[0002] Video tag recognition is an important part of video content features. By automatically generating tags for massive user-generated content (UGC) videos, we can provide video content features of different granularities for downstream content distribution links (such as recommendation systems and content operations). However, due to the diversity of UGC video content, the tag libraries used in actual business scenarios often contain hundreds of thousands or even millions of tags, and accurately matching the corresponding video tags for each video faces huge challenges. Summary of the invention

[0003] The embodiments of the present application provide a video tag recognition method, a video tag recognition device, an electronic device, a computer-readable storage medium, and a computer program product, which can quickly and accurately recognize the video tag of a video.

[0004] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0005] According to one aspect of an embodiment of the present application, a video tag identification method is provided, including: obtaining a video to be identified, and performing multimodal feature extraction on the video to be identified to obtain a first multimodal feature, wherein the first multimodal feature includes multiple modal features corresponding to multiple modalities; retrieving multiple reference videos similar to the video to be identified based on the first multimodal feature, each of the reference videos carrying a video tag; constructing context learning information based on second multimodal features corresponding to each of the reference videos and video tags carried by each of the reference videos, as well as the first multimodal feature of the video to be identified; and identifying the video tag of the video to be identified based on the context learning information.

[0006] According to one aspect of an embodiment of the present application, a video tag identification device is provided, including: an acquisition module, used to acquire a video to be identified, and perform multimodal feature extraction on the video to be identified to obtain a first multimodal feature, wherein the first multimodal feature includes multiple modal features corresponding to multiple modalities; a retrieval module, used to retrieve multiple reference videos similar to the video to be identified based on the first multimodal feature, each of the reference videos carrying a video tag; a construction module, used to construct context learning information based on second multimodal features corresponding to each of the reference videos and video tags carried by each of the reference videos, as well as the first multimodal feature of the video to be identified; and an identification module, used to identify the video tag of the video to be identified based on the context learning information.

[0007] In one embodiment of the present application, the first multimodal feature includes a first video frame feature, and the second multimodal feature includes a second video frame feature; the construction module is further used to perform feature conversion processing on the first video frame feature to obtain a first visual text mark sequence, and perform feature conversion processing on the second video frame feature corresponding to each of the reference videos to obtain a second visual text mark sequence corresponding to each of the reference videos; obtain the first text information corresponding to the video to be identified and the second text information corresponding to each of the reference videos; construct the context learning information according to the second visual text mark sequence corresponding to each of the reference videos, the second text information and the video label, as well as the first visual text mark sequence and the first text information.

[0008] In one embodiment of the present application, the number of the first video frame features includes multiple; the construction module is further used to perform feature fusion processing on the multiple first video frame features to obtain target video features; and the target video features are aligned to a preset text feature space through a pre-trained feature alignment module to obtain the first visual text tag sequence.

[0009] In one embodiment of the present application, the construction module is further used to construct a learning example based on a first text tag used to isolate different information under the same video, and the second visual text tag sequence, the second text information and the video label corresponding to the same reference video, so as to obtain multiple learning examples corresponding to multiple reference videos; splicing the multiple learning examples to obtain learning information, wherein the multiple learning examples in the learning information are separated by the second text tag used to isolate information of different videos; constructing a recognition example based on the first text tag, the first visual text tag sequence and the first text information; generating the context learning information based on the learning information and the recognition example.

[0010] In one embodiment of the present application, the acquisition module is further used to obtain the video title of the video to be identified, and the recognition text converted from the audio information of the video to be identified; generate a context description of the video to be identified based on the video scene of the video to be identified; and generate the first text information based on the video title, the recognition text and the context description.

[0011] In one embodiment of the present application, the first multimodal feature includes a target video feature and a first text feature; the retrieval module is used to obtain a pre-established video feature retrieval library and a text feature retrieval library, the video feature retrieval library includes a mapping relationship between each candidate video and the video feature corresponding to the candidate video, and the text feature retrieval library includes a mapping relationship between each candidate video and the text feature corresponding to the candidate video; the video feature retrieval library and the text feature retrieval library are respectively searched according to the target video feature, and the video feature retrieval library and the text feature retrieval library are respectively searched according to the first text feature to obtain a plurality of target candidate videos similar to the video to be identified; the reference video is selected from the plurality of target candidate videos according to the video similarity between the plurality of target candidate videos and the video to be identified.

[0012] In one embodiment of the present application, the multiple target candidate videos include multiple target candidate videos retrieved by each retrieval method; the retrieval module is further used to calculate the similarity between the video to be identified and each target candidate video for the multiple target candidate videos retrieved by each retrieval method; for each target candidate video, calculate the average similarity between the target candidate video and the video to be identified in each retrieval method; and select the reference video from the multiple target candidate videos based on the average similarity.

[0013] In one embodiment of the present application, the acquisition module is further used to extract multiple video frames from the video to be identified, and divide the multiple video frames into multiple segments; extract a target video frame from the multiple segments, and perform video feature extraction on the target video frame to obtain a first video frame feature; obtain the first text information corresponding to the video to be identified, and perform text feature extraction on the first text information to obtain a first text feature; obtain the first multimodal feature based on the first video frame feature and the first text feature.

[0014] In one embodiment of the present application, recognition is further used to perform sequence feature conversion on the context learning information to obtain a context learning sequence, wherein the sequence feature is an input feature supported by a label generation model; the context learning sequence is input into the label generation model, wherein the label generation model is obtained by freezing the original model parameters of a preset language model and adjusting the newly added model parameters of the language model according to the sample context learning sequence, wherein the newly added model parameters are related to a low-rank adaptive module introduced into the language model; and a target video label of the video to be identified output by the label generation model is obtained.

[0015] In one embodiment of the present application, the device also includes a training module for obtaining a first sample video feature, a first sample text information and a first sample video label corresponding to the first sample video, and a second sample video feature, a second sample text information and a second sample video label corresponding to the second sample video; performing feature alignment processing on the first sample video feature and the second sample video feature according to the pre-trained initial feature alignment module to obtain a first sample visual mark sequence and a second sample visual mark sequence, and constructing sample context learning information according to the first sample visual mark sequence, the first sample text information and the first sample video label, and the second sample visual mark sequence and the second sample text information; performing sequence feature conversion on the sample context learning information to obtain the sample context learning sequence; introducing the low-rank adaptive module into the language model to introduce the newly added model parameters into the language model through the low-rank adaptive module; keeping the original model parameters of the language model frozen, and adjusting the newly added model parameters of the language model according to the sample context learning sequence and the second sample video label to obtain the label generation model.

[0016] In one embodiment of the present application, the training module is also used to obtain a sample image and a sample description text corresponding to the sample image; perform feature extraction on the sample image to obtain a sample visual feature, and input the sample visual feature into the module to be trained so that the module to be trained aligns the sample visual feature to the input text feature space of the language model; obtain the sample text feature corresponding to the alignment description instruction, and input the sample text feature and the target sample visual text feature output by the feature alignment module into the language model, wherein the model parameters of the language model remain frozen; obtain the sample prediction description text output by the language model, and train the module to be trained according to the sample description text and the sample prediction description text to obtain the initial feature alignment module.

[0017] In one embodiment of the present application, the training module is further used to input the sample context learning sequence into the language model introduced into the low-rank adaptive module, and obtain the predicted sample label output by the language model; according to the difference between the predicted sample label and the second sample video label, the newly added model parameters of the language model are adjusted to obtain the label generation model; the training module is also used to adjust the module parameters of the initial feature alignment module according to the difference between the predicted sample label and the second sample video label.

[0018] According to one aspect of an embodiment of the present application, an embodiment of the present application provides an electronic device, comprising one or more processors; a storage device for storing one or more computer programs, when the one or more computer programs are executed by the one or more processors, the electronic device implements the video tag recognition method as described above.

[0019] According to one aspect of an embodiment of the present application, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor of an electronic device, the electronic device executes the video tag recognition method as described above.

[0020] According to one aspect of an embodiment of the present application, an embodiment of the present application provides a computer program product, including a computer program, wherein the computer program is stored in a computer-readable storage medium, and a processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device performs the above-mentioned video tag recognition method.

[0021] In the technical solution provided in the embodiments of the present application, a video to be identified is obtained, and multimodal features are extracted for the video to be identified to generate a first multimodal feature containing multiple modal features, and then multiple reference videos similar to the video to be identified are retrieved based on the first multimodal feature, each reference video carries a video label, and the reference video not only provides rich comparison information, but also provides important contextual basis for label identification of the video to be identified; contextual learning information is constructed based on the second multimodal features corresponding to each reference video and the video labels carried by each reference video, as well as the first multimodal features of the video to be identified. Through the contextual learning information, label identification is not limited to feature analysis of a single modality, but uses labels and multimodal features of multiple reference videos for contextual comparison and learning, providing a reference for identifying video labels of the video to be identified, thereby quickly and accurately identifying video labels of the video to be identified, ensuring that the identified video labels are more accurate and reliable.

[0022] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings.

[0024] Figure 1 It is a schematic diagram of an implementation environment involved in this application.

[0025] Figure 2 It is a flowchart of a video tag recognition method shown in an exemplary embodiment of the present application.

[0026] Figure 3 It is a schematic diagram of another video tag recognition method shown in an exemplary embodiment of the present application.

[0027] Figure 4-1 It is a schematic diagram of context learning information shown in an exemplary embodiment of the present application.

[0028] Figure 4-2 It is a schematic diagram of another kind of context learning information shown in an exemplary embodiment of the present application.

[0029] Figure 5 It is a flowchart of another video tag identification method shown in an exemplary embodiment of the present application.

[0030] Figure 6 It is a flowchart of another video tag identification method shown in an exemplary embodiment of the present application.

[0031] Figure 7 It is a flowchart of another video tag identification method shown in an exemplary embodiment of the present application.

[0032] Figure 8 It is a flowchart of another video tag identification method shown in an exemplary embodiment of the present application.

[0033] Fig. 9 It is a training diagram of a feature training module shown in an exemplary embodiment of the present application.

[0034] Fig.10 It is a schematic diagram of an application chain of a video tag identification method shown in an exemplary embodiment of the present application.

[0035] Fig.11 It is a schematic diagram of the overall structural flow of video tag recognition shown by an exemplary embodiment of the present application.

[0036] Fig.12 It is a structural block diagram of a video tag recognition device shown in an exemplary embodiment of the present application.

[0037] Fig.13 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0038] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0039] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0040] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all contents and operations, nor must they be executed in the order described. For example, some operations may be decomposed, while some operations may be combined or partially combined, so the actual execution order may change according to actual conditions.

[0041] It should also be noted that the "multiple" mentioned in this application refers to two or more than two. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship.

[0042] The technical solutions of the embodiments of the present application are introduced in detail below.

[0043] See also Figure 1 , Figure 1 1 is a schematic diagram of an implementation environment involved in the present application. The implementation environment includes a terminal 10 and a server 20.

[0044] The terminal 10 is used for the video to be identified, and sends the video to be identified to the server 20 .

[0045] The server 20 is used to perform multimodal feature extraction on the video to be identified to obtain a first multimodal feature, where the first multimodal feature includes multiple modal features corresponding to multiple modalities, and retrieve multiple reference videos similar to the video to be identified based on the first multimodal feature, each reference video carrying a video tag, wherein the server also performs multimodal feature extraction on each reference video to obtain a second multimodal feature corresponding to each reference video, and then constructs context learning information based on the second multimodal features and video tags corresponding to each reference video, and the first multimodal feature of the video to be identified, and identifies the video tag of the video to be identified through the context learning information.

[0046] The server may also send the video tag of the video to be identified to the terminal, so that the terminal can classify the video or recommend the video according to the video tag.

[0047] In some embodiments, the server 20 may also obtain the video to be identified by itself, and then perform multimodal feature extraction, similar video retrieval, and context learning information construction, and identify the video tag of the video to be identified based on the context learning information.

[0048] In some embodiments, the terminal 10 can also independently implement the processing of video tag recognition, that is, the terminal 10 obtains the video to be identified to perform multimodal feature extraction, similar video retrieval and context learning information construction, and then identifies the video tag of the video to be identified.

[0049] Among them, the aforementioned terminal 10 can be any electronic device that can obtain modal data of the target object, such as a smart phone, a tablet, a laptop computer, a computer, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, etc. The server 20 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network) and big data and artificial intelligence platforms. There is no restriction on this here.

[0050] The terminal 10 and the server 20 establish a communication connection in advance through a network, so that the terminal 10 and the server 20 can communicate with each other through the network. The network can be a wired network or a wireless network, and there is no limitation here.

[0051] It should be noted that in the specific implementation of the present application, at least one of the video to be identified and the reference video is related to an object. When the embodiment of the present application is applied to a specific product or technology, it is necessary to obtain the permission or consent of the object, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0052] The various implementation details of the technical solutions of the embodiments of the present application are elaborated in detail below.

[0053] like Figure 2 As shown, Figure 2 is a flowchart of a video tag recognition method shown in an embodiment of the present application, which can be applied to Figure 1 In the implementation environment shown, the method can be executed by the terminal or the server, or by the terminal and the server together. In the embodiment of the present application, the method is executed by the server as an example for explanation. The video tag identification method may include S210 to S240, which are described in detail as follows.

[0054] S210: Obtain a video to be identified, and perform multimodal feature extraction on the video to be identified to obtain a first multimodal feature, where the first multimodal feature includes multiple modal features corresponding to multiple modalities.

[0055] In the embodiment of the present application, the video to be identified refers to a video whose video tag has not yet been determined, and the video tag is a label used to identify and classify video content. The label can be a specific keyword, phrase, or any descriptive information related to the video theme.

[0056] A video to be identified is obtained, and multimodal feature extraction is performed on the video to be identified, wherein modality can be understood as different representations or acquisition methods of data, such as images and texts belong to different modalities, and multimodality includes at least two modalities. Multimodal feature extraction is performed on the video to be identified, and then the first multimodal feature obtained includes multiple modal features obtained by performing feature extraction on multiple modalities respectively. For example, feature extraction can be performed on the video content of the video to be identified, and feature extraction can be performed on the text information of the video to be identified. It can also be feature extraction on the audio information of the video to be identified. The first multimodal feature includes video features corresponding to the video content, text features corresponding to the text information, and audio features corresponding to the audio information.

[0057] In one example, the video to be identified includes information of multiple modalities, and feature extraction can be performed on a specific modality of the video to be identified according to the business scenario to which the video to be identified belongs or the instructions of the terminal; for example, if the business scenario to which the video to be identified belongs is a short video scene using a template, and the template includes uniform background music, feature extraction may not be performed on the audio information of the video to be identified, but feature extraction may be performed on the text and video content of the video to be identified.

[0058] In another example, feature extraction can be performed on characteristic modalities of the video to be identified based on the video type of the video to be identified. For example, for a music video, feature extraction can be performed on the video content and sound; for a TV / movie video segment, feature extraction can be performed on the video content, text, and sound of the video.

[0059] S220: Retrieve a plurality of reference videos similar to the video to be identified according to the first multimodal feature, each reference video carrying a video tag.

[0060] In an embodiment of the present application, a video library is pre-stored, and the video library includes multiple candidate videos carrying video tags. The video tags of the candidate videos can be obtained by identification through steps S210~S240, or can be determined by labeling by relevant personnel, which is not limited here.

[0061] After obtaining the first multimodal feature of the video to be identified, multiple reference videos similar to the video to be identified are retrieved from the video library according to the first multimodal feature, wherein the similarity between the retrieved reference videos and the video to be identified is greater than a preset similarity threshold.

[0062] In one example, the process of retrieving similar videos based on the first multimodal feature may be to compare each modal feature included in the first multimodal feature with the corresponding modal feature of the candidate video in the video library, and then select the reference video corresponding to the feature similarity higher than the preset similarity threshold; such as comparing the video feature included in the first multimodal feature with the video feature of each candidate video in the video library to select a first reference video set; comparing the text feature included in the first multimodal feature with the text feature of each candidate video in the video library to select a second reference video set; comparing the video feature included in the first multimodal feature with the text feature of each candidate video in the video library to select a third reference video set; comparing the text feature included in the first multimodal feature with the video feature of each candidate video in the video library to select a fourth reference video set, and obtaining a final reference video similar to the video to be identified based on the first to fourth reference video sets.

[0063] In another example, the process of retrieving similar videos based on the first multimodal feature may be to fuse the multiple modal features included in the first multimodal feature to obtain a fused feature, and compare the fused feature with the fused features of each candidate video in the video library to select a fifth reference video set.

[0064] In another example, a final reference video similar to the video to be identified can be obtained based on the above-mentioned first to fourth reference video sets and fifth reference video set. For example, the intersection video of the first to fourth reference video sets and the fifth reference video set is used as the final reference video; or the first to fourth reference video sets and the fifth reference video set are all used as the final reference video.

[0065] S230: Construct contextual learning information according to the second multimodal features corresponding to the reference videos, the video tags carried by the reference videos, and the first multimodal features of the video to be identified.

[0066] In an embodiment of the present application, it is necessary to perform multimodal feature extraction on each reference video to obtain a second multimodal feature, wherein the process of performing multimodal feature extraction on the video to be identified and the reference video is the same, and the modal feature type included in the first multimodal feature is the same as the modal feature type included in the second multimodal feature, for example, the feature type of the first multimodal feature includes video features and text features, and the feature type of the second multimodal feature also includes video features and text features.

[0067] It can be understood that there is a mapping relationship between the second multimodal features of the reference video and the video label, that is, the reference video is a learning example related to the video label, and then contextual learning information is constructed according to the second multimodal features corresponding to each reference video and the video label carried by each reference video, as well as the first multimodal features of the video to be identified. The contextual learning information includes the contextual information provided by the learning example, which is prompt information with analogy learning.

[0068] In one example, the context learning information may be obtained by splicing the second multimodal features corresponding to the reference videos, the video tags carried by the reference videos, and the first multimodal features of the video to be identified.

[0069] S240: Identify the video tag of the video to be identified based on the context learning information.

[0070] As described above, the contextual learning information includes the contextual information provided by the learning examples, and the video tags of the video to be identified are identified based on the contextual learning information. The contextual information provided by the learning examples can be used to guide the identification of the video tags of the video to be identified, and then the target video tags of the video to be identified are obtained.

[0071] In one example, the context learning information is input into a pre-trained label generation model, and the label generation model generates a video label for the video to be identified according to the context learning information.

[0072] In an embodiment of the present application, a video to be identified is obtained, and multimodal features are extracted for the video to be identified to generate a first multimodal feature containing multiple modal features, and then multiple reference videos similar to the video to be identified are retrieved based on the first multimodal feature, each reference video carries a video label, and the reference video not only provides rich comparison information, but also provides important contextual basis for label identification of the video to be identified; contextual learning information is constructed based on the second multimodal features corresponding to each reference video and the video labels carried by each reference video, as well as the first multimodal feature of the video to be identified. Through the contextual learning information, label identification is not limited to feature analysis of a single modality, but uses labels and multimodal features of multiple reference videos for contextual comparison and learning, providing a reference for identifying video labels of the video to be identified, reducing possible deviations under a single modality, and being able to quickly and accurately identify video labels of the video to be identified, ensuring that the identified video labels are more accurate and reliable.

[0073] In one embodiment of the present application, another video tag recognition method is provided. The video tag recognition method can be applied to Figure 1 In the implementation environment shown, the method can be executed by a terminal or a server, or by a terminal and a server together. In the embodiment of the present application, the method is described by taking the server as an example. Figure 3 As shown in Figure 2 On the basis of S210~S240 shown in , S230 is expanded to S310~S330, wherein the first multimodal feature includes the first video frame feature, the second multimodal feature includes the second video frame feature, and the video frame feature is a feature extraction of the video frame of the video to be identified; S310~S330 are described in detail as follows.

[0074] S310, performing feature conversion processing on the first video frame features to obtain a first visual text mark sequence, and performing feature conversion processing on the second video frame features corresponding to each reference video to obtain a second visual text mark sequence corresponding to each reference video.

[0075] In an embodiment of the present application, the video label is text-type information, and the video label and video frame features come from different forms and modalities. In order to facilitate the subsequent construction of contextual learning information, the video frame features can be converted into a visual text tag sequence through feature conversion processing. The visual text tag sequence is a token sequence corresponding to the video frame features, wherein the process of performing feature conversion processing on the first video frame features and the second video frame features is the same.

[0076] In one example, the number of first video frame features includes multiple, and a first visual text mark sequence is performed on the first video frame features through feature conversion, including: performing feature fusion processing on the multiple first video frame features to obtain target video features; aligning the target video features to a preset text feature space through a pre-trained feature alignment module to obtain a first visual text mark sequence.

[0077] Among them, since there are multiple first video frame features, feature fusion processing is performed on the multiple first video frame features to obtain a target video feature, and the target video feature can better contain comprehensive video content; the feature fusion processing can be an average pooling processing or a weighted processing, wherein the weight of each first video frame feature can be determined according to the position of the video frame corresponding to the first video frame feature in the video to be identified, such as the closer the position of the video frame in the video to be identified is to the middle position, the greater the weight of the video frame feature corresponding to the video frame; the closer the position of the video frame in the video to be identified is to the start or end position, the smaller the weight of the video frame feature corresponding to the video frame.

[0078] In other embodiments of the present application, if there is only one first video frame feature, the first video frame feature can be enhanced to expand the video frame feature, and then feature fusion processing is performed based on the first video frame feature and the expanded video frame feature to obtain the target video feature.

[0079] In an embodiment of the present application, the feature alignment module is pre-trained, and the target video features are input into the feature alignment module to align the target video features to a preset text feature space, which can be the input text feature space of a pre-trained label generation model, that is, to align non-text tokens to the space of the label generation model input text tokens, thereby "translating" the non-text features into content that the label generation model can understand, so that the subsequent label generation model can identify video labels based on contextual learning information.

[0080] Similarly, feature fusion is performed on multiple second video frame features to obtain video features, and the video features are aligned to a preset text feature space through a pre-trained feature alignment module to obtain a second visual text tag sequence.

[0081] S320: Obtain first text information corresponding to the video to be identified and second text information corresponding to each reference video.

[0082] In an embodiment of the present application, the first text information can be obtained based on the accompanying text information of the video to be identified, such as the accompanying text information including the video title, video release time, video release platform, text content detected and identified from the video, such as subtitles, text obtained by converting human voice, etc.; the text type contained in the second text information corresponding to the reference video is the same as the text type contained in the first text information.

[0083] It is understandable that the accompanying text information of the video to be identified has more and more complex information content. In order to construct efficient and accurate context learning information, the accompanying text information can be screened to obtain the first text information.

[0084] In one example, obtaining first text information corresponding to a video to be identified includes: obtaining a video title of the video to be identified, and an identification text converted from audio information of the video to be identified; generating a context description of the video to be identified based on a video scene of the video to be identified; and generating first text information based on the video title, the identification text, and the context description.

[0085] Among them, metadata can be parsed from the video file of the video to be identified to extract the video title. If the extracted video title includes special characters or formats, it needs to be cleaned and normalized, such as removing special characters, processing spaces, normalizing upper and lower cases, etc.; if the extracted video title is in a different language from the video tag, the extracted video title can also be converted into the language corresponding to the video tag.

[0086] It is understandable that the video to be identified includes audio information, which is not background audio but audio emitted by an object contained in the video to be identified, such as an audio conversation. The audio information can be converted into recognized text through automatic speech recognition technology.

[0087] The video title and recognition text are the basic text information attached to the video. In order to further add descriptive language to provide more contextual background and further enrich the contextual information brought by the video itself, the context description can be determined according to the video scene of the video to be recognized. Among them, the scene content in the video can be analyzed by object detection, such as determining the corresponding scene of the video by identifying the objects appearing in the video; further analyze the timing information of the video to be recognized, determine the scene change information of the video, so as to help understand the context of the video, and then generate a context description based on the scene and scene change information of the video. For example, if elements such as the beach and the sun are detected in the video, a context description similar to "on a sunny beach" can be generated.

[0088] In one example, the situation description can be further refined based on the recognition of keywords and voice emotions (such as cheerfulness, seriousness) in the text. For example, if laughter or cheering is detected in the audio, it can be inferred that the situation of the video may be a happy or celebratory scene.

[0089] After obtaining the video title, recognition text and situation description, the video title, recognition text and situation description can be spliced ​​together to obtain the first text information, that is, meaningful text information is obtained from a video, and a first text information with context and contextual understanding is generated to provide a basis for subsequent label generation.

[0090] In one example, if there are repeated words in the video title, recognition text and situation description, the first text information is spliced ​​after deleting the repeated words; in one example, if the information content of the first text information is small, such as the number of text words contained in the first text information is less than a preset threshold, the text data of the first text information can be enhanced to generate diversified text inputs through synonym replacement, sentence reconstruction, text expansion and other methods.

[0091] The process of acquiring the second text information is the same as the process of acquiring the first text information, and will not be described in detail here.

[0092] S330: Construct context learning information according to the second visual text mark sequence, the second text information and the video label corresponding to each reference video, and the first visual text mark sequence and the first text information.

[0093] In an embodiment of the present application, the second visual text marker sequence, the second text information and the video label corresponding to each reference video, and the first visual text marker sequence and the first text information can be spliced ​​together to obtain contextual learning information; wherein, the second visual text marker sequence and the second text information corresponding to the same reference video can be used as input, and the video label can be used as output, and the input and output can be spliced ​​together to obtain an input-output learning example, and contextual information is provided through the learning example, and the learning examples corresponding to multiple reference videos are spliced ​​with the first visual text marker sequence and the first text information of the video to be identified to obtain contextual learning information.

[0094] In one example, contextual learning information is constructed based on the second visual features, second text information and video labels corresponding to each reference video, as well as the first visual text label sequence and the first text information, including: constructing a learning example based on the first text label used to isolate different information under the same video, and the second visual text label sequence, the second text information and the video label corresponding to the same reference video to obtain multiple learning examples corresponding to multiple reference videos; splicing the multiple learning examples to obtain learning information, wherein the multiple learning examples in the learning information are separated by the second text label used to isolate information from different videos; constructing a recognition example based on the first text label, the first visual text label sequence and the first text information; and generating contextual learning information based on the learning information and the recognition example.

[0095] For example, the first text tag and the second text tag are special tokens, such as the first text tag is <pad>, used to isolate different information in the same video, refer to the learning example corresponding to video 1, that is, the aforementioned input-output example, such as: Second visual text label sequence 1 <pad>Second text message 1 <pad>Video label 1; reference video 2 corresponding learning example, such as: second visual text label sequence 2 <pad>Second text message 2 <pad>Video tag 2, and so on to obtain multiple learning examples corresponding to multiple reference videos. The second text tag is <eoc>, used to isolate information from different videos, and concatenate multiple learning examples to obtain learning information, which is: the second visual text label sequence 1 <pad>Second text message 1 <pad>Video tag 1 <eoc>Second visual text tag sequence 2 <pad>Second text message 2 <pad>Video tag 2 <eoc>, and so on, where the video tag 1 and the second visual text tag sequence are information of different videos, so through <eoc>segmentation.

[0096] In the embodiment of the present application, a recognition example is constructed, and the recognition example is: a first visual text mark sequence <pad>The first text information; then the learning information and the recognition example are spliced ​​together to form contextual learning information with prompt information, and the prompt information can be a mapping relationship between a visual text tag sequence, text information and a video tag; wherein the learning information and the recognition example also belong to information of different videos, so the learning information and the recognition example are spliced ​​together through the second text tag to obtain the following Figure 4-1 Contextual learning information shown.

[0097] It is understandable that the marking forms of the first text mark and the second text mark can be flexibly adjusted according to actual conditions. <pad>and <eoc>An example is provided for an embodiment of the present application.

[0098] In other embodiments of the present application, when constructing contextual learning information, in addition to combining multiple reference videos similar to the video to be identified, a negative example video that is unrelated to the video to be identified can also be introduced as a comparison, such as splicing the third visual text tag sequence, the third text information and the video label of the negative example video through the first text tag to obtain negative information, and then splicing the learning information, the negative information and the recognition example to obtain Figure 4-2 Contextual learning information shown.

[0099] It should be noted that Figure 3 For more detailed information on S210~S220 and S240, please refer to Figure 2 S210~S220 and S240 shown are not repeated here.

[0100] In an embodiment of the present application, by first listing the multimodal features and video labels of a reference video to construct a learning example, then listing the multimodal features of a video to be identified to construct an identification example, and then concatenating the learning example and the identification example, an input containing contextual information of similar videos is constructed, which can effectively utilize multimodal information for contextual learning, so as to facilitate the subsequent accurate generation of content labels for the video to be identified.

[0101] The present application embodiment provides another video tag recognition method, which can be applied to Figure 1 In the implementation environment shown, the method can be executed by a terminal or a server, or by a terminal and a server together. In the embodiment of the present application, the method is described by taking the server as an example. Figure 5 As shown in Figure 2 Based on the basis shown in Figure 2 The S220 shown in is expanded to S510~S530. The first multimodal feature includes a target video feature and a first text feature, wherein the target video feature can be obtained by fusion of multiple first video frame features, and the feature dimension of the target video feature is the same as the feature dimension of the first text feature. S510~S530 are described in detail as follows.

[0102] S510, obtaining a pre-established video feature retrieval library and a text feature retrieval library, wherein the video feature retrieval library includes a mapping relationship between each candidate video and the video features corresponding to the candidate video, and the text feature retrieval library includes a mapping relationship between each candidate video and the text features corresponding to the candidate video.

[0103] In an embodiment of the present application, a video feature retrieval library and a text feature retrieval library are pre-established, wherein the video feature retrieval library A includes a mapping relationship between each candidate video and the video features corresponding to the candidate video; and the text feature retrieval library B includes a mapping relationship between each candidate video and the text features corresponding to the candidate video.

[0104] S520, searching the video feature retrieval library and the text feature retrieval library respectively according to the target video feature, and searching the video feature retrieval library and the text feature retrieval library respectively according to the first text feature, to obtain a plurality of target candidate videos similar to the video to be identified.

[0105] According to the target video feature and the first text feature, searches are performed in retrieval library A and retrieval library B respectively. There are four retrieval methods: 1) Search retrieval library A with the target video feature to determine the candidate video set 1 corresponding to the video features similar to the target video features in retrieval library A; 2) Search retrieval library B with the target video feature to determine the candidate video set 2 corresponding to the text features similar to the target video features in retrieval library B; 3) Search retrieval library A with the first text feature to determine the candidate video set 3 corresponding to the video features similar to the first text features in retrieval library A; 4) Search retrieval library B with the first text feature to determine the candidate video set 4 corresponding to the text features similar to the first text features in retrieval library B.

[0106] Among them, during video retrieval, the feature similarity between the target video feature and the video feature and the text feature is calculated, and the feature similarity between the first text feature and the video feature and the text feature is calculated, wherein the feature similarity can be calculated by the cosine similarity of the feature vector, the Euclidean distance, etc., and then multiple candidate videos with high similarity are selected as the candidate video set, such as selecting K candidate videos corresponding to the top-K feature similarities to obtain the candidate video set, wherein the number of candidate videos included in each candidate video set can be the same or different.

[0107] In one example, duplicate videos are deleted from candidate video set 1 to candidate video set 4, and the candidate videos can be sorted from high to low according to similarity, and a specified number of candidate videos are selected as multiple target candidate videos.

[0108] In another example, the videos in candidate video set 1 to candidate video set 4 may be directly used as target candidate videos.

[0109] S530: Select a reference video from the multiple target candidate videos according to the video similarities between the multiple target candidate videos and the video to be identified.

[0110] In one example, the video similarity between each target candidate video and the video to be identified may be calculated, and then multiple reference videos may be selected from the multiple target candidate videos according to the video similarity.

[0111] In another example, the multiple target candidate videos include the multiple target candidate videos retrieved by each retrieval method, i.e., the target candidate videos in candidate video set 1 to the target candidate videos in candidate video set 4. For the multiple target candidate videos retrieved by each retrieval method, the similarity between the video to be identified and each target candidate video is calculated; for each target candidate video, the average similarity between the target candidate video and the video to be identified in each retrieval method is calculated; and a reference video is selected from the multiple target candidate videos according to the average similarity.

[0112] For multiple target candidate videos in the candidate video set 1 in retrieval method 1), calculate the video similarity between each target candidate video and the video to be identified; similarly, for retrieval methods 2)~4), calculate the video similarity between each target candidate video and the video to be identified; then, for each target candidate video, if the target candidate video appears in multiple candidate video sets at the same time, calculate the average similarity between the target candidate video and the video to be identified in each candidate video set; for example, retrieval method 1) retrieves four target candidate videos a, b, c, and d, and their video similarities with the video to be identified are {a: 0.4, b: 0.2, c: 0.1, d: 0.7} respectively; retrieval method 2) retrieves four target candidate videos a, b, d, and e, and their video similarities with the video to be identified are {a: 0.3, b: 0.5, d: 0.6, e: 0.6} respectively; retrieval method 3) retrieves three target candidate videos a, b, and c, and their video similarities with the video to be identified are {a: 0.4, b: 0.2, c: 0.1, d: 0.7} respectively. 0.5,c:0.3}; retrieval method 4) retrieves four target candidate videos a, b, e, and f, and their video similarities with the video to be identified are {a: 0.6, b: 0.2, e: 0.4, f: 0.2} respectively; for the target candidate video a, the average similarity between the target candidate video a and the video to be identified is 0.425; the average similarity of the target candidate video b is: 0.35; if a target candidate video does not appear in a retrieval method, then in this retrieval method, the similarity of the target candidate video is 0, such as the average similarity of the target candidate video c is (0.1+0+0.3+0) / 4=0.1, and the average similarity of the target candidate video d is 0.425; the average similarity of the target candidate video e is 0.25; the average similarity of the target candidate video f is 0.05.

[0113] When selecting a reference video from multiple target candidate videos according to the average similarity, the average similarity can be sorted from high to low, and then the top-N videos with high similarity can be selected. For example, the target candidate videos a, b, and d can be selected as reference videos, where N is less than K.

[0114] In other embodiments of the present application, when calculating the average similarity of each target candidate video, it is not necessarily necessary to assign the same weight to the four retrieval methods. Weights can be dynamically assigned to each retrieval method, such as the weights corresponding to retrieval method 1) and retrieval method 3) are greater than the weights corresponding to retrieval method 2) and retrieval method 4), and then the weighted average similarity of the target candidate videos is calculated, and the reference video is selected based on the weighted average similarity.

[0115] It should be noted that Figure 5 For detailed description of S210, S230~S240 shown in Figure 2 S210, S230~S240 shown are not described in detail here.

[0116] In the embodiment of the present application, video features and text features are used to perform cross-search in two different retrieval libraries, so that multimodal information can be used to the greatest extent to retrieve target candidate videos similar to the video to be identified; by calculating the average similarity of the target candidate video with the video to be identified in each retrieval method, the accuracy and reliability of selecting reference videos are improved, and then the top-N videos most similar to the video to be identified and their label results are obtained.

[0117] The present application also provides another video tag recognition method, which can be applied to Figure 1 In the implementation environment shown, the method can be executed by a terminal or a server, or by a terminal and a server together. In the embodiment of the present application, the method is described by taking the server as an example. Figure 6 As shown, in Figure 2 Based on the basis shown in Figure 2 The S210 shown in FIG. 1 is expanded to S610 to S640. S610 to S640 are described in detail as follows.

[0118] S610: Obtain a video to be identified, extract multiple video frames from the video to be identified, and divide the multiple video frames into multiple segments.

[0119] S620: extract a target video frame from the multiple segments, and perform video feature extraction on the target video frame to obtain a first video frame feature.

[0120] S630: Obtain first text information corresponding to the video to be identified, and perform text feature extraction on the first text information to obtain a first text feature.

[0121] S640: Obtain a first multimodal feature according to the first video frame feature and the first text feature.

[0122] In an embodiment of the present application, multiple video frames can be extracted from the video to be identified according to the video length of the video to be identified. For example, one video frame is extracted from the video to be identified at preset time intervals, and the longer the video length of the video to be identified, the longer the preset time is; for example, if the video to be identified is a 30S video, one video frame is extracted every 1s; if the video to be identified is a 1min video, one video frame is extracted every 2s; multiple video frames can also be randomly extracted from the video to be identified, and the video to be identified can also be divided into three segments, and video frames are extracted from the three segments according to different proportions.

[0123] After extracting multiple video frames, the multiple video frames can be divided into multiple segments, wherein the multiple video frames can be first sorted by time or randomly sorted, and then the multiple segments are equally cut, and then target video frames are extracted from the multiple segments, such as randomly extracting 1 frame or multiple frames from each segment to obtain multiple target video frames, and video features are extracted from the target video frames through a pre-trained video feature encoder to obtain first video frame features.

[0124] Get the first text information corresponding to the video to be identified, see Figure 3 In the illustrated embodiment, details will not be described one by one here; the first text feature is extracted from the first text information by a pre-trained text feature encoder.

[0125] The first video frame feature and the first text feature are used as the first multimodal feature.

[0126] In one example, a text feature encoder and a video feature encoder are trained using a CLIP (Contrastive Language–Image Pre-training) method, so that text features extracted by the text feature encoder and video features extracted by the video feature encoder are close to each other.

[0127] It should be noted that Figure 6 For more detailed information about S220~S240 shown in Figure 2 S220~S240 shown in FIG. 1 are not described in detail here.

[0128] In an embodiment of the present application, multiple video frames are extracted from a video, and then the video frames are divided into segments, and target video frames are extracted from the segments for video feature extraction, so that the obtained first video frame features are representative while avoiding feature uniformity; and based on the characteristics of the CLIP model, the video text features extracted by the encoder are ideally close to each video frame feature, thereby making the first multimodal feature more accurate and reliable.

[0129] In one embodiment of the present application, another video tag recognition method is also provided. The video tag recognition method can be applied to Figure 1 In the implementation environment shown, the method can be executed by a terminal or a server, or by a terminal and a server together. In the embodiment of the present application, the method is described by taking the server as an example. Figure 7 As shown in Figures 2 to 6 Based on the basis shown in Figure 2 The process of performing video tag recognition in S240 is expanded to S710 to S730. S710 to S730 are described in detail as follows.

[0130] S710 , performing sequence feature conversion on the context learning information to obtain a context learning sequence, where the sequence feature is an input feature supported by the label generation model.

[0131] In an embodiment of the present application, the context learning information is a continuous long text sequence. In order to facilitate the understanding and processing of the label generation model, the context learning information needs to be processed by sequence feature conversion, and the continuous long text sequence is converted into a token sequence that the model can understand and process. The token is the basic input unit of the language model, which can be a single character, word or subword, so that the model can better understand and process the semantics and structure of the text.

[0132] In one example, the context learning information may be converted into sequence features by a tokenizer of a language model to obtain a context learning sequence.

[0133] S720. Input the context learning sequence into the label generation model. The label generation model is obtained by freezing the original model parameters of the preset language model and adjusting the newly added model parameters of the language model according to the sample context learning sequence. The newly added model parameters are related to the low-rank adaptive module introduced into the language model.

[0134] In an embodiment of the present application, the label generation model can be obtained by training a preset language model, and the language model is obtained by pre-training the model, so the language model can be applied to the field of natural language processing (NLP), such as machine translation, speech recognition, text generation, etc., wherein the language model includes but is not limited to a large language model (LLM), a multimodal large language model (MLLM), and other generative models; on this basis, in order to enable the language model to be optimized for the application scenario of the content label, the language model is fine-tuned to better fit the needs of the business scenario. Among them, the embodiment of the present application uses a low-rank adaptation module (LoRA) to efficiently fine-tune the parameters of the language model. The LoRA introduces new model parameters, such as a low-rank matrix, into the language model. The new model parameters are trainable, while the original model parameters of the language model remain frozen, and the new model parameters of the language model are adjusted according to the sample context learning sequence, and then the language model containing the new model parameters and the original model parameters is used as the label generation model.

[0135] S730: Obtain a target video label of the video to be identified output by the label generation model.

[0136] The label generation model can learn from analogies to identify video labels of the video to be identified based on the contextual information provided by the learning examples in the contextual learning information.

[0137] It should be noted that Figure 7 For more detailed information about S210~S230, please refer to Figure 2 S210 to S230 shown are not described in detail here.

[0138] In an embodiment of the present application, contextual learning information includes reference prompts for generating video labels based on multimodal features, which are converted into a language model to better understand and process token sequences, and then a label generation model based on the language model can refer to the prompts to quickly generate labels for the video to be identified, and the label generation model is efficiently fine-tuned based on the language model so that it can better adapt to the task requirements of video labels.

[0139] It is worth noting that, in one embodiment of the present application, another video tag recognition method is also provided, which can be applied to Figure 1 In the implementation environment shown, the method can be executed by a terminal or a server, or by a terminal and a server together. In the embodiment of the present application, the method is described by taking the server as an example. Figure 8 As shown in Figure 7 Based on the above, the training process of the label generation model is added, namely S810 to S850. S810 to S850 are described in detail as follows.

[0140] S810: Obtain first sample video features, first sample text information, and first sample video labels corresponding to the first sample video, and second sample video features, second sample text information, and second sample video labels corresponding to the second sample video.

[0141] In the embodiment of the present application, the model is trained by a training set, the training set includes multiple sample videos, each sample video carries a video label, the first sample video is a reference sample video in training, and the second sample video is a sample video to be identified in training; the first sample video and the second sample video are similar videos; the process of obtaining sample video features of the sample video can be seen in Figure 6 For sample text information, see Figure 3 As shown, no further description is given here.

[0142] S820. Perform feature alignment processing on the first sample video feature and the second sample video feature respectively according to the pre-trained initial feature alignment module to obtain a first sample visual marker sequence and a second sample visual marker sequence, and construct sample context learning information according to the first sample visual marker sequence, the first sample text information and the first sample video label, and the second sample visual marker sequence and the second sample text information.

[0143] In an embodiment of the present application, an initial feature alignment module is pre-trained, and the initial feature alignment module can map video features to a space aligned with text features, so that visual information can be better understood and utilized by the language model. The first sample video features and the second sample video features are respectively subjected to feature alignment processing by the initial feature alignment module to map the first sample video features and the second sample video features to a space aligned with text features of the language model, and obtain a first sample visual marker sequence and a second sample visual marker sequence, respectively. Then, sample context learning information is constructed according to the first sample visual marker sequence, the first sample text information and the first sample video label, and the second sample visual marker sequence and the second sample text information. For the construction process, please refer to Figure 3 shown.

[0144] In one example, an initial feature alignment module can be trained through image and text description pairs in conjunction with a language model, and the training steps of the initial feature alignment module include: obtaining a sample image and a sample description text corresponding to the sample image; extracting features from the sample image to obtain sample visual features, and inputting the sample visual features into a module to be trained so that the module to be trained aligns the sample visual features to the input text feature space of the language model; obtaining sample text features corresponding to the alignment description instructions, and inputting the sample text features and the target sample visual text features output by the feature alignment module into the language model, wherein the model parameters of the language model remain frozen; obtaining the sample prediction description text output by the language model, and training the module to be trained based on the sample description text and the sample prediction description text to obtain the initial feature alignment module.

[0145] The module to be trained can be a network with a fully connected (FC) layer, or a multi-layer perceptron (MLP) or other types of neural network structures, such as q-former; the sample description text is used to describe the image content of the sample image. Fig. 9 As shown, the sample image is input into the video feature encoder, and the sample visual features are extracted by the video feature encoder. The sample visual features contain the main visual information of the sample image, such as objects, backgrounds, colors, etc. At the same time, an alignment description instruction is obtained, and the alignment description instruction is used to instruct the language model to generate a description text corresponding to the image. The alignment description instruction can be a text-type instruction, such as one of the following sentences: "Briefly describe the following picture.", "Provide a brief description of the given picture.", "Briefly explain the provided picture.", etc.; In order to facilitate the language model to process the alignment description instruction, the alignment description instruction is converted into a sample text feature through the Text Tokenizer, and the sample text feature is input into the language model.

[0146] Since visual features and text features come from different modalities, directly inputting them into a language model usually does not work well. Therefore, the sample visual features need to be input into the module to be trained. The module to be trained maps the visual features to a space consistent with the text features through learning to obtain the target sample visual text features so that they can be aligned with the text features in the same or similar feature space.

[0147] It should be noted that when training the module to be trained, the model parameters of the language model remain frozen. After the sample text features and the target sample visual text features are input into the language model, the language model combines the target sample visual text features and the sample text features to generate a sample prediction description text. The parameters of the module to be trained can be adjusted according to the difference between the sample prediction description text and the sample description text corresponding to the sample image until the module to be trained converges to obtain the initial feature alignment module. Through training, the initial feature alignment module can effectively align the visual features to a space consistent with the text features, so that the language model can perform well under multimodal input.

[0148] S830 , performing sequence feature conversion on the sample context learning information to obtain a sample context learning sequence.

[0149] The sample context learning sequence is obtained by performing sequence feature conversion on the sample context learning information through the tokenizer of the language model.

[0150] S840. Introduce a low-rank adaptive module into the language model, so as to introduce new model parameters into the language model through the low-rank adaptive module.

[0151] S850: Keep the original model parameters of the language model frozen, and adjust the newly added model parameters of the language model according to the sample context learning sequence and the second sample video label to obtain a label generation model.

[0152] In an embodiment of the present application, a LoRA module is introduced into a language model, and the LoRA module introduces a low-rank matrix into certain layers of the model; during the forward propagation process, the LoRA module transforms the input data through a low-rank matrix, and then passes it to the next layer of the model; this transformation is equivalent to adding an extra layer of parameter adjustment on the basis of the language model, that is, introducing new model parameters into the language model without changing the parameters of the original model; during the training process, only the new model parameters are trained, and the original model parameters of the language model are kept frozen without updating the original model parameters; in this way, the model can be fine-tuned while keeping the original model parameters unchanged.

[0153] In one example, a sample context learning sequence is input into a language model that introduces a low-rank adaptive module, and a predicted sample label output by the language model is obtained; the language model can learn by analogy from the sample context learning sequence to generate a predicted sample label corresponding to a second sample video; and the second sample video corresponds to a second sample video label, so the newly added model parameters of the language model can be adjusted according to the difference between the predicted sample label and the second sample video label to obtain a label generation model; such as calculating a loss function according to the difference between the predicted sample label and the second sample video label, such as a contrast loss function, a mean square error loss function, etc., and then optimizing the newly added model parameters according to the loss function until the difference between the predicted sample label and the second sample video label is less than a preset threshold to obtain a label generation model.

[0154] It should be noted that while training the LoRA module, the initial feature alignment module can be further optimized, that is, the module parameters of the initial feature alignment module are adjusted according to the difference between the predicted sample label and the second sample video label. For example, the module parameters of the initial feature alignment module are further optimized according to the loss function until the difference between the predicted sample label and the second sample video label is less than a preset threshold, thereby obtaining the final feature alignment module.

[0155] In an embodiment of the present application, the feature alignment module is first preliminarily trained in combination with the language model, and the parameters of the language model are kept frozen. Through training, the feature alignment module learns to align visual features with text features, thereby realizing effective conversion and fusion between multimodal features. Then, LoRA is introduced into the language model. LoRA uses low-rank decomposition to achieve effective adjustment of the model output with minimal parameter changes to adapt to the video labeling task requirements, train LoRA, and continue to train the adapter so that the model can generate content labels for the video to be identified based on the multimodal context instances, and the feature alignment of the feature alignment module is more accurate.

[0156] For ease of understanding, the present application embodiment also provides a video tag recognition method, such as Fig.10 As shown, the video enters the content processing link from the content production link, obtains the corresponding content features through human-machine collaboration, and enters the downstream content distribution link. The video tag recognition method proposed in the embodiment of the present application belongs to the machine tagging link.

[0157] The whole process of video tag recognition can be found in Fig.11 As shown, it includes video multimodal feature extraction, similar video retrieval, context learning instruction construction and model recognition process, where the LLM model is taken as an example. LLM can choose open source Chinese LLM, such as chinese-llama, BLOOM, etc.

[0158] The video multimodal feature extraction includes: extracting two modes from the original video, video frames and video text. Video frames: First, extract one frame per second from a video, and then divide it into M segments. In the training mode, randomly select one frame from each segment, and in the test mode, select the middle frame from each segment, and get a total of M frames; Video text: Extract the title of the video and the text converted by the video ASR, and splice them into a sentence.

[0159] In the embodiment of the present application, the reason for not using the original audio features is: based on the business scenario, many background sounds in short videos are unified popular background music, which does not have good discriminability, and important spoken information has been extracted by ASR and merged into the video text information. The original audio signal is not used as one of the inputs here.

[0160] The embodiment of this application uses the video feature encoder (cv encoder) and text feature encoder (text encoder) trained by the CLIP method to extract video multimodal features. Specifically, the video frame has a total of M images, and the video frame features are calculated by the cv encoder, denoted as f = {f v 1, f v 2,…, f v M }; The video title and other texts such as ASR are spliced ​​together, and the video text features are calculated by text encoder, denoted as f t Based on the feature alignment characteristics of the CLIP model, ideally, the video text features are close to the features of each video frame.

[0161] Before retrieving similar videos, it is necessary to construct a retrieval library. For each video, the video frame feature F = {f v 1, f v 2,…, f v M } and video text features f t In order to simplify the calculation process, the video feature F is average pooled to obtain a feature f with the same dimension as the text feature v , each video in the retrieval library has a label annotation result. At this time, two retrieval libraries can be built: Retrieval library A: {f v ==> video}, use the visual features of video frames to build a library; retrieval library B: {f t ==> video}, use video text features to build a database.

[0162] The above steps of constructing the retrieval library can be completed by offline calculation.

[0163] For the video to be identified, extract the visual features f v and text features f t After that, cross-search is performed on retrieval libraries A and B, and the top-K results of the 4-way search are returned. Based on the above introduction, the features of CLIP ideally align the visual and feature features. In addition to searching retrieval library A with visual features and searching retrieval library B with text features, cross-searches such as searching retrieval library B with visual features and searching retrieval library A with text features can also be used. The videos in the above 4-way top-K results are respectively calculated for similarity with the video to be identified, and then the average similarity of the retrieved videos in the 4-way results is calculated (if a video is not in a certain top-K, the similarity is considered to be 0). The top-N videos and corresponding label results are obtained based on the average similarity, which are used as subsequent context reference samples. Among them, K can be set to a value larger than N, such as K = 2 * N.

[0164] The embodiment of the present application draws on the basic idea of ​​in-context learning (ICL), that is, learning from analogy, and extends it to multimodal application scenarios; in the embodiment of the present application, the top-N samples closest to the video content to be identified are recalled through the similar video retrieval method described above, and the input of LLM is constructed using the following template.

[0165] Specifically, the template first lists the multimodal features and corresponding label results of the top-N videos, and then gives the features of the video to be identified, allowing LLM to generate its content label. <pad>and <eoc>They are all special tokens. <pad>Used to separate different modes and result token sequences within a video; <eoc>Represents the end of chunk, which is used to separate token sequences of different videos. The video text i in the template is the information spliced ​​together by the video title and ASR text (here refers to the original text, not the text feature f t ); Video tag i is the label annotation result text corresponding to the video; [visual token_i] in the template represents the token sequence of visual features, here through the video frame feature f v After being converted by an adapter (feature alignment module) that needs to be trained; the entire input sequence is generated into a token sequence (i.e., the aforementioned context learning sequence) by the LLM tokenizer; thus, through this method, an input similar to the context demonstration example is constructed for the LLM; see Figure 4-1 shown.

[0166] The model recognition process is that LLM receives a token sequence (i.e., the aforementioned context learning sequence) and generates a label based on this sequence.

[0167] It should be pointed out that the embodiment of the present application is different from the traditional one that does not train LLM at all. Considering that the task of content labeling is somewhat subjective, proper fine-tuning of LLM can better fit the needs of business scenarios. Therefore, a LoRA is used to efficiently fine-tune the parameters of LLM. During the training process, all LLM parameters remain frozen, and only the LoRA and adapter modules are trained.

[0168] The LLM training approach consists of two key phases.

[0169] Phase 1: Train the adapter to align visual features. In this phase, only the adapter is trained without adding LoRA, and the image-text description pair is used for training.

[0170] At this stage, no other information is provided to the LLM. Only visual tokens can be received, forcing the adapter to successfully "translate" the visual information to the LLM in order to generate the correct image description text, so that the adapter can be better trained. For details, see Fig. 9 shown.

[0171] Phase 2: Training LoRA. In this phase, LLM joins LoRA and trains the adapter and LoRA at the same time, allowing the model to generate label results for the video to be identified based on the context instance, such as Fig.11 As shown, at this stage, the input token sequence is generated according to the template as described above, without the need for manual design of instructions.

[0172] In the embodiment of the present application, if during the training process, the LLM model generates label results for the video to be identified, the selected context example combination and template structure can be optimized according to the quality feedback of the generated labels. For example, if the quality of the generated labels for certain categories is low, the context example combinations related to these categories can be preferentially selected for secondary learning.

[0173] It can be understood that after the model training is completed, the label of the new video can be predicted; after the visual features and text features of the video are extracted, by retrieving similar videos and constructing a context-enhanced input token sequence according to the template, LLM can directly generate the corresponding content label results.

[0174] The video tag recognition method provided in this application utilizes the inherent strong reasoning ability of the large model and also takes into account the differences in standards under business characteristics and scenarios, and proposes a multimodal large model based on context enhancement. By combining the idea of ​​context learning, a similar video retrieval is introduced, and the top-N recalled similar videos and corresponding label results of the video to be identified are input to the large model in the form of reference samples for reasoning together, thereby enhancing the features of the input model and achieving the training goal with fewer training samples.

[0175] Here, an apparatus embodiment of the present application is introduced, which can be used to execute the video tag identification method in the above-mentioned embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the above-mentioned embodiment of the video tag identification method of the present application.

[0176] The present application embodiment provides a video tag recognition device, such as Fig.12 As shown, the device includes.

[0177] The acquisition module 1210 is used to acquire a video to be identified, and perform multimodal feature extraction on the video to be identified to obtain a first multimodal feature, where the first multimodal feature includes multiple modal features corresponding to multiple modalities.

[0178] The retrieval module 1220 is used to retrieve a plurality of reference videos similar to the video to be identified according to the first multimodal feature, each of the reference videos carrying a video tag.

[0179] The construction module 1230 is used to construct context learning information according to the second multimodal features corresponding to each of the reference videos, the video tags carried by each of the reference videos, and the first multimodal features of the video to be identified.

[0180] The identification module 1240 is used to identify the video tag of the video to be identified according to the context learning information.

[0181] In one embodiment of the present application, based on the aforementioned scheme, the first multimodal feature includes a first video frame feature, and the second multimodal feature includes a second video frame feature; the construction module is further used to perform feature conversion processing on the first video frame feature to obtain a first visual text mark sequence, and perform feature conversion processing on the second video frame features corresponding to each of the reference videos to obtain a second visual text mark sequence corresponding to each of the reference videos; obtain the first text information corresponding to the video to be identified and the second text information corresponding to each of the reference videos; construct the context learning information according to the second visual text mark sequence corresponding to each of the reference videos, the second text information and the video label, as well as the first visual text mark sequence and the first text information.

[0182] In one embodiment of the present application, based on the aforementioned scheme, the number of the first video frame features includes multiple; the construction module is further used to perform feature fusion processing on the multiple first video frame features to obtain target video features; and the target video features are aligned to the preset text feature space through a pre-trained feature alignment module to obtain the first visual text tag sequence.

[0183] In one embodiment of the present application, based on the aforementioned scheme, the construction module is further used to construct a learning example based on a first text tag for isolating different information under the same video, and the second visual text tag sequence, the second text information and the video label corresponding to the same reference video, so as to obtain multiple learning examples corresponding to multiple reference videos; splicing the multiple learning examples to obtain learning information, wherein the multiple learning examples in the learning information are separated by the second text tag for isolating information of different videos; constructing a recognition example based on the first text tag, the first visual text tag sequence and the first text information; generating the contextual learning information based on the learning information and the recognition example.

[0184] In one embodiment of the present application, based on the aforementioned scheme, the acquisition module is further used to obtain the video title of the video to be identified, and the recognition text converted from the audio information of the video to be identified; generate a context description of the video to be identified according to the video scene of the video to be identified; and generate the first text information according to the video title, the recognition text and the context description.

[0185] In one embodiment of the present application, based on the aforementioned scheme, the first multimodal feature includes a target video feature and a first text feature; the retrieval module is used to obtain a pre-established video feature retrieval library and a text feature retrieval library, the video feature retrieval library includes a mapping relationship between each candidate video and the video feature corresponding to the candidate video, and the text feature retrieval library includes a mapping relationship between each candidate video and the text feature corresponding to the candidate video; the video feature retrieval library and the text feature retrieval library are respectively searched according to the target video feature, and the video feature retrieval library and the text feature retrieval library are respectively searched according to the first text feature to obtain multiple target candidate videos similar to the video to be identified; the reference video is selected from the multiple target candidate videos according to the video similarity between the multiple target candidate videos and the video to be identified.

[0186] In one embodiment of the present application, based on the aforementioned scheme, the multiple target candidate videos include multiple target candidate videos retrieved by each retrieval method; the retrieval module is further used to calculate the similarity between the video to be identified and each target candidate video for the multiple target candidate videos retrieved by each retrieval method; for each target candidate video, calculate the average similarity between the target candidate video and the video to be identified in each retrieval method; and select the reference video from the multiple target candidate videos according to the average similarity.

[0187] In one embodiment of the present application, based on the aforementioned scheme, the acquisition module is further used to extract multiple video frames from the video to be identified, and divide the multiple video frames into multiple segments; extract a target video frame from the multiple segments, and perform video feature extraction on the target video frame to obtain a first video frame feature; obtain the first text information corresponding to the video to be identified, and perform text feature extraction on the first text information to obtain a first text feature; obtain the first multimodal feature based on the first video frame feature and the first text feature.

[0188] In one embodiment of the present application, based on the aforementioned scheme, recognition is further used to perform sequence feature conversion on the context learning information to obtain a context learning sequence, and the sequence feature is an input feature supported by a label generation model; the context learning sequence is input into the label generation model, and the label generation model is obtained by freezing the original model parameters of a preset language model and adjusting the newly added model parameters of the language model according to the sample context learning sequence, and the newly added model parameters are related to the low-rank adaptive module introduced into the language model; and the target video label of the video to be identified output by the label generation model is obtained.

[0189] In one embodiment of the present application, based on the aforementioned scheme, the device also includes a training module for obtaining a first sample video feature, a first sample text information and a first sample video label corresponding to the first sample video, and a second sample video feature, a second sample text information and a second sample video label corresponding to the second sample video; performing feature alignment processing on the first sample video feature and the second sample video feature according to the pre-trained initial feature alignment module, respectively, to obtain a first sample visual mark sequence and a second sample visual mark sequence, and constructing sample context learning information according to the first sample visual mark sequence, the first sample text information and the first sample video label, and the second sample visual mark sequence and the second sample text information; performing sequence feature conversion on the sample context learning information to obtain the sample context learning sequence; introducing the low-rank adaptive module into the language model to introduce the newly added model parameters into the language model through the low-rank adaptive module; keeping the original model parameters of the language model frozen, and adjusting the newly added model parameters of the language model according to the sample context learning sequence and the second sample video label to obtain the label generation model.

[0190] In one embodiment of the present application, based on the aforementioned scheme, the training module is also used to obtain a sample image and a sample description text corresponding to the sample image; perform feature extraction on the sample image to obtain a sample visual feature, and input the sample visual feature into the module to be trained so that the module to be trained aligns the sample visual feature to the input text feature space of the language model; obtain the sample text feature corresponding to the alignment description instruction, and input the sample text feature and the target sample visual text feature output by the feature alignment module into the language model, wherein the model parameters of the language model remain frozen; obtain the sample prediction description text output by the language model, and train the module to be trained according to the sample description text and the sample prediction description text to obtain the initial feature alignment module.

[0191] In one embodiment of the present application, based on the aforementioned scheme, the training module is further used to input the sample context learning sequence into the language model introduced into the low-rank adaptive module, and obtain the predicted sample label output by the language model; according to the difference between the predicted sample label and the second sample video label, the newly added model parameters of the language model are adjusted to obtain the label generation model; the training module is also used to adjust the module parameters of the initial feature alignment module according to the difference between the predicted sample label and the second sample video label.

[0192] It should be noted that the device provided in the above embodiment and the method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here.

[0193] The device provided in the above embodiment may be arranged in a terminal or in a server.

[0194] An embodiment of the present application also provides an electronic device, comprising one or more processors and a storage device, wherein the storage device is used to store one or more computer programs, and when the one or more computer programs are executed by one or more processors, the electronic device implements the above-mentioned video tag recognition method.

[0195] Fig.13 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0196] It should be noted that Fig.13 The computer system 1300 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0197] like Fig.13 As shown, the computer system 1300 includes a processor (Central Processing Unit, CPU) 1301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (Read-Only Memory, ROM) 1302 or the program loaded from the storage part 1308 to the random access memory (Random Access Memory, RAM) 1303, such as executing the method in the above embodiment. In the RAM 1303, various programs and data required for system operation are also stored. The CPU 1301, the ROM 1302 and the RAM 1303 are connected to each other through the bus 1304. The input / output (Input / Output, I / O) interface 1305 is also connected to the bus 1304.

[0198] In some embodiments, the following components are connected to the I / O interface 1305: an input section 1306 including a keyboard, a mouse, etc.; an output section 1307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 1308 including a hard disk, etc.; and a communication section 1309 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1309 performs communication processing via a network such as the Internet. A drive 1310 is also connected to the I / O interface 1305 as needed. A removable medium 1311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1310 as needed so that a computer program read therefrom is installed into the storage section 1308 as needed.

[0199] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 1309, and / or installed from a removable medium 1311. When the computer program is executed by the processor (CPU) 1301, various functions defined in the system of the present application are executed.

[0200] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (Erasable Programmable Read Only Memory), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier, which carries a computer-readable computer program. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0201] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the devices, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and a computer program.

[0202] The units or modules involved in the embodiments described in this application may be implemented by software or hardware, and the units or modules described may also be set in a processor. The names of these units or modules do not, in some cases, constitute limitations on the units or modules themselves.

[0203] Another aspect of the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the video tag recognition method as described above is implemented. The computer-readable storage medium can be included in the electronic device described in the above embodiment, or it can exist independently without being assembled into the electronic device.

[0204] Another aspect of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the electronic device performs the video tag recognition method as described above in the above embodiments.

[0205] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0206] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0207] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person of ordinary skill in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.< / eoc> < / pad> < / eoc> < / pad> < / eoc> < / pad> < / pad> < / eoc> < / eoc> < / pad> < / pad> < / eoc> < / pad> < / pad> < / eoc> < / pad> < / pad> < / pad> < / pad> < / pad>

Claims

1. A video tag recognition method, characterized in that: include: Acquire a video to be identified, and perform multimodal feature extraction on the video to be identified to obtain a first multimodal feature, wherein the first multimodal feature includes multiple modal features corresponding to multiple modalities, and the first multimodal feature includes a first video frame feature; Retrieving multiple reference videos similar to the video to be identified according to the first multimodal feature, performing multimodal feature extraction on each of the reference videos to obtain a second multimodal feature, wherein the second multimodal feature includes a second video frame feature, and each of the reference videos carries a video tag; Performing feature conversion processing on the first video frame features to obtain a first visual text label sequence, and performing feature conversion processing on the second video frame features corresponding to each of the reference videos to obtain a second visual text label sequence corresponding to each of the reference videos; Acquire first text information corresponding to the video to be identified and second text information corresponding to each of the reference videos; Taking a second visual text tag sequence and a second text information corresponding to the same reference video as input, and a video label as output, the input and output are concatenated to obtain an input-output learning example; splicing learning examples corresponding to multiple reference videos with a first visual text tag sequence and first text information of the video to be identified to obtain context learning information, wherein the context learning information includes context information provided by the learning examples, which is prompt information with analogy learning; The video tag of the video to be identified is identified according to the context learning information.

2. The method according to claim 1, characterized in that The number of the first video frame features includes a plurality; performing feature conversion processing on the first video frame features to obtain a first visual text tag sequence includes: Performing feature fusion processing on the plurality of first video frame features to obtain target video features; The target video features are aligned to a preset text feature space through a pre-trained feature alignment module to obtain the first visual text tag sequence.

3. The method according to claim 1, characterized in that: The second visual text mark sequence and the second text information corresponding to the same reference video are used as input, the video label is used as output, and the input and output are concatenated to obtain an input-output learning example; The context learning information is obtained by concatenating the learning examples corresponding to the multiple reference videos with the first visual text tag sequence and the first text information of the video to be identified, including: Constructing a learning example based on a first text tag for isolating different information under the same video, and the second visual text tag sequence corresponding to the same reference video, the second text information, and the video label to obtain multiple learning examples corresponding to multiple reference videos; splicing the multiple learning examples to obtain learning information, wherein the multiple learning examples in the learning information are separated by a second text tag for isolating information of different videos; constructing a recognition example based on the first textual token, the first sequence of visual textual tokens, and the first textual information; The context learning information is generated based on the learning information and the recognition example.

4. The method according to claim 1, characterized in that: The obtaining of the first text information corresponding to the video to be identified and the second text information corresponding to each of the reference videos includes: Obtaining the video title of the video to be identified and the identification text obtained by converting the audio information of the video to be identified; Generating a context description of the video to be identified according to the video scene of the video to be identified; The first text information is generated according to the video title, the recognition text and the situation description.

5. The method according to claim 1, characterized in that: The first multimodal feature includes a target video feature and a first text feature; retrieving a plurality of reference videos similar to the video to be identified according to the first multimodal feature includes: Obtaining a pre-established video feature retrieval library and a text feature retrieval library, wherein the video feature retrieval library includes a mapping relationship between each candidate video and a video feature corresponding to the candidate video, and the text feature retrieval library includes a mapping relationship between each candidate video and a text feature corresponding to the candidate video; The video feature retrieval library and the text feature retrieval library are searched respectively according to the target video feature, and the video feature retrieval library and the text feature retrieval library are searched respectively according to the first text feature to obtain a plurality of target candidate videos similar to the video to be identified; The reference video is selected from the multiple target candidate videos according to the video similarities between the multiple target candidate videos and the video to be identified.

6. The method according to claim 5, characterized in that The multiple target candidate videos include multiple target candidate videos retrieved by each retrieval method; and selecting the reference video from the multiple target candidate videos according to the video similarity between the multiple target candidate videos and the video to be identified includes: For each of the multiple target candidate videos retrieved by the retrieval method, calculating the similarity between the video to be identified and each target candidate video; For each target candidate video, calculating the average similarity between the target candidate video and the video to be identified in each retrieval method; The reference video is selected from the multiple target candidate videos according to the average similarity.

7. The method according to claim 1, characterized in that The step of extracting a multimodal feature from the video to be identified to obtain a first multimodal feature includes: Extracting a plurality of video frames from the video to be identified, and dividing the plurality of video frames into a plurality of segments; Extracting a target video frame from the multiple clips, and performing video feature extraction on the target video frame to obtain a first video frame feature; Acquire first text information corresponding to the video to be identified, and perform text feature extraction on the first text information to obtain a first text feature; The first multimodal feature is obtained according to the first video frame feature and the first text feature.

8. The method according to any one of claims 1 to 7, characterized in that: The step of identifying the video tag of the video to be identified according to the context learning information includes: Performing sequence feature conversion on the context learning information to obtain a context learning sequence, wherein the sequence feature is an input feature supported by a label generation model; Inputting the context learning sequence into the label generation model, the label generation model is obtained by freezing the original model parameters of the preset language model and adjusting the newly added model parameters of the language model according to the sample context learning sequence, the newly added model parameters being related to the low-rank adaptive module introduced into the language model; Obtain a target video label of the video to be identified output by the label generation model.

9. The method according to claim 8, characterized in that The training step of the label generation model includes: Obtaining a first sample video feature, a first sample text information, and a first sample video label corresponding to the first sample video, and a second sample video feature, a second sample text information, and a second sample video label corresponding to the second sample video; Performing feature alignment processing on the first sample video feature and the second sample video feature according to the pre-trained initial feature alignment module to obtain a first sample visual mark sequence and a second sample visual mark sequence, and constructing sample context learning information according to the first sample visual mark sequence, the first sample text information and the first sample video label, and the second sample visual mark sequence and the second sample text information; Performing sequence feature conversion on the sample context learning information to obtain the sample context learning sequence; Introducing the low-rank adaptive module into the language model, so as to introduce the newly added model parameters into the language model through the low-rank adaptive module; The original model parameters of the language model are kept frozen, and the newly added model parameters of the language model are adjusted according to the sample context learning sequence and the second sample video label to obtain the label generation model.

10. The method according to claim 9, characterized in that Before performing feature alignment processing on the video features according to the pre-trained initial feature alignment module, the method further includes: Obtaining a sample image and a sample description text corresponding to the sample image; Extracting features from the sample image to obtain sample visual features, and inputting the sample visual features into a module to be trained, so that the module to be trained aligns the sample visual features to an input text feature space of the language model; Acquire sample text features corresponding to the alignment description instruction, and input the sample text features and the target sample visual text features output by the feature alignment module into the language model, wherein the model parameters of the language model remain frozen; The sample prediction description text output by the language model is obtained, and the module to be trained is trained according to the sample description text and the sample prediction description text to obtain the initial feature alignment module.

11. The method according to claim 9, characterized in that The adjusting the newly added model parameters of the language model according to the sample context learning sequence to obtain the label generation model includes: Inputting the sample context learning sequence into the language model introduced into the low-rank adaptive module, and obtaining the predicted sample label output by the language model; According to the difference between the predicted sample label and the second sample video label, adjusting the newly added model parameters of the language model to obtain the label generation model; The method further comprises: According to the difference between the predicted sample label and the second sample video label, the module parameters of the initial feature alignment module are adjusted.

12. A video processing device, characterized in that: include: An acquisition module, used for acquiring a video to be identified, and performing multimodal feature extraction on the video to be identified to obtain a first multimodal feature, wherein the first multimodal feature includes multiple modal features corresponding to multiple modalities, and the first multimodal feature includes a first video frame feature; A retrieval module, configured to retrieve a plurality of reference videos similar to the video to be identified according to the first multimodal feature, perform multimodal feature extraction on each of the reference videos to obtain a second multimodal feature, wherein the second multimodal feature includes a second video frame feature, and each of the reference videos carries a video tag; A construction module is used to perform feature conversion processing on the first video frame features to obtain a first visual text label sequence, and perform feature conversion processing on the second video frame features corresponding to each of the reference videos to obtain a second visual text label sequence corresponding to each of the reference videos; obtain the first text information corresponding to the video to be identified and the second text information corresponding to each of the reference videos; take the second visual text label sequence and the second text information corresponding to the same reference video as input, and the video label as output, and splice the input and output to obtain an input-output learning example; splice the learning examples corresponding to multiple reference videos with the first visual text label sequence and the first text information of the video to be identified to obtain context learning information, wherein the context learning information includes context information provided by the learning example, which is prompt information with analogy learning; The recognition module is used to recognize the video tag of the video to be recognized according to the context learning information.

13. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to execute the method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor of an electronic device, the electronic device executes the method according to any one of claims 1 to 11.

15. A computer program product, characterized in that The computer program product comprises a computer program, wherein the computer program is stored in a computer-readable storage medium, and a processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device executes the method according to any one of claims 1 to 11.

Images