Language video model training and video processing methods, equipment, media and products

Abstract

The embodiment of the present invention provides a language video model training and video processing method, device, medium and product, which belongs to the field of AI. The method includes: obtaining a training sample video and its corresponding detailed description text and summary description text; determining a first text feature of the detailed description text, a second text feature of the summary description text, and a first video feature of the training sample video through a text encoder and a video encoder in the language video model; performing principal component extraction on the first video feature to obtain a second video feature based on a first similarity between the first text feature and the second text feature; determining a first loss function value based on the first video feature and the first text feature, and determining a second loss function value based on the second video feature and the second text feature; training the language video model based on the first loss function value and the second loss function value, so that the language video model has a good ability to understand long video description texts.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of artificial intelligence technology, and in particular, to a language video model training and video processing method, device, medium and product. Background Art

[0002] The main goal of the Contrastive Language-Image Pre-training (CLIP) model is to learn matching images and texts through contrastive learning. During the training process, the CLIP model learns to encode images and texts into a unified vector space, which enables it to understand the relationship between them both linguistically and visually. In this way, the CLIP model can be used in application scenarios such as zero-shot classification, text-image retrieval, and text-to-image generation.

[0003] In the training process of the original CLIP model, images and the corresponding summary description text of the images are usually used as training samples for training. However, compared with images, the continuous image frames of videos contain a lot of detailed information and other information, such as the order and development direction of activities, the direction of camera movement, etc. In this application scenario, the video CLIP model trained using the original CLIP model training method is difficult to accurately capture the complex relationships and attributes in the video image frames, such as the positional relationship between objects, detailed features in multiple dimensions, etc.

[0004] Therefore, it is necessary to propose a new training method for language video models to enhance the language video model's ability to understand videos. Summary of the invention

[0005] Embodiments of the present invention provide a language video model training and video processing method, device, medium and product to enhance the language video model's ability to understand videos.

[0006] In a first aspect, an embodiment of the present invention provides a video generation method, the method comprising:

[0007] Obtain a training sample video and a detailed description text and a summary description text corresponding to the training sample video;

[0008] Determine, by means of a text encoder included in the language video model, a first text feature corresponding to the detailed description text and a second text feature corresponding to the summary description text, respectively, and determine, by means of a video encoder included in the language video model, a first video feature corresponding to the training sample video;

[0009] Based on the first similarity between the first text feature and the second text feature, the first video feature is subjected to principal component extraction to reconstruct a second video feature corresponding to the training sample video based on the extracted plurality of principal component attribute features, wherein the second similarity between the second video feature and the second text feature matches the first similarity;

[0010] Determine a first loss function value according to the first video feature and the first text feature, and determine a second loss function value according to the second video feature and the second text feature, wherein the first loss function value and the second loss function value are both contrastive learning loss function values;

[0011] The language video model is trained according to the first loss function value and the second loss function value.

[0012] In a second aspect, an embodiment of the present invention provides a video processing method, the method comprising:

[0013] Obtain video description text and video dataset to be retrieved;

[0014] Determining textual characteristics of the video description text by a text encoder in a trained language video model;

[0015] Determine video features of multiple videos in the video dataset respectively by using a video encoder in the language video model;

[0016] Determine the similarities between the text features and the video features of the plurality of videos respectively, so as to output a target video whose similarity satisfies a set condition;

[0017] Wherein, the language video model is trained by the language video model training method described in the first aspect.

[0018] In a third aspect, an embodiment of the present invention provides a video processing method, the method comprising:

[0019] Obtain the target video and the text dataset to be retrieved;

[0020] Determine text features of multiple description texts in the text dataset respectively by using a text encoder in the trained language video model;

[0021] Determining video features of the target video by a video encoder in the language video model;

[0022] Determine the similarities between the video features and the text features of the plurality of description texts, respectively, to output a target description text whose similarity satisfies a set condition;

[0023] Wherein, the language video model is trained by the language video model training method described in the first aspect.

[0024] In a fourth aspect, an embodiment of the present invention provides a language video model training device, the device comprising:

[0025] An acquisition module, used to acquire a training sample video and a detailed description text and a summary description text corresponding to the training sample video;

[0026] A first determination module is used to determine a first text feature corresponding to the detailed description text and a second text feature corresponding to the summary description text respectively through a text encoder included in the language video model, and to determine a first video feature corresponding to the training sample video through a video encoder included in the language video model;

[0027] A reconstruction module, for extracting principal components from the first video feature based on a first similarity between the first text feature and the second text feature, so as to reconstruct a second video feature corresponding to the training sample video based on the extracted plurality of principal component attribute features, wherein a second similarity between the second video feature and the second text feature matches the first similarity;

[0028] A second determination module is used to determine a first loss function value according to the first video feature and the first text feature, and to determine a second loss function value according to the second video feature and the second text feature, wherein the first loss function value and the second loss function value are both contrastive learning loss function values;

[0029] A training module is used to train the language video model according to the first loss function value and the second loss function value.

[0030] In a fifth aspect, an embodiment of the present invention provides a video processing device, the device comprising:

[0031] An acquisition module is used to acquire video description text and video data sets to be retrieved;

[0032] A determination module, configured to determine text characteristics of the video description text by using a text encoder in a trained language video model; and to determine video features of a plurality of videos in the video data set by using a video encoder in the language video model;

[0033] An output module, used to determine the similarities between the text features and the video features of the plurality of videos, respectively, to output a target video whose similarity meets a set condition;

[0034] Wherein, the language video model is trained by the language video model training method as described in the first aspect.

[0035] In a sixth aspect, an embodiment of the present invention provides a video processing device, the device comprising:

[0036] An acquisition module is used to acquire the target video and the text dataset to be retrieved;

[0037] A determination module, configured to respectively determine text features of a plurality of descriptive texts in the text data set by using a text encoder in a trained language video model; and to determine video features of the target video by using a video encoder in the language video model;

[0038] An output module, used for determining the similarity between the video feature and the text features of the plurality of description texts respectively, so as to output a target description text whose similarity meets a set condition;

[0039] Wherein, the language video model is trained by the language video model training method as described in the first aspect.

[0040] In the seventh aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a communication interface; wherein, the memory stores executable code, and when the executable code is executed by the processor, the processor can at least implement the language video model training method or video processing method described in the above aspects.

[0041] In an eighth aspect, an embodiment of the present invention provides a non-transitory machine-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor can at least implement the language video model training method or the video processing method as described in the above aspects.

[0042] In the ninth aspect, an embodiment of the present invention provides a computer program product, which includes a computer program, and when the computer program product is executed by a processor of an electronic device, the processor can at least implement the language video model training method or video processing method described in the above aspects.

[0043] In the solution provided by the embodiment of the present invention, when training the language video model, the training data used includes the training sample video and the detailed description text and summary description text corresponding to the training sample video, wherein the detailed description text includes more rich detailed description content than the summary description text, and therefore has a longer length. Afterwards, the first text feature corresponding to the detailed description text and the second text feature corresponding to the summary description text are determined respectively by the text encoder contained in the language video model, and the first video feature corresponding to the training sample video is determined by the video encoder contained in the language video model. Afterwards, the first video feature is extracted based on the first similarity between the first text feature and the second text feature, so as to reconstruct the second video feature corresponding to the training sample video based on the extracted multiple principal component attribute features, wherein the second similarity between the second video feature and the second text feature matches the first similarity. Extracting the principal component attribute features of the first video feature that meet the first similarity requirement based on the guidance of the above-mentioned first similarity can dynamically adapt to the drastic distribution changes of the high-dimensional feature space during the model learning process (that is, the distribution of high-dimensional features of different training sample videos may change drastically). Finally, a first loss function value is determined according to the first video feature and the first text feature, and a second loss function value is determined according to the second video feature and the second text feature, and the language video model is trained according to the first loss function value and the second loss function value, wherein the first loss function value and the second loss function value are both comparative learning loss function values. It can be seen that through the above scheme, the language video model can better learn the distribution of the feature space and enhance the ability to understand the detailed description text, thereby improving the performance and applicability of the language video model in the video scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 A schematic diagram of the composition of a video generation system provided by an embodiment of the present invention;

[0046] Figure 2 A flow chart of a language video model training method provided by an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of collecting a training sample data set provided by an embodiment of the present invention;

[0048] Figure 4A logic diagram of a language video model training method provided by an embodiment of the present invention;

[0049] Figure 5 A flowchart of another language video model training method provided by an embodiment of the present invention;

[0050] Figure 6 A flowchart of another language video model training method provided by an embodiment of the present invention;

[0051] Figure 7 A flowchart of another language video model training method according to an embodiment of the present invention;

[0052] Figure 8 A flowchart of a video processing method provided by an embodiment of the present invention;

[0053] Fig. 9 A flowchart of another video processing method provided by an embodiment of the present invention;

[0054] Fig.10 A structural schematic diagram of a language video model training device provided by an embodiment of the present invention;

[0055] Fig.11 A schematic diagram of the structure of a video processing device provided by an embodiment of the present invention;

[0056] Fig.12 A schematic diagram of the structure of a video processing device provided by an embodiment of the present invention;

[0057] Fig.13 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0060] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the case where there is no conflict between the embodiments, the following embodiments and the features in the embodiments can be combined with each other. In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0061] First, some terms involved in the embodiments of the present invention are explained.

[0062] Video Understanding: Video understanding is a complex computer vision and machine learning task that aims to enable computers to understand video content like humans. This includes identifying objects, activities, scenes in videos, and events, emotions, and intentions in video sequences. Video understanding is widely used in video surveillance, autonomous driving, video search, content recommendation and other fields.

[0063] Contrastive Learning (CL): Contrastive learning is an unsupervised or semi-supervised learning method that learns the feature representation of samples by comparing similar samples (positive samples) with dissimilar samples (negative samples). In this process, the model is trained to reduce the distance between similar samples and expand the distance between dissimilar samples. Contrastive learning has shown good results in fields such as natural language processing, image recognition, and sound analysis.

[0064] Contrastive Language-Image Pre-training (CLIP): is a class of deep learning models that are specifically designed to process and understand tasks that involve both text descriptions and images. These models are usually based on the Transformer architecture and are pre-trained on a large scale to learn the association between text and images.

[0065] Text-similarity-guided Primary Component Matching (TPCM): is a method that combines text similarity analysis and principal component analysis to find similar patterns or topics in text data and map them to a low-dimensional space for easy analysis and visualization.

[0066] Principal Component Extraction (PCE): is a specific implementation of principal component analysis. Its goal is to find the most important directions (principal components) of data features, which can capture most of the variability in the data.

[0067] ViCLIP (Vision and Language CLIP) is a CLIP model used in video scenes. It combines the capabilities of vision and language and expands and improves the CLIP model.

[0068] Multimodal pretraining: refers to the pretraining process on multiple different modalities such as natural language processing, computer vision and audio processing. This technology aims to enable multimodal pretraining models to understand and process multiple types of data at the same time, such as text, images and audio, so as to better complete multimodal tasks such as image-text matching and visual question answering. Multimodal pretraining models improve the generalization ability and efficiency of the model on specific tasks by learning the intrinsic connections between different modalities.

[0069] Large Language Model (LLM): refers to a deep learning model trained with a large amount of text data that can generate natural language text or understand the meaning of language text. Large language models can handle a variety of natural language tasks, such as text classification, question answering, and dialogue.

[0070] CLIP is a key development in the field of vision-language pre-training. It integrates a text encoder and an image encoder through contrastive learning to align the two modalities of text and image. This method is widely used in application scenarios such as zero-shot classification, text-image retrieval, and text-to-image generation. Usually, in the training of the original CLIP model, the training samples used are sample images and the corresponding summary description text of the sample images. In the summary description text, the content of the sample image is briefly described. For example, the sample image shows a scene of a girl running, and the summary description text corresponding to the sample image may describe the sample image as "a girl running".

[0071] It can be seen that the sample image is only briefly described in the summary description text, which forces the image encoder in the CLIP model to focus mainly on the main features of the input image and ignore smaller but potentially crucial details. In addition, the text encoder of the CLIP model relies on position embedding, and the short description text limits the CLIP model's ability to understand long texts. Compared with images, this problem is particularly evident when processing videos because videos usually contain rich details. This makes it difficult to accurately capture the complex relationships and properties of objects in video image frames when training a video CLIP model in a video scene according to the original CLIP model training method.

[0072] In view of this, the embodiment of the present invention provides a new model training idea, by constructing a training data set for training, taking the training sample video, the detailed description text corresponding to the training sample video and the summary description text as pre-training data, obtaining the first text feature of the detailed description text corresponding to the training sample video, the second text feature of the summary description text corresponding to the training sample video and the first video feature of the training sample video, and taking the first similarity between the first text feature and the second text feature as a guide, the first video feature is extracted as the principal component, thereby reconstructing the second video feature corresponding to the training sample video based on the extracted multiple principal component attribute features, and then constructing a contrastive learning loss function value for contrastive learning based on the first text feature, the second text feature, the first video feature and the second video feature, and training the language video model according to the contrastive learning loss function value. The core of the model training idea is to add the detailed description text of the training sample video to the pre-training data, and use the principal component matching guided by the text similarity, so that the language video model can better learn the distribution of the feature space, enhance the ability to understand the detailed description text, and thus improve the performance and applicability of the language video model in more complex video scenes. The language video model in the embodiment of the present invention can be a video CLIP model applied to video scenes.

[0073] The language video model of the embodiment of the present invention can be used in application scenarios such as video retrieval and video description retrieval. For example, in a video retrieval scenario, the video description text to be retrieved and the video data set can be used as the input of the language video model, so that it outputs a target video whose similarity meets a preset condition. For another example, in a video description retrieval scenario, the target video and the text data set to be retrieved can be used as the input of the language video model, so that it outputs a target description text whose similarity meets the set condition.

[0074] The language video model training method, video processing method, etc. provided in the embodiments of the present invention below can be executed by a setting service provided by the cloud, and the service can be deployed in a cloud server or a cloud server cluster.

[0075] Take the language video model training scheme as an example. Figure 1 As shown, the embodiment of the present invention may provide a server for language video model training, wherein the server may be the cloud server or cloud server cluster.

[0076] In one example, the server may be server 1000, such as Figure 1As shown, the server includes a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, and an input device 1600. Although the server may also include a speaker, a microphone, etc., these components are irrelevant to the present invention and are omitted here. Among them, the processor 1100 may be, for example, a central processing unit CPU, a microprocessor MCU, etc. The memory 1200 may include, for example, a ROM (read-only memory), a RAM (random access memory), a non-volatile memory such as a hard disk, etc. The interface device 1300 may include, for example, a USB interface, a serial interface, etc. The communication device 1400 may be capable of wired or wireless communication, for example. The display device 1500 may be, for example, a liquid crystal display. The input device 1600 may include, for example, a touch screen, a keyboard, etc.

[0077] Figure 1 The server shown is merely illustrative and does not in any way imply any limitation on the present invention, its application or use. In the embodiment of the present invention, the memory 1200 in the server 1000 is used to store instructions for controlling the processor 1100 to operate to perform the language video model training method provided by the embodiment of the present invention. It should be understood by those skilled in the art that although Figure 1 In the figure, multiple devices of the server 1000 are shown, but the present invention may only involve some of the devices, for example, the server 1000 only involves the processor 1100 and the storage device 1200. A technician can design instructions according to the scheme disclosed in the present invention. How instructions control the processor to operate is well known in the art, so it will not be described in detail here.

[0078] The training process of the language video model and the video processing process provided by the embodiment of the present invention are described in detail below in combination with the following embodiments.

[0079] Figure 2 A flow chart of a language video model training method provided by an embodiment of the present invention, such as Figure 2 As shown, the method may include the following steps:

[0080] 201. Obtain a training sample video and a detailed description text and a summary description text corresponding to the training sample video.

[0081] 202. Determine, through a text encoder included in the language video model, a first text feature corresponding to the detailed description text and a second text feature corresponding to the summary description text, and determine, through a video encoder included in the language video model, a first video feature corresponding to the training sample video.

[0082] 203. Based on the first similarity between the first text feature and the second text feature, perform principal component extraction on the first video feature to reconstruct a second video feature corresponding to the training sample video based on the extracted multiple principal component attribute features, wherein the second similarity between the second video feature and the second text feature matches the first similarity.

[0083] 204. Determine a first loss function value according to the first video feature and the first text feature, and determine a second loss function value according to the second video feature and the second text feature, wherein the first loss function value and the second loss function value are both contrastive learning loss function values.

[0084] 205. Train a language video model according to the first loss function value and the second loss function value.

[0085] In the process of training a language video model, a large number of video-description text pairs are usually required. The description text includes a shorter summary description text and a longer detailed description text. The summary description text is used to describe the content of the corresponding video in a short text, while the detailed description text describes more details in the video content. However, in practical applications, video data sets with a large number of detailed description texts are still relatively scarce. Based on this, an automatic data collection scheme is provided in an embodiment of the present invention. In the automatic data collection scheme of the embodiment of the present invention, a training sample data set is collected using multiple video sources. In the training sample data set, each training sample data includes a training sample video and a detailed description text and a summary description text corresponding to the training sample video. In the summary description text corresponding to the training sample video, the main content of the training sample video is described, and in the detailed description text corresponding to the training sample video, the details of the training sample video are described. For example, a running video is shown in the training sample video. The content described in the summary description text corresponding to the training sample video can be: "A person is running", and the content described in the detailed description text corresponding to the training sample video can be: "A long-haired girl wearing black sportswear and black sports shoes is running in the park".

[0086] In the embodiment of the present invention, when collecting the training sample data set, multiple video sources are used, and the videos from the multiple video sources are processed respectively to form the training sample data. Specifically, Figure 3 As shown, the data source mainly includes at least one of the following: a video narration dataset, a video instruction tuning dataset, a video dataset with summary description text, and a video dataset with detailed description text.

[0087] Among them, video narrative data usually contains descriptions generated by annotators, which can describe the entire scene, main activities, and events involving multiple actors and objects. In the video narrative dataset composed of video narrative data, for each video contained therein, individual-level description text of the main objects in the video and their background can be annotated. The objects can include people, animals, objects, etc.

[0088] In order to make the video narration data in the video narration dataset more suitable for training the language video model, when obtaining the training sample video and the detailed description text and summary description text corresponding to the training sample video, the first training sample video and the multiple individual-level description texts corresponding to the first training sample video can be obtained from the video narration dataset. Among them, the multiple individual-level description texts correspond to the multiple objects contained in the first training sample video, and each individual-level description text is used to describe the visual feature information presented by the corresponding object in the first training sample video. The first training sample video is any video in the video narration dataset.

[0089] After obtaining the first training sample video and multiple individual-level description texts corresponding to the first training sample video, the multiple individual-level description texts are aggregated into an overall-level description text through a large language model, and a summary description text corresponding to the overall-level description text is generated, wherein the overall-level description text contains common information in the multiple individual-level description texts.

[0090] The individual level description text refers to the description of the individual objects in the first training sample video, mainly describing the individual information. The overall level description describes all the objects in the first training sample video as a whole, mainly describing the common information between all the objects.

[0091] For example, the first training sample video shows a scene with two people. Assume that both are skateboarding. In this case, there will be individual-level description texts corresponding to the two people. Each individual-level description text will describe the corresponding individual in the scene, such as what kind of clothes the corresponding individual wears and how he skateboards in what kind of environment. The overall description text will describe the common information of the two people in the scene, such as describing the two boys skateboarding in what kind of environment.

[0092] In the above embodiment, a large language model is pre-trained, and the aggregation process of multiple individual-level description texts to the overall-level description text is completed through the large language model. In addition, the same large language model or another pre-trained large language model can be used to input the overall-level description text into the large language model, so that it performs summary generation processing to output the corresponding summary description text.

[0093] For the video instruction tuning dataset, when obtaining the training sample video and the detailed description text and summary description text corresponding to the training sample video, the target task data is obtained from the video instruction tuning dataset. The target task data is the data of any task among the multiple tasks covered by the video instruction tuning dataset. Each task data includes a video and a question-answer pair corresponding to the video. After that, the question-answer pair in the target task data is sent to the large language model so that the large language model determines whether the target task data corresponds to the video summary generation task. If so, it is determined that the video in the target task data is the second training sample video of the language video model, and the reply information in the question-answer pair in the target task data is the detailed description text corresponding to the second training sample video. Then, the summary description text corresponding to the second training sample video is generated based on the reply information.

[0094] Among them, the task data of various tasks covered by the video instruction tuning dataset include, for example, the task data of the creative question-answering task, the task data of the video summary generation task, the task data of the generative question task, etc. In this embodiment, only the task data of the video summary generation task is required, so it is necessary to determine whether the above target task data is the task data of the video summary generation task through the large language model. That is to say, in this embodiment, the filtering of data irrelevant to the video summary generation task in the video instruction tuning dataset is realized through the large language model.

[0095] Specifically, when the large language model is used to determine whether the target task data corresponds to the video summary generation task, the prompt word template used may include the following prompt information: Task description information - please determine whether the input question and answer pair corresponds to the video summary generation task, if so, output yes, otherwise output no. Example prompt information - give at least one question and answer pair example corresponding to the video summary generation task, and at least one question and answer pair example that does not correspond to the video summary generation task can also be given. Based on the above prompt word template, the question and answer pairs contained in the currently obtained target task data can be filled into the prompt word template to obtain the corresponding prompt word, which is then input into the large language model to obtain the judgment result output by the large language model: whether the target task data corresponds to the video summary generation task.

[0096] If it is determined that the target task data corresponds to a video summary generation task, the video in the target task data is determined to be the second training sample video of the language video model, and the answer information in the question-answer pair in the target task data is used as the detailed description text corresponding to the second training sample video. Afterwards, the answer information is input into another large language model, and the large language model performs summary generation processing to obtain the summary description text corresponding to the answer information as the summary description text corresponding to the second training sample video.

[0097] For a set video data set with summary description text, when obtaining a training sample video and a detailed description text and a summary description text corresponding to the training sample video, a third training sample video and a summary description text corresponding to the third training sample video are obtained from the set video data set, a plurality of key frames are extracted from the third training sample video, detailed description texts corresponding to the plurality of key frames are generated through a set multimodal model, and the summary description text corresponding to the third training sample video and the detailed description texts corresponding to the plurality of key frames are input into a large language model to generate a detailed description text corresponding to the third training sample video. The third training sample video is any video in the set video data set.

[0098] It is understood that for the third training sample video, the key frame extraction strategy is not specifically limited, and the key frames may be extracted from the third training sample video in an evenly spaced manner, or other key frame identification methods may be used to extract the key frames. The number of extracted key frames is less than the number of video frames contained in the third training sample video.

[0099] The summary description text corresponding to the third training sample video and the detailed description texts corresponding to multiple key frames are input into the large language model to generate the detailed description text corresponding to the third training sample video. With the assistance of the summary description text, the hallucination phenomenon in the detailed description text of the key frames can be alleviated, making the detailed description text corresponding to the third training sample video more accurate.

[0100] For a video dataset with detailed description text, when obtaining training sample videos and detailed description text and summary description text corresponding to the training sample videos, a certain number of videos are randomly sampled from the dataset to be determined as the fourth training sample videos of the language video model, and the detailed description text data corresponding to the videos are determined as the detailed description text corresponding to the fourth training sample video of the language video model. At the same time, the detailed description text corresponding to the fourth training sample video is sent to the large language model, so that the large language model outputs the summary description text corresponding to the fourth training sample video.

[0101] Furthermore, in order to expand the training sample data, for the aforementioned first training sample video and the detailed description text and summary description text corresponding to the first training sample video, the second training sample video and the detailed description text and summary description text corresponding to the second training sample video, the third training sample video and the detailed description text and summary description text corresponding to the third training sample video, and the fourth training sample video and the detailed description text and summary description text corresponding to the fourth training sample video, the detailed description texts and / or summary description texts corresponding to these training sample videos can be respectively rewritten with similar semantics through a large language model to obtain another description text with the same or similar semantics.

[0102] In practical applications, after obtaining the training sample video and the detailed description text and summary description text corresponding to the training sample video, the training sample video that does not meet the requirements can also be filtered. Optionally, the training sample video whose similarity between the video feature and the text feature of the description text is lower than a set value can be filtered out by using the existing CLIP model used in the video scene (such as the ViCLIP model) and the CLIP model used in the image scene (such as the Long-CLIP model) to obtain the final training sample data set. Wherein, when the CLIP model used in the video scene is used, any of the above-collected training sample videos and its corresponding detailed description text can be input into the model to obtain the corresponding video features and text features, and the similarity between the two is calculated. If the similarity is lower than the set value, the training sample video is filtered out. Wherein, when the CLIP model used in the image scene is used, the multiple frames of images contained in any of the above-collected training sample videos and their corresponding detailed description text can be input into the model to obtain the image features corresponding to each frame of the image and the text features corresponding to the detailed description text. The average of multiple image features can be calculated as the video feature, and the similarity between the video feature and the text feature is calculated. If the similarity is lower than the set value, the training sample video is filtered out.

[0103] Usually, a language video model includes a text encoder and a video encoder, where the text encoder is used to extract text features from the description text, and the video encoder is used to extract video features from the video. Figure 4 , after obtaining the training sample video And detailed description text corresponding to the training sample video and summary description text Afterwards, the training samples are viewed as The video and the detailed description text corresponding to the training sample video and summary description text Input to the language video model. The detailed description text is obtained by the text encoder in the language video model. Extract the first text feature , from the abstract description text Extract the second text feature , and from the training sample video through the video encoder in the language video model Extract the first video feature .

[0104] In the detailed description text Extract the first text feature , from the abstract description text Extract the second text feature , and from the training sample video Extract the first video feature After that, principal component matching based on text similarity guidance (i.e., based on the first text feature With the second text feature The first similarity between , Second text feature and the first video feature Get the second video feature .

[0105] In practical applications, the PCE method is used to In the process of principal component extraction, PCE includes component decomposition function (F), component filtering function (E) and component reconstruction function (F -1 Among them, the component decomposition function (F) is used to decompose the features into attribute features of different importance, the component filtering function (E) is used to filter out unimportant attribute features, and the component reconstruction function (F -1 ) is used to reconstruct the remaining attribute features after filtering. It should be noted that in the related art, when implementing the filter function (E), a fixed value is usually selected, such as 32 most important attribute features to be retained. However, in the scenario of language video model training, since the training sample videos contain more content and details, the strategy of selecting a fixed number of the most important attribute features to be retained cannot adapt to the drastic distribution changes in the high-dimensional feature space of different training sample videos during the learning process, resulting in the language video model being unable to align the semantics of the description text with the training sample video.

[0106] Based on this, in the embodiment of the present invention, it is proposed to use the first text feature and the second text feature The first similarity between As a signal to guide the PCE process. Specifically, first calculate the first text feature and the second text feature The first similarity between Then, based on the first similarity, the first video feature Extract the principal components and reconstruct the second video features corresponding to the training sample video based on the extracted multiple principal component attribute features , where the second video feature With the second text feature The second similarity between them matches the first similarity. Wherein, both the first similarity and the second similarity can be calculated based on the cosine similarity calculation method. That is, the first similarity The guiding role of is reflected in: the conditions that the multiple principal component attribute features finally selected need to meet are: the second video feature reconstructed based on multiple principal component attribute features With the second text feature The second similarity between the two should match the first similarity, where the match can be equal to or greater than the first similarity. With summary description text The corresponding second text feature The similarity calculation is performed because the reconstructed second video features Mainly contains training sample videos The main features of the text, while the abstract describes It is used to describe the training sample video The main content of.

[0107] In a possible implementation, the first text feature With the second text feature The first similarity between As a guide, the first video features Perform principal component extraction to reconstruct the second video feature corresponding to the training sample video based on the extracted multiple principal component attribute features Specifically, the first video feature may be subjected to principal component extraction to obtain N principal component attribute features. The N principal component attribute features are extracted in sequence according to the degree of importance.

[0108] After obtaining N principal component attribute features, the features used to reconstruct the second video are determined through the following iterative process Multiple principal component attribute features: According to the order of N principal component attribute features, the first K principal component attribute features are selected. The video features corresponding to the training sample video are reconstructed based on the K principal component attribute features. If the video features reconstructed based on the K principal component attribute features are consistent with the second text feature The similarity between them is lower than the first similarity , then expand the range of K principal component attribute features, K is less than or equal to N. In this way, through continuous iterative attempts, multiple principal component attribute features that meet the above conditions can be obtained.

[0109] After getting the first text feature , Second text feature , First video features And the second video feature Afterwards, according to the first video feature and the first text feature Determine the first loss function value , and according to the second video feature and the second text feature Determine the second loss function value Among them, the first loss function value and the second loss function value are all contrast loss function values. Then, according to the first loss function value and the second loss function value , training language video models.

[0110] Specifically, the total loss function value can be determined according to the first loss function value and the second loss function value: , the language video model is trained according to the total loss function value. represents the hyperparameter, The calculation process of is as described above. Based on the scheme provided by the above embodiment of the present invention, by obtaining the first text feature of the detailed description text corresponding to the training sample video, the second text feature of the summary description text corresponding to the training sample video and the first video feature of the training sample video, and guided by the first similarity between the first text feature and the second text feature, the first video feature is subjected to principal component extraction, thereby reconstructing the second video feature corresponding to the training sample video based on the extracted multiple principal component attribute features, and then constructing a contrastive learning loss function value for contrastive learning based on the first text feature, the second text feature, the first video feature and the second video feature, and training the language video model according to the contrastive learning loss function value, so that the language video model can better learn the distribution of the video feature space, enhance the ability to understand the detailed description text, and thus improve the performance and applicability of the language video model in the video scene.

[0111] In the embodiment of the present invention, it is believed that a language video model with excellent performance can not only better understand the detailed description text of the video, but also have the following characteristics: given a video and its corresponding detailed description text, the model should give a higher similarity score result to the following description: at the same level of detail, a more accurate and less likely to produce hallucinations description. In order to achieve this goal, in the process of training the language video model, the embodiment of the present invention also introduces the Hallucination-aware Description Ranking (HDR) task, such as Figure 5 As shown, another language video model training method provided by an embodiment of the present invention may include the following steps:

[0112] 501. Generate a set of hallucination perception description texts by gradually increasing the hallucination degree of the detailed description texts corresponding to the training sample videos, wherein the set of hallucination perception description texts includes a plurality of detailed description texts with different hallucination degrees.

[0113] 502. Determine text features corresponding to a plurality of detailed description texts respectively through a text encoder.

[0114] 503. Determine third similarities between text features corresponding to the plurality of detailed description texts and the first video feature respectively.

[0115] 504. Determine a first similarity difference between a third similarity corresponding to the first detailed description text and a third similarity corresponding to the second detailed description text, wherein the first detailed description text and the second detailed description text are any two of a plurality of detailed description texts, and an illusion degree of the first detailed description text is lower than an illusion degree of the second detailed description text.

[0116] 505. Determine a third loss function value with the goal of making the first similarity difference greater than a first set threshold.

[0117] 506. Train the language video model according to the first loss function value, the second loss function value, and the third loss function value.

[0118] Specifically, before increasing the hallucination degree of the detailed description text corresponding to the training sample video, it is necessary to first use a grammatical analysis tool to perform part-of-speech tagging on the detailed description text and parse the grammatical structure of the detailed description text. Optionally, the grammatical analysis tool can be NLTK (Natural Language Toolkit) and spaCy (Natural Language Processing Tool Library). In actual applications, other grammatical analysis tools can also be selected according to needs. The embodiment of the present invention does not specifically limit the type of grammatical analysis tools.

[0119] After tagging the detailed description text with parts of speech and parsing the grammatical structure of the detailed description text, we can selectively use semantically different words that belong to the same syntactic category to replace specific words, such as replacing nouns, numbers, colors, or direction-related terms, verbs, etc., and perform this replacement multiple times, thereby gradually increasing the hallucination degree of the detailed description text corresponding to the training sample video and generating a set of hallucination perception description texts. Among them, m represents the text set describing hallucination perception The number of multiple detailed descriptions with different degrees of hallucination.

[0120] In a specific example, the process of replacing words in the detailed description text may be:

[0121] The original detailed description text reads: Two boys are seen interacting in a setting with a floor, benches, a ceiling, aboard, a wall, and curtains as the backdrop. One of the boys is wearing a white t-shirt and moves to the left before throwing an object towards the other boy, who is wearing a blue t-shirt andstands facing him.

[0122] The Chinese translation of the original detailed description text is: Two boys interact in an environment with a floor, bench, ceiling, blackboard, wall, and curtain background. A boy wearing a white T-shirt moves to the left and throws an object at another boy wearing a blue T-shirt and facing him.

[0123] The first replacement result hypothesis is: Two girls are seen interacting in a setting with a floor, benches, a ceiling, a board, a window, and curtains as the backdrop. One of the girls is wearing a bluejacket and moves to the left before throwing an object towards the other girl, who is wearing a blue jacket andstands facing her.

[0124] The Chinese translation of the first replacement result is: Two girls interact in an environment with a floor, bench, ceiling, blackboard, windows and curtains. A girl wearing a blue jacket moves to the left and throws an object at another girl wearing a blue jacket and facing him. The bold font indicates the replacement content.

[0125] The second replacement result is assumed to be: Four girls are seen interacting in a setting with a refrigerator, benches, a ceiling, a board, a window, and curtains as the backdrop. One of the girls is wearing a blue dress and moves to the right before lifting an object towards the other girl, who is wearing a red dress and stands facing her.

[0126] The Chinese translation of the second replacement result is: Four girls interact in an environment with a refrigerator, bench, ceiling, wooden board, window and curtain as the background. One of the girls is wearing a blue dress, moving to the right, and then holding an object to another girl, and the other is wearing a red dress and standing in front of her.

[0127] As you can see, in the process of replacing words, "boy" is replaced by "girl", "wall" is replaced by "window", "white" is replaced by "blue", and "T-shirt" is replaced by "jacket". Then, based on the first replacement result, "two" is replaced by "four", "jacket" is replaced by "dress", "blue" is replaced by "red", "throw" is replaced by "lift", and so on. The illusion of describing the text in detail is gradually increased.

[0128] After generating a plurality of detailed description texts with different degrees of hallucination, the text features corresponding to the plurality of detailed description texts are respectively determined by a text encoder, and the third similarities between the text features corresponding to the plurality of detailed description texts and the first video feature are respectively determined.

[0129] Specifically, any two first detailed description texts in a plurality of detailed description texts are used. and the second detailed description text For example, assuming that the first detailed description text The level of hallucination is lower than the second detailed description text The degree of hallucination is determined by the first detailed description text Text features With the first video feature The third similarity between , and determine the second detailed description text Text features With the first video feature The third similarity between .

[0130] After that, calculate the first detailed description text The corresponding third similarity With the second detailed description text The corresponding third similarity The first similarity difference between .

[0131] Next, the first similarity difference Greater than the first set threshold As the target, determine the value of the third loss function .

[0132] Among them, the formula for the third loss function value is as follows:

[0133] ;

[0134] Among them, ReLU is the activation function, in the first similarity difference Greater than the first set threshold When the activation function ReLU value is ; In the first similarity difference Less than or equal to the first set threshold When the activation function value ReLU is 0.

[0135] The language video model is trained according to the first loss function value, the second loss function value and the third loss function value. Specifically, the total loss function value can be determined as , the language video model is trained according to the total loss function value. Among them, represents a hyperparameter.

[0136] The technical solution of the embodiment of the present invention can enable the language video model to better learn the distribution of the feature space and enhance the ability to understand the detailed description text, while further enabling the language video model to have the ability to recognize more accurate and less likely to produce hallucinations descriptions, thereby further improving the performance of the language video model.

[0137] The embodiment of the present invention also believes that a language video model with excellent performance can not only better understand the detailed description text of the video, but also should have the following characteristics: given a video and its corresponding detailed description text, the model should give a higher similarity score result to the following description: the description text with richer and more accurate context. In order to achieve this goal, in the process of training the language video model, the embodiment of the present invention also introduces the detail-aware description ranking (DDR) task, such as Figure 6 As shown, another language video model training method provided by an embodiment of the present invention may include the following steps:

[0138] 601. Generate a detail-aware description text set by gradually deleting detail description information in the detailed description text corresponding to the training sample video, wherein the detail-aware description text set includes multiple detailed description texts with different levels of detail.

[0139] 602. Determine text features corresponding to the multiple detailed description texts respectively through a text encoder;

[0140] 603. Determine fourth similarities between text features corresponding to the plurality of detailed description texts and the first video feature respectively.

[0141] 604. Determine a second similarity difference between a fourth similarity corresponding to the first detailed description text and a fourth similarity corresponding to the second detailed description text, wherein the first detailed description text and the second detailed description text are any two of a plurality of detailed description texts, and a detail level of the first detailed description text is higher than a detail level of the second detailed description text.

[0142] 605. Determine a fourth loss function value with the goal of making the second similarity difference greater than a second set threshold.

[0143] 606. Train the language video model according to the first loss function value, the second loss function value, the third loss function value, and the fourth loss function value.

[0144] Specifically, before gradually deleting the detailed description information in the detailed description text corresponding to the training sample video, it is necessary to first use a grammatical analysis tool to perform part-of-speech tagging on the detailed description text and parse the grammatical structure of the detailed description text. Optionally, the grammatical analysis tool can be NLTK (Natural Language Toolkit) and spaCy (Natural Language Processing Tool Library). In actual applications, other grammatical analysis tools can also be selected according to needs. The embodiment of the present invention does not specifically limit the type of grammatical analysis tools.

[0145] After tagging the detailed description text with parts of speech and parsing the grammatical structure of the detailed description text, the clauses, adjectives, numbers or dependency parsing subtrees in the current detailed description text can be randomly deleted. Through this process, a set of detail-aware description texts with gradually reduced details can be recursively generated. . Where m represents the detail-aware description text set The number of multiple long description texts with different levels of detail in the .

[0146] In a specific example, the process of deleting words in the detailed description text may be:

[0147] The original detailed description text reads: Two boys are seen interacting in a setting with a floor, benches, a ceiling, aboard, a wall, and curtains as the backdrop. One of the boys is wearing a white t-shirt and moves to the left before throwing an object towards the other boy, who is wearing a blue t-shirt andstands facing him.

[0148] The Chinese translation of the original detailed description text is: Two boys interact in an environment with a floor, bench, ceiling, blackboard, wall and curtain background. A boy wearing a white T-shirt moves to the left and throws an object at another boy wearing a blue T-shirt and facing him. Among them, bold fonts indicate content that needs to be deleted.

[0149] The first deletion result is assumed to be: Boys are seen interacting in a setting. One of the boys is wearing a t-shirt and moves to the left before throwing an objecttowards the other boy, who stands facing him.

[0150] The Chinese translation of the first deletion result is: Boys interact in an environment. A boy wearing a T-shirt moves to the left and throws an object at another boy facing him.

[0151] The second deletion result is assumed to be: Boys are seen interacting in a setting. One of the boys is wearing a t-shirt and moves before throwing an object towards the other boy.

[0152] The Chinese translation of the first deletion result is: Boys interact in an environment. A boy wearing a T-shirt moves and throws an object at another boy.

[0153] It can be seen that in the process of deleting details, the detailed description sentences such as "two, with floor, bench, ceiling, blackboard, wall and curtain background, white, wearing blue T-shirts and" were deleted first, and then the detailed description sentences such as "to the left, facing him" were deleted. The detailed description information in the detailed description text was gradually reduced.

[0154] After generating a plurality of detailed description texts with different levels of detail, the text features corresponding to the plurality of detailed description texts are respectively determined by a text encoder, and the fourth similarities between the text features corresponding to the plurality of detailed description texts and the first video feature are respectively determined.

[0155] Specifically, any two first detailed description texts among the multiple detailed description texts are used. and the second detailed description text For example, assuming that the first detailed description text The level of detail is higher than the second detailed description text The level of detail, determines the first detailed description text Text features With the first video feature The fourth similarity between , and determine the second detailed description text Text features With the first video feature The fourth similarity between .

[0156] After that, calculate the first detailed description text The corresponding fourth similarity With the second detailed description text The corresponding fourth similarity The second similarity difference between .

[0157] Next, the second similarity difference Greater than the second set threshold As the target, determine the fourth loss function value .

[0158] Among them, the formula for the fourth loss function value is as follows:

[0159] ;

[0160] Among them, ReLU is the activation function, and in the second similarity difference Greater than the second set threshold When the activation function ReLU value is ; In the second similarity difference Less than or equal to the second set threshold When the activation function value ReLU is 0.

[0161] Afterwards, the language video model may be trained according to the first loss function value, the second loss function value, and the fourth loss function value. Alternatively, the language video model may be trained according to the first loss function value, the second loss function value, the third loss function value, and the fourth loss function value.

[0162] Specifically, the total loss function value can be determined , the language video model is trained according to the total loss function value. Among them, represents a hyperparameter.

[0163] The technical solution of the embodiment of the present invention can enable the language video model to better learn the distribution of the feature space and enhance the ability to understand the detailed description text, while further enabling the language video model to have the ability to recognize the description text with richer contextual details, thereby further improving the performance of the language video model.

[0164] The language video model trained by the language video model training method according to the embodiment of the present invention should have the ability to distinguish correct text from incorrect text in the detailed description text. Therefore, in order to better evaluate this ability of the language video model, in the embodiment of the present invention, a benchmark dataset for long video description ranking (LVDR) is proposed to test this ability of the language video model using the test sample video in the benchmark dataset and the detailed description text corresponding to the test sample video.

[0165] Figure 7 FIG. 1 is a flow chart of another language video model training method according to an embodiment of the present invention. Figure 7 As shown, the method may include the following steps:

[0166] 701. Obtain a test sample video and a detailed description text corresponding to the test sample video.

[0167] 702. Generate a test description text set by gradually increasing the error degree of the detailed description text corresponding to the test sample video, wherein the test description text set includes a plurality of detailed description texts with different error degrees.

[0168] 703. Determine text features corresponding to a plurality of detailed description texts respectively through a text encoder, and determine a third video feature corresponding to the test sample video through a video encoder.

[0169] 704. Determine fifth similarities between text features corresponding to the plurality of detailed description texts and the third video feature respectively.

[0170] 705. Determine a third similarity difference between a fifth similarity corresponding to the first detailed description text and a fifth similarity corresponding to the second detailed description text, wherein the first detailed description text and the second detailed description text are any two of a plurality of detailed description texts, and an error level of the first detailed description text is lower than an error level of the second detailed description text.

[0171] 706. If the third similarity difference is greater than a third set threshold, updating the count value used to represent the number of correct recognitions, so as to complete the test of the language video model based on the updated count value.

[0172] In the embodiment of the present invention, some test sample videos and detailed description texts corresponding to the test sample videos are randomly selected. Then, an operation similar to the above embodiment of increasing the illusion degree of the detailed description text is performed, and the error degree of the detailed description text corresponding to each test sample video is gradually increased, iterated p-1 times, and q words are modified in each iteration, so as to generate a test description text set corresponding to the test sample video, wherein the test description text set includes multiple detailed description texts with different error degrees, p is an integer greater than 1, and q is an integer greater than or equal to 1.

[0173] After generating the test description text set, the text features corresponding to the multiple detailed description texts are respectively determined by the text encoder, and the third video feature corresponding to the test sample video is determined by the video encoder.

[0174] Next, fifth similarities between the text features corresponding to the plurality of detailed description texts and the third video feature are determined respectively.

[0175] Among them, taking any two first detailed description texts and second detailed description texts corresponding to the same test sample video as an example, the fifth similarity between the text feature corresponding to the first detailed description text and the third video feature is recorded as , the fifth similarity between the text feature corresponding to the second detailed description text and the third video feature is recorded as .

[0176] The number of correct identifications is counted using the following formula:

[0177] ;

[0178] in, is the indicator function, used in Greater than When the count value increases by 1, Less than or equal to When , the count value increases by 0. m represents the total number of detailed description texts contained in the test description text set.

[0179] After the test is completed and the count value is obtained, the ability of the language video model to distinguish the correct text from the incorrect text in the detailed description text is evaluated according to the count value. Specifically, assuming that the error level of the first detailed description text is lower than the error level of the second detailed description text, the expected result of the indicator function is: Greater than , and if it is found that the actual output indication result is not so, the above count value will not be increased by 1. Therefore, for m detailed description texts, the above similarity size comparison is performed every two. If the model can completely and accurately distinguish the error degree of different detailed description texts, then the final count value will be m(m-1) / 2. If the actual count result is less than this value, it means that the model can continue to be optimized and trained.

[0180] Figure 8 A flowchart of a video processing method provided by an embodiment of the present invention is shown in FIG. Figure 8 As shown, the method may include the following steps:

[0181] 801. Obtain video description text and a video dataset to be retrieved.

[0182] 802. Determine text features of the video description text using a text encoder in a trained language video model.

[0183] 803. Determine video features of multiple videos in the video dataset respectively through a video encoder in the language video model.

[0184] 804. Determine similarities between the text features and the video features of the plurality of videos, respectively, to output a target video whose similarity satisfies a set condition.

[0185] The language video model is trained by the language video model training method of the aforementioned embodiment.

[0186] In this embodiment, the video description text and the video data set to be retrieved are used as inputs of the language video model, so that the text encoder in the language video model determines the text features corresponding to the video description text, and at the same time, the video encoder in the language video model respectively determines the video features of multiple videos in the video data set to be retrieved. The video description text can be a detailed description text or a summary description text.

[0187] After obtaining the text features corresponding to the video description text and the video features corresponding to the multiple videos in the video data set to be retrieved, the language video model calculates the similarities between the text features and the multiple video features respectively, and then obtains the target video that meets the set conditions based on the comparison result of the similarity and the set conditions and outputs the target video.

[0188] For example, suppose that a user needs to find a video that best reflects the content described in the video description text based on the video description text. At this time, there are M videos, and these M videos can all be used to reflect the content of the video description text. It is difficult for the user to determine which video can more accurately reflect the content of the video description text. At this time, the video description text and the video data set to be retrieved are input into the trained language video model, and the language video model is requested to determine the video that best reflects the video description text from the M videos.

[0189] The language video model determines the text features corresponding to the video description text through the text encoder, and determines the video features of M videos through the video encoder, and then calculates the similarity between the text features and the M video features, and sorts the M similarities. The video with the highest similarity to the text features of the video description text or a similarity greater than a set threshold is determined as the target video, and the language video model outputs the target video. In this way, the language video model can help users determine the target video that best reflects the video description text.

[0190] The specific training method of the language video model in this embodiment can refer to the relevant description in the aforementioned embodiment, which will not be repeated here.

[0191] According to the technical solution of this embodiment, in a video retrieval scenario, a target video that is more accurate and better matched with the video description text can be obtained.

[0192] Fig. 9 A flowchart of another video processing method provided by an embodiment of the present invention is shown in FIG. Fig. 9 As shown, the method may include the following steps:

[0193] 901. Obtain a target video and a text dataset to be retrieved.

[0194] 902. Determine text features of multiple description texts in the text dataset respectively through the text encoder in the trained language video model.

[0195] 903. Determine video features of the target video using a video encoder in a language video model.

[0196] 904. Determine the similarities between the video features and the text features of the plurality of description texts, respectively, to output a target description text whose similarity satisfies a set condition.

[0197] The language video model is trained by the language video model training method of the aforementioned embodiment.

[0198] In this embodiment, the target video and the text data set to be retrieved are used as inputs of the language video model, so that the text encoder in the language video model determines the text features of multiple descriptive texts in the text data set respectively, and the video encoder in the language video model determines the video features of the target video.

[0199] After determining the text features of multiple description texts and the video features of the target video, the language video model calculates the similarity between the video feature distribution and the text features of the multiple description texts, compares the similarity with the set conditions, determines the description text that meets the set conditions as the target description text, and outputs the target description text.

[0200] For example, suppose that a user needs to accurately describe a target video. At this time, there are N description texts, all of which are used to describe the target video. It is difficult for the user to determine which description text can more accurately describe the target video. At this time, a text dataset consisting of the target video and the N description texts is input into a trained language video model, and the language video model is requested to determine the description text that most accurately describes the target video from the N description texts.

[0201] The language video model determines the text features corresponding to the N description texts through the text encoder, and determines the video features of the target video through the video encoder, and then calculates the similarity between the text features of the N description texts and the video features of the target video, and sorts the N similarities, and determines the description text with the highest similarity to the video features of the target video as the target description text, and the language video model outputs the target description text. In this way, the language video model can help users determine the description text that can most accurately describe the target video.

[0202] The specific training method of the language video model in this embodiment can refer to the relevant description in the aforementioned embodiment, which will not be repeated here.

[0203] According to the technical solution of this embodiment, the description text that best matches the target video can be determined from the text data set to be retrieved.

[0204] Fig.10 Schematic diagram of the structure of a language video model training device provided by an embodiment of the present invention. Fig.10 As shown, the device 10 includes: an acquisition module 11, a first determination module 12, a reconstruction module 13, a second determination module 14 and a training module 15.

[0205] The acquisition module 11 is used to acquire the training sample video and the detailed description text and summary description text corresponding to the training sample video;

[0206] A first determination module 12 is used to determine the first text feature corresponding to the detailed description text and the second text feature corresponding to the summary description text respectively through the text encoder included in the language video model, and to determine the first video feature corresponding to the training sample video through the video encoder included in the language video model;

[0207] A reconstruction module 13, for extracting principal components from the first video features based on the first similarity between the first text features and the second text features, so as to reconstruct second video features corresponding to the training sample video based on the extracted plurality of principal component attribute features, wherein the second similarity between the second video features and the second text features matches the first similarity;

[0208] A second determination module 14 is used to determine a first loss function value according to the first video feature and the first text feature, and to determine a second loss function value according to the second video feature and the second text feature, wherein the first loss function value and the second loss function value are both contrastive learning loss function values;

[0209] The training module 15 is used to train the language video model according to the first loss function value and the second loss function value.

[0210] Fig.10 The device shown can execute the steps in the language video model training method in the aforementioned embodiment. The detailed execution process and technical effects are described in the aforementioned embodiment and will not be repeated here.

[0211] Fig.11 FIG. 1 is a schematic diagram of the structure of a video processing device provided by an embodiment of the present invention. Fig.11 As shown, the device 20 includes: an acquisition module 21, a determination module 22 and an output module 23.

[0212] The acquisition module 21 is used to acquire the video description text and the video data set to be retrieved.

[0213] The determination module 22 is used to determine the text characteristics of the video description text through the text encoder in the trained language video model; and to determine the video characteristics of multiple videos in the video data set through the video encoder in the language video model.

[0214] The output module 23 is used to determine the similarities between the text features and the video features of the multiple videos, so as to output a target video whose similarity meets a set condition.

[0215] The language video model is trained by the language video model training method in the aforementioned embodiment.

[0216] Fig.11 The device shown can execute the steps of the video processing method in the aforementioned embodiment. The detailed execution process and technical effects can be found in the description of the aforementioned embodiment, which will not be repeated here.

[0217] Fig.12 FIG. 1 is a schematic diagram of the structure of a video processing device provided by an embodiment of the present invention. Fig.12 As shown, the device 30 includes: an acquisition module 31, a determination module 32 and an output module 33.

[0218] The acquisition module 31 is used to acquire the target video and the text data set to be retrieved.

[0219] The determination module 32 is used to respectively determine the text characteristics of multiple descriptive texts in the text data set through the text encoder in the trained language video model; and determine the video characteristics of the target video through the video encoder in the language video model.

[0220] The output module 33 is used to determine the similarities between the video features and the text features of the plurality of description texts, so as to output a target description text whose similarity meets a set condition.

[0221] The language video model is trained by the language video model training method in the aforementioned embodiment.

[0222] Fig.12 The device shown can execute the steps of the video processing method in the aforementioned embodiment. The detailed execution process and technical effects can be found in the description of the aforementioned embodiment, which will not be repeated here.

[0223] The embodiment of the present invention also provides an electronic device, such as Fig.13 As shown, the electronic device 40 may include: a processor 41, a memory 42, and a communication interface 43. The memory 42 stores executable code, and when the executable code is executed by the processor 41, the processor 41 can at least implement the language video model training method or video processing method provided in the aforementioned embodiments.

[0224] In addition, an embodiment of the present invention provides a non-temporary machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor can at least implement a language video model training method or a video processing method as provided in the aforementioned embodiments.

[0225] The device embodiments described above are merely illustrative, and the network elements described as separate components may or may not be physically separated. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art may understand and implement the solution without creative effort.

[0226] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software. Based on such an understanding, the above technical solution can essentially or in other words be embodied in the form of a computer product, and the present invention can be implemented in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0227] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A language video model training method, characterized in that: include: Obtain a training sample video and a detailed description text and a summary description text corresponding to the training sample video; Determine, by means of a text encoder included in the language video model, a first text feature corresponding to the detailed description text and a second text feature corresponding to the summary description text, respectively, and determine, by means of a video encoder included in the language video model, a first video feature corresponding to the training sample video; Based on the first similarity between the first text feature and the second text feature, the first video feature is subjected to principal component extraction to reconstruct a second video feature corresponding to the training sample video based on the extracted plurality of principal component attribute features, wherein the second similarity between the second video feature and the second text feature matches the first similarity; Determine a first loss function value according to the first video feature and the first text feature, and determine a second loss function value according to the second video feature and the second text feature, wherein the first loss function value and the second loss function value are both contrastive learning loss function values; Training the language video model according to the first loss function value and the second loss function value; The extracting of principal components of the first video feature based on the first similarity between the first text feature and the second text feature, so as to reconstruct the second video feature corresponding to the training sample video based on the extracted plurality of principal component attribute features, comprises: Performing principal component extraction on the first video feature to obtain N principal component attribute features; The plurality of principal component attribute features used to reconstruct the second video feature are determined by the following iterative process: According to the order of the N principal component attribute features, the first K principal component attribute features are screened out; Reconstructing the video features corresponding to the training sample video based on the K principal component attribute features; If the similarity between the video feature reconstructed based on the K principal component attribute features and the second text feature is lower than the first similarity, the range of the K principal component attribute features is expanded, and K is less than or equal to N.

2. The method according to claim 1, characterized in that The method further comprises: By gradually increasing the hallucination degree of the detailed description text corresponding to the training sample video, a hallucination perception description text set is generated, wherein the hallucination perception description text set includes a plurality of detailed description texts with different hallucination degrees; Determine, by means of the text encoder, text features corresponding to the plurality of detailed description texts respectively; respectively determining a third similarity between the text features corresponding to the plurality of detailed description texts and the first video feature; Determining a first similarity difference between a third similarity corresponding to a first detailed description text and a third similarity corresponding to a second detailed description text, wherein the first detailed description text and the second detailed description text are any two of the plurality of detailed description texts, and an illusion degree of the first detailed description text is lower than an illusion degree of the second detailed description text; Determine a third loss function value with the goal of making the first similarity difference greater than a first set threshold; The step of training the language video model according to the first loss function value and the second loss function value includes: The language video model is trained according to the first loss function value, the second loss function value, and the third loss function value.

3. The method according to any one of claims 1 to 2, characterized in that: The method further comprises: Generate a detail-aware description text set by gradually deleting detail description information in the detailed description text corresponding to the training sample video, wherein the detail-aware description text set includes a plurality of detailed description texts with different levels of detail; Determine, by means of the text encoder, text features corresponding to the plurality of detailed description texts respectively; respectively determining fourth similarities between text features corresponding to the plurality of detailed description texts and the first video features; Determining a second similarity difference between a fourth similarity corresponding to a first detailed description text and a fourth similarity corresponding to a second detailed description text, wherein the first detailed description text and the second detailed description text are any two of the plurality of detailed description texts, and a detail level of the first detailed description text is higher than a detail level of the second detailed description text; Determine a fourth loss function value with the goal of making the second similarity difference greater than a second set threshold; The step of training the language video model according to the first loss function value and the second loss function value comprises: The language video model is trained according to the first loss function value, the second loss function value and the fourth loss function value.

4. The method according to claim 1, characterized in that The method further comprises: Obtain a test sample video and a detailed description text corresponding to the test sample video; By gradually increasing the error degree of the detailed description text corresponding to the test sample video, a test description text set is generated, wherein the test description text set includes a plurality of detailed description texts with different error degrees; Determine, by means of the text encoder, text features corresponding to the plurality of detailed description texts respectively, and determine, by means of the video encoder, a third video feature corresponding to the test sample video; respectively determining fifth similarities between text features corresponding to the plurality of detailed description texts and the third video feature; Determining a third similarity difference between a fifth similarity corresponding to a first detailed description text and a fifth similarity corresponding to a second detailed description text, wherein the first detailed description text and the second detailed description text are any two of the plurality of detailed description texts, and an error degree of the first detailed description text is lower than an error degree of the second detailed description text; If the third similarity difference is greater than a third set threshold, the count value used to represent the number of correct recognitions is updated to complete the test of the language video model based on the updated count value.

5. The method according to claim 1, characterized in that The step of obtaining a training sample video and a detailed description text and a summary description text corresponding to the training sample video includes: Acquire a first training sample video and a plurality of individual-level description texts corresponding to the first training sample video from a video narration dataset, wherein the plurality of individual-level description texts respectively correspond to a plurality of objects included in the first training sample video, and each individual-level description text is used to describe visual feature information of a corresponding object presented in the first training sample video; The multiple individual-level description texts are aggregated into an overall-level description text through a large language model, and a summary description text corresponding to the overall-level description text is generated, wherein the overall-level description text contains common information in the multiple individual-level description texts.

6. The method according to claim 1, characterized in that The step of obtaining a training sample video and a detailed description text and a summary description text corresponding to the training sample video includes: Acquire target task data from the video instruction tuning dataset, where the target task data is data of any task among multiple tasks covered by the video instruction tuning dataset, and each task data includes a video and a question-answer pair corresponding to the video; Sending the question-answer pairs in the target task data to the large language model so that the large language model determines whether the target task data corresponds to a video summary generation task; If so, determining that the video in the target task data is the second training sample video of the language video model, and the answer information in the question-answer pair in the target task data is the detailed description text corresponding to the second training sample video; Generate a summary description text corresponding to the second training sample video according to the reply information.

7. The method according to claim 1, characterized in that The step of obtaining a training sample video and a detailed description text and a summary description text corresponding to the training sample video includes: Acquire a third training sample video and a summary description text corresponding to the third training sample video from a set video data set; Extracting a plurality of key frames from the third training sample video; Generate detailed description texts corresponding to the multiple key frames through a set multimodal model; The summary description text corresponding to the third training sample video and the detailed description texts corresponding to the multiple key frames are input into the large language model to generate the detailed description text corresponding to the third training sample video.

8. The method according to claim 1, characterized in that The method further comprises: The detailed description text and / or the abstract description text corresponding to the training sample video is rewritten with similar semantics through a large language model to expand the training sample data.

9. A video processing method, characterized in that: The method comprises: Obtain video description text and video dataset to be retrieved; Determining text features of the video description text by a text encoder in a trained language video model; Determine video features of multiple videos in the video dataset respectively by using a video encoder in the language video model; Determine the similarities between the text features and the video features of the plurality of videos respectively, so as to output a target video whose similarity satisfies a set condition; Wherein, the language video model is trained by the language video model training method described in any one of claims 1-8.

10. A video processing method, characterized in that: The method comprises: Obtain the target video and the text dataset to be retrieved; Determine text features of multiple description texts in the text dataset respectively through a text encoder in the trained language video model; Determining video features of the target video by a video encoder in the language video model; Determine the similarities between the video features and the text features of the plurality of description texts, respectively, to output a target description text whose similarity satisfies a set condition; Wherein, the language video model is trained by the language video model training method described in any one of claims 1-8.

11. An electronic device, characterized in that: include: A memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the method as claimed in any one of claims 1 to 10.

12. A non-transitory machine-readable storage medium, characterized in that: The non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor is caused to perform the method according to any one of claims 1 to 10.

13. A computer program product, characterized in that include: A computer program, when executed by a processor of an electronic device, causes the processor to perform the method according to any one of claims 1 to 10.

Images