Video description text generation method and device, electronic equipment and readable medium

By combining multiple large models for generating descriptive text and constructing and fusing prompt text, the problems of missing details and low reliability in video descriptive text generation are solved, resulting in richer and more accurate video descriptions.

CN123115463APending Publication Date: 2026-11-17JD DIGITS HAIYI INFORMATION TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510637838.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-16
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing video description text generation solutions suffer from missing details and low information reliability. Traditional deep learning models may miss video content, and large models may cause text content bias and errors.

Method used

By combining multiple descriptive texts to generate a large model that outputs initial descriptive text, and then fusing them by constructing prompt texts, the final generation is performed using a pre-trained LLM, which fully extracts detailed information from the video content and alleviates the illusion of reasoning.

Benefits of technology

It improves the reliability and richness of video description text, avoids omissions, and enhances the diversity and accuracy of generated text.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115463A_ABST
    Figure CN123115463A_ABST
Patent Text Reader

Abstract

The present disclosure provides a video description text generation method and device, an electronic device and a readable medium, relating to the technical field of computers. The method can obtain an initial description text of a target video, wherein the initial description text is obtained based on the target video by a plurality of description text generation large models; on this basis, a first prompt text is constructed based on the initial description text, and the initial description text and the first prompt text are input into a target text generation large model, so that the target text generation large model fuses the initial description text based on the first prompt text to obtain an output target text. The method can combine the initial description text output by the plurality of description text generation large models, fully mine the video content detail information, and avoid omission; the target text generation large model fuses the plurality of initial description texts obtained, which can effectively alleviate the inference hallucination problem and improve the reliability of the target text.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and specifically to a method for generating video description text, a device for generating video description text, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Video captioning refers to the technique of summarizing or describing video content using abstract text to help users quickly understand the video content. It can also be applied in quiet environments or scenarios where users have hearing impairments. The core of video captioning technology lies in converting video content into natural language descriptive text, combining technological achievements from the fields of computer vision and natural language processing, such as visual feature extraction, recurrent neural networks, attention mechanisms, sequence-to-sequence models, multi-model fusion, pre-trained models, and so on.

[0003] Currently, different video description text generation schemes have their own limitations. For example, using traditional deep learning models may result in missing video description details, brief text content, and insufficient and incomplete extraction of video content; while using large models is limited by illusions, which may lead to biases and errors in the text content and low information reliability.

[0004] It should be noted that the information in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide a video description text generation method, a video description text generation device, an electronic device, and a computer-readable storage medium that can combine initial description texts output by multiple large description text generation models to fully extract detailed information from video content and avoid omissions; by fusing multiple initial description texts obtained from a single large target text generation model, the problem of inference illusion can be effectively alleviated and the reliability of the target text can be improved.

[0006] According to a first aspect of this disclosure, a method for generating video description text is provided. The method may include: obtaining initial description text of a target video, wherein the initial description text is obtained by multiple description text generation models based on the target video output; constructing a first prompt text based on the initial description text; and inputting the initial description text and the first prompt text into a target text generation model, such that the target text generation model fuses the initial description text based on the first prompt text to obtain the output target text.

[0007] In one exemplary embodiment, the descriptive text generation model includes a video descriptive text generation model. Obtaining the initial descriptive text of a target video includes: constructing a second prompt text based on the target video; and inputting the second prompt text and the target video into the video descriptive text generation model, so that the video descriptive text generation model outputs the initial descriptive text based on the second prompt text and the target video.

[0008] In one exemplary embodiment, constructing a second prompt text based on a target video includes: obtaining at least one example of descriptive text from a historical video; and constructing the second prompt text based on the target video and the example descriptive text.

[0009] In one exemplary embodiment, the descriptive text generation big model includes an image descriptive text generation big model, which obtains initial descriptive text for a target video, including: extracting keyframes from the target video; constructing third prompt text based on the keyframes; and inputting the third prompt text and the keyframes into the image descriptive text generation big model, so that the image descriptive text generation big model outputs initial descriptive text based on the keyframes and the third prompt text.

[0010] In one exemplary embodiment, constructing a third prompt text based on keyframes includes: obtaining at least one example of descriptive text for a keyframe in a historical video; and constructing the third prompt text based on the keyframe and the example of descriptive text for the keyframe.

[0011] In one exemplary embodiment, the descriptive text generation big model includes a video visual question answering (VQA) big model. Obtaining the initial descriptive text of the target video includes: constructing a first question text based on the target video; inputting the first question text and the target video into the video VQA big model, so that the video VQA big model outputs a corresponding first answer text based on the first question text on the target video, and uses the first answer text as the initial descriptive text corresponding to the video VQA big model.

[0012] In an exemplary embodiment, the descriptive text generation big model includes an image VQA big model. Obtaining the initial descriptive text of the target video includes: extracting keyframes from the target video; constructing a second question text based on the keyframes; inputting the second question text and the keyframes into the image VQA big model, so that the image VQA big model outputs a corresponding second answer text based on the second question text and the keyframes, and uses the second answer text as the initial descriptive text corresponding to the image VQA big model.

[0013] According to a second aspect of this disclosure, a video description text generation apparatus is provided. The apparatus may include: an initial text acquisition module for acquiring initial description text of a target video, wherein the initial description text is obtained by multiple description text generation models based on the target video output; a prompt text construction module for constructing a first prompt text based on the initial description text; and a target text fusion module for inputting the initial description text and the first prompt text into a target text generation model, so that the target text generation model fuses the initial description text based on the first prompt text to obtain the output target text.

[0014] According to a third aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to implement the above-described method by executing the executable instructions.

[0015] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the method described above.

[0016] The video description text generation method provided in this disclosure can obtain initial description text of a target video, which is obtained by multiple large-scale description text generation models based on the target video. Based on this, a first prompt text is constructed from the initial description text, and the initial description text and the first prompt text are input into a target text generation model. The target text generation model then fuses the initial description text with the first prompt text to obtain the output target text. This method can combine initial description texts output by multiple large-scale description text generation models to fully extract detailed information from the video content and avoid omissions. The fusion of multiple initial description texts by a single target text generation model effectively alleviates the problem of inference illusion and improves the reliability of the target text.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0019] Figure 1 This schematically illustrates one of the flowcharts of a video description text generation method in an exemplary embodiment of the present disclosure.

[0020] Figure 2 This schematically illustrates a second flowchart of the steps of a video description text generation method in an exemplary embodiment of this disclosure.

[0021] Figure 3 This schematic diagram illustrates the structural block diagram of a video description text generation apparatus in an exemplary embodiment of the present disclosure.

[0022] Figure 4 A schematic diagram of the composition of an electronic device to which the exemplary embodiments of the present disclosure may be applied is shown. Detailed Implementation

[0023] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0024] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0025] Figure 1 One of the steps of a video description text generation method according to an exemplary embodiment of this disclosure is shown. Figure 1 As shown, the video description text generation method of this disclosure embodiment may include steps 101 to 103:

[0026] In step 101, the initial description text of the target video is obtained. The initial description text is obtained by a large model for generating multiple description texts based on the output of the target video.

[0027] In this embodiment, the target video can be any video from which descriptive text is to be generated, and this embodiment does not impose specific restrictions on the source and type of the target video. Each large-scale text generation model can output corresponding initial descriptive text based on the target video. This initial descriptive text is used to describe and summarize the video content of the target video, such as a plot summary for a drama video, an outline summary for a science video, and product information for a promotional video. Different types of large-scale descriptive text generation models differ in model structure and training samples, and the knowledge distribution obtained after pre-training also differs, resulting in different focuses in the understanding of the target video and the generation of descriptive text. Based on this, by using multiple large-scale descriptive text generation models, based on the diverse understanding of the target video, multiple versions of generated text with different focuses can be fully explored to obtain the initial descriptive text of the target video.

[0028] In step 102, a first prompt text is constructed based on the initial description text.

[0029] In this embodiment of the disclosure, the first prompt text is used to instruct the target text generation model to fuse different versions of generated text in the initial description text. The first prompt text can be constructed based on the initial description text. The first prompt text may include explanatory prompts about the initial description text, explanatory prompts about the fusion method, and may further include explanatory prompts about the text format, such as the number of words and sentences. This embodiment of the disclosure does not impose specific limitations on these aspects.

[0030] In step 103, the initial description text and the first prompt text are input into the target text generation model, so that the target text generation model is fused with the first prompt text based on the initial description text to obtain the output target text.

[0031] In this embodiment, the target generation model can be a pre-trained LLM (Large Language Model), such as GPT-4, or it can be fine-tuned based on a pre-trained LLM to enable the LLM to better understand the text fusion task indicated by the first prompt text and fuse the initial description text to obtain the output target text. The target text can integrate multiple versions of generated text with different emphases from the initial description text, resulting in a more complete and richer description of the video content, meeting diverse needs.

[0032] Figure 2 A second flowchart illustrating the steps of a video description text generation method according to an exemplary embodiment of this disclosure is shown. Figure 2 As shown, the video description text generation method of this disclosure embodiment may include the following steps 201 to 212.

[0033] When the large model for describing text generation includes a large model for video description text generation, the method may include the following steps 201 to 202:

[0034] In step 201, a second prompt text is constructed based on the target video.

[0035] In step 202, the second prompt text and the target video are input into the video description text generation model, so that the video description text generation model outputs the initial description text based on the second prompt text on the basis of the target video.

[0036] In this embodiment of the disclosure, the large-scale video description text generation model can refer to a large-scale model that generates text from a video based on the prompt text. Large-scale video description text generation models typically perform well on videos with complex dynamics, capable of describing objects, spatial, and temporal information within the video. In one optional embodiment, the PLLaVA (Parameter-free LLaVA) algorithm can be used, or other large-scale video text generation models can be employed; this embodiment of the disclosure does not impose specific limitations on this approach.

[0037] Furthermore, the second prompt text constructed based on the target video can be based on an algorithm, a prompt template, or manually written. The second prompt text can include the following content to achieve the corresponding prompting effect:

[0038] Phrases such as "Describe this video" and "Describe this video" are used to indicate the task object and content of the large-scale video description text generation model.

[0039] "Pay attention to all objects in the video," "Pay attention to vehicles in the video," "Pay attention to pedestrians in the video," "Pay attention to the tableware used in the video," etc., to prompt the video description text generation model to focus on specific objects when generating text.

[0040] "The description should help the AI ​​regenerate the video" serves as a hint regarding the quality of the text generated by the large model for video description text generation.

[0041] The prompts, such as "The description should not exceed 20 sentences" and "The description should be between 300 and 400 words," provide guidance on the length of the text generated by the large model for generating video description text.

[0042] It should be noted that the content of the second prompt text is merely an illustrative example. Since the second prompt text affects the initial description text output by the large model for generating video description text, those skilled in the art can construct the second prompt text according to actual needs, such as selecting and adjusting the above content, or adding new content to achieve the desired prompt effect. This disclosure does not impose any specific limitations on this.

[0043] In this embodiment of the disclosure, based on the second prompt text, a video description text generation model can generate an initial description text corresponding to the video description text generation model on the target video based on the second prompt text. This initial description text also includes generated text output by other description text generation models, thus including different versions of generated text.

[0044] In an exemplary embodiment of this disclosure, step 202 may include the following steps A1 to A2:

[0045] In step A1, obtain at least one example of the descriptive text for a historical video.

[0046] In step A2, a second prompt text is constructed based on the target video and the example descriptive text.

[0047] In this embodiment of the disclosure, the historical video can be any video data from which descriptive text is generated before the target video. The descriptive text example is descriptive text obtained based on the historical video. This descriptive text example can be obtained from a large descriptive text generation model, or it can be obtained by fusing the outputs of multiple large descriptive text models, or it can be manually written. This embodiment of the disclosure does not impose specific limitations on this. Each historical video can correspond to one descriptive text example.

[0048] Having obtained descriptive text examples, additional descriptive text examples can be added to the second prompt text to help the video descriptive text generation model better understand the processing task. For example, the second prompt text can include at least one descriptive text example using phrases such as "Here are some good descriptive examples" or "You can refer to the following descriptive examples," as shown below:

[0049] 1. A stylish woman walks down a street in Tokyo, bathed in warm neon lights and animated city signs. She wears a black leather jacket, a long red skirt, and black boots, carrying a black handbag. She wears sunglasses and red lipstick. She walks with confidence and nonchalance. The street is damp and reflective, creating a mirror-like effect with the colorful lights. Many pedestrians stroll by.

[0050] 2. Several enormous woolly mammoths trot across the snow-covered grassland, their long manes fluttering in the wind. In the distance, snow-covered trees and magnificent snow-capped mountains rise, the afternoon light, combined with wispy clouds and the distant, high-hanging sun, creating a warm glow. The low-angle shot is breathtaking, capturing large, furry mammals with beautiful photographic effects and depth of field.

[0051] 3. Drone footage captures the scene of waves crashing against the rugged cliffs of Big Surgarai Beach. The azure waters churn with whitecaps, and the golden light of the setting sun illuminates the rocky shore. In the distance, a small island sits atop a lighthouse, its edges covered in green shrubs. The steep drop from the road to the beach is spectacular, with the cliff edges jutting out into the sea. This view captures the pristine beauty of the coastline and the rugged landscape of the Pacific Coast Highway.

[0052] The above-described text examples may be partially or entirely incorporated into the second prompt text, or other described text examples may be incorporated into the second prompt text. This disclosure does not impose specific limitations on this.

[0053] When describing a large text generation model, including an image-description large text generation model, the method may include the following steps 203 to 205:

[0054] In step 203, keyframes are extracted from the target video.

[0055] In step 204, a third prompt text is constructed based on the keyframe.

[0056] In step 205, the third prompt text and keyframes are input into the large model for generating image description text, so that the large model for generating image description text outputs the initial description text based on the third prompt text and the keyframes.

[0057] In this embodiment, the large-scale image description text generation model possesses more targeted knowledge, enabling more accurate descriptions of special, rare, or difficult-to-identify objects within the image. For example, in science videos, the model can more accurately identify and describe different tools, such as round-bottomed flasks and flat-bottomed flasks, avoiding the problem of failing to distinguish between them when focusing on dynamic video information, thus preventing the loss or omission of detailed information. The image description text generation model can deeply mine the detailed information of the target video, ensuring the diversity and richness of the generated text. In an optional embodiment, the LLaVA (Large Language and Vision Assistant) algorithm can be used, or other large-scale image-to-text models can be employed; this embodiment does not impose specific limitations on this.

[0058] In this embodiment of the disclosure, keyframes can be extracted from the target video as input to a large-scale image description text generation model. There can be one or more keyframes. Keyframes can be extracted or fused using algorithms to represent video content, or they can be directly extracted from image frames at specific moments, such as the starting image frame, intermediate moment image frames, and the last image frame. This embodiment of the disclosure does not impose specific limitations on this.

[0059] Furthermore, the third-party prompt text constructed based on keyframes can be based on algorithms, prompt templates, or manual writing. The third-party prompt text can include the following content to achieve the corresponding prompting effect:

[0060] "Describe this image," "Describe this image," etc., to prompt the task object and content of the large-scale image description text generation model.

[0061] The text provides prompts such as "Pay attention to all objects in the image," "Pay attention to vehicles in the image," "Pay attention to pedestrians in the image," and "Pay attention to the tableware used in the image" to highlight the object focus of the large model for generating text from images.

[0062] "The description should help the artificial intelligence regenerate the image" serves as a hint regarding the quality of the text generated by the large model for image description text generation.

[0063] The prompts, such as "The description should not exceed 20 sentences" and "The description should be between 300 and 400 words," provide guidance on the length of text generated by the large model for generating image description text.

[0064] It should be noted that the content of the third prompt text mentioned above is only an illustrative example. Since the third prompt text affects the initial description text output by the large model for generating image description text, those skilled in the art can construct the third prompt text according to actual needs, such as selecting and adjusting the above content, or adding new content to achieve the desired prompt effect. This disclosure does not impose any specific limitations on this.

[0065] In this embodiment of the disclosure, based on the obtained third prompt text, an image description text generation model can generate an initial description text corresponding to the image description text generation model on keyframes based on the third prompt text. This initial description text also includes generated text output by other description text generation models, thus including different versions of generated text.

[0066] In an exemplary embodiment of this disclosure, step 204 may include the following steps B1 to B2:

[0067] In step B1, obtain at least one example of descriptive text for a keyframe in a historical video.

[0068] In step B2, a third prompt text is constructed based on the keyframe and the example description text of the keyframe.

[0069] In this embodiment of the disclosure, the historical video and descriptive text examples can be referred to the relevant descriptions of steps A1 to A2 above, and the keyframes can be referred to the relevant descriptions of step 203 above. To avoid repetition, they will not be described again here.

[0070] When describing a large text generation model that includes a large video VQA (Visual Question Answering) model, the method may include the following steps 206 to 207:

[0071] In step 206, the first question text is constructed based on the target video.

[0072] In step 207, the first question text and the target video are input into the video VQA big model, so that the video VQA big model outputs the corresponding first answer text based on the first question text on the basis of the target video, and the first answer text is used as the initial description text corresponding to the video VQA big model.

[0073] In this embodiment, a large-scale VQA model can also be used to generate the initial descriptive text. The large-scale VQA model corresponds to open-ended question-answering tasks, outputting answer text given the input video content and question. The large-scale VQA model typically utilizes additional textual cues based on visual understanding, thereby enhancing the richness and comprehensiveness of the generated text. The video VQA model is used to extract relevant semantic information from the video content based on the question to output the answer.

[0074] Furthermore, the first question text constructed based on the target video can be based on an algorithm, a prompt template, or manually written. The first question text can include the following content to achieve the corresponding questioning effect:

[0075] Questions such as "What people are in the video?" and "What objects are in the video?" are asked to the video VQA big data model based on semantic information related to objects in the target video.

[0076] Questions such as "What's above / below the scene shown in the video?" and "What's to the east / west / south / north of the scene shown in the video?" are posed to the video VAQ big data model based on semantic information related to the location of the scene in the target video.

[0077] Questions such as "What are the people in the video doing?" and "What are the objects in the video doing?" are posed to the video VQA big data model based on semantic information related to motion state in the target video.

[0078] The questions "What changes have occurred to the people in the video?" and "What changes have occurred to the objects in the video?" are used to ask the video VQA big data model questions based on semantic information related to the changed state in the target video.

[0079] It should be noted that the content of the first question text mentioned above is only an illustrative example. Since the first question text affects the first answer text output by the video VQA large model, those skilled in the art can construct the first question text according to actual needs, such as selecting and adjusting the above content, or adding new content to achieve the desired questioning effect. This disclosure does not impose any specific limitations on this.

[0080] In this embodiment of the disclosure, based on the obtained first question text, the video VQA big model can output the first answer text corresponding to the video VQA big model on the target video as the initial description text. This initial description text also includes generated text output by other description text generation big models, thus including different versions of generated text.

[0081] When describing a large text generation model that includes an image VQA large model, the method may include steps 208 to 210 as follows.

[0082] In step 208, keyframes are extracted from the target video.

[0083] In step 209, a second question text is constructed based on the keyframes.

[0084] In step 210, the second question text and keyframes are input into the image VQA large model, so that the image VQA large model outputs the corresponding second answer text based on the second question text on the basis of the keyframes, and the second answer text is used as the initial description text corresponding to the image VQA large model.

[0085] In this embodiment of the disclosure, step 208 can be referred to the relevant description of step 203 above. To avoid repetition, it will not be repeated here.

[0086] In this embodiment of the disclosure, steps 209 to 210 can be referred to the relevant descriptions of steps 206 to 207 above. To avoid repetition, they will not be repeated here.

[0087] The method may also include steps 211 to 212.

[0088] In step 211, a first prompt text is constructed based on the initial description text.

[0089] In this embodiment of the disclosure, step 211 can be referred to the relevant description of step 102 above. To avoid repetition, it will not be repeated here.

[0090] The first prompt text, constructed based on the initial description text, can be built using an algorithm, a prompt template, or manually. The first prompt text may include the following content to achieve the corresponding prompt effect:

[0091] The text includes phrases such as "The above content is text generated by some large models" and "The following includes video description text generated by multiple large models," to indicate the target text generation model's task object and content.

[0092] The prompts include phrases like "Don't miss any information from each large model" and "Consider the generated text of each large model," to highlight the object's focus when generating fused text from the target text using large models.

[0093] "The correct side is judged by a comprehensive assessment of the conflicting statements between generated texts," and "The generated text with better quality is compared in pairs," to indicate how the target text generation model integrates different versions in the initial description text.

[0094] The prompts include "The description should not exceed 20 sentences," "The description should be between 300 and 400 words," and "Form a comprehensive and diverse text," providing guidance on the length and quality requirements of the target text.

[0095] It should be noted that the content of the first prompt text is merely an illustrative example. Since the first prompt text affects the target text output by the target text generation model, those skilled in the art can construct the first prompt text according to actual needs, such as selecting and adjusting the above content, or adding new content to achieve the desired prompt effect. This disclosure does not impose any specific limitations on this.

[0096] In step 212, the initial description text and the first prompt text are input into the target text generation model, so that the target text generation model is fused with the first prompt text based on the initial description text to obtain the output target text.

[0097] In this embodiment of the disclosure, step 212 can be referred to the relevant description of step 103 above. To avoid repetition, it will not be repeated here.

[0098] In this embodiment, video can be generated based on the output target text, and the quality of the target text can be evaluated based on the video generation result. On the same test set, according to the Vbench evaluation criteria, the fused target text shows a significant improvement compared to video description text generated by traditional deep learning models or independent large models.

[0099] The video description text generation method provided in this disclosure can obtain initial description text of a target video, which is obtained by multiple large-scale description text generation models based on the target video. Based on this, a first prompt text is constructed from the initial description text, and the initial description text and the first prompt text are input into a target text generation model. The target text generation model then fuses the initial description text with the first prompt text to obtain the output target text. This method can combine initial description texts output by multiple large-scale description text generation models to fully extract detailed information from the video content and avoid omissions. The fusion of multiple initial description texts by a single target text generation model effectively alleviates the problem of inference illusion and improves the reliability of the target text.

[0100] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0101] Further reference Figure 3 As shown in the exemplary embodiment of this disclosure, a structural block diagram of a video description text generation device 300 is provided. The device 300 may include: an initial text acquisition module 301, used to acquire initial description text of a target video, wherein the initial description text is obtained by multiple description text generation models based on the target video output; a prompt text construction module 302, used to construct a first prompt text based on the initial description text; and a target text fusion module 303, used to input the initial description text and the first prompt text into a target text generation model, so that the target text generation model fuses the initial description text based on the first prompt text to obtain the output target text.

[0102] In one exemplary embodiment, the descriptive text generation model includes a video descriptive text generation model and an initial text acquisition module 301, which is used to construct a second prompt text based on the target video; input the second prompt text and the target video into the video descriptive text generation model, so that the video descriptive text generation model outputs the initial descriptive text based on the second prompt text on the basis of the target video.

[0103] In one exemplary embodiment, the initial text acquisition module 301 is specifically used to acquire at least one example of descriptive text for a historical video; and to construct a second prompt text based on the target video and the example of descriptive text.

[0104] In one exemplary embodiment, the descriptive text generation big model includes an image descriptive text generation big model and an initial text acquisition module 301, which is used to extract keyframes from a target video; construct a third prompt text based on the keyframes; and input the third prompt text and the keyframes into the image descriptive text generation big model, so that the image descriptive text generation big model outputs the initial descriptive text based on the keyframes and the third prompt text.

[0105] In one exemplary embodiment, the initial text acquisition module 301 is specifically used to acquire at least one example of descriptive text for a keyframe in a historical video; and to construct a third prompt text based on the keyframe and the example of descriptive text for the keyframe.

[0106] In an exemplary embodiment, the descriptive text generation big model includes a video visual question answering (VQA) big model and an initial text acquisition module 301, which is used to construct a first question text based on a target video; input the first question text and the target video into the video VQA big model, so that the video VQA big model outputs a corresponding first answer text based on the first question text on the basis of the target video, and uses the first answer text as the initial descriptive text corresponding to the video VQA big model.

[0107] In an exemplary embodiment, the descriptive text generation big model includes an image VQA big model and an initial text acquisition module 301, which is used to extract keyframes from the target video; construct a second question text based on the keyframes; input the second question text and the keyframes into the image VQA big model, so that the image VQA big model outputs a corresponding second answer text based on the second question text on the basis of the keyframes, and uses the second answer text as the initial descriptive text corresponding to the image VQA big model.

[0108] The video description text generation apparatus provided in this disclosure can acquire initial description text of a target video. This initial description text is obtained by multiple large-scale description text generation models based on the target video. Based on this, a first prompt text is constructed from the initial description text. The initial description text and the first prompt text are then input into a target text generation model, allowing the target text generation model to fuse the initial description text with the first prompt text to obtain the output target text. This method can combine initial description texts output by multiple large-scale description text generation models to fully extract detailed information from the video content and avoid omissions. The fusion of multiple initial description texts by a single target text generation model effectively alleviates the problem of inference illusion and improves the reliability of the target text.

[0109] The specific details of each module in the above-mentioned video description text generation device have been described in detail in the method section of the implementation. For any undisclosed details, please refer to the implementation content of the method section. That is, the explanation and beneficial effects of the video description text generation method of the above-mentioned embodiments are also applicable to the video description text generation device of this disclosure, and will not be elaborated on here.

[0110] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0111] The following reference Figure 4 To describe an electronic device 400 according to such an embodiment of the present disclosure. Figure 4 The electronic device 400 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0112] like Figure 4 As shown, the electronic device 400 is manifested in the form of a general-purpose computing device. The components of the electronic device 400 may include, but are not limited to: at least one processing unit 410, at least one storage unit 420, a bus 430 connecting different system components (including storage unit 420 and processing unit 410), and a display unit 440.

[0113] The storage unit stores program code that can be executed by the processing unit 410, causing the processing unit 410 to perform the steps described in the "Exemplary Methods" section above, according to various exemplary embodiments of this disclosure.

[0114] Storage unit 420 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 421 and / or cache memory 422, and may further include a read-only memory (ROM) 423.

[0115] Storage unit 400 may also include a program / utility 424 having a set (at least one) of program modules 425, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0116] Bus 430 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0117] Electronic device 400 can also communicate with one or more external devices 500 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 400, and / or any device that enables electronic device 400 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 450. Furthermore, electronic device 400 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 460. As shown, network adapter 460 communicates with other modules of electronic device 400 via bus 430. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0118] Furthermore, exemplary embodiments of this disclosure also provide a computer-readable storage medium storing a program product capable of implementing the methods described above. In some possible embodiments, various aspects of this disclosure may also be implemented as a program product including program code that, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0119] It should be noted that the computer-readable medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0120] In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0121] Furthermore, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0122] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

Claims

1. A method for generating a video description text, characterized by, include: Obtain the initial description text of the target video, which is obtained by a large model for generating multiple description texts based on the output of the target video; The first prompt text is constructed based on the initial description text; The initial description text and the first prompt text are input into the target text generation model, so that the target text generation model is fused based on the first prompt text on the basis of the initial description text to obtain the output target text.

2. The method of claim 1, wherein, The large-scale model for generating descriptive text includes a large-scale model for generating video descriptive text. The step of obtaining the initial descriptive text of the target video includes: A second prompt text is constructed based on the target video; The second prompt text and the target video are input into the video description text generation model, so that the video description text generation model outputs the initial description text based on the second prompt text and the target video.

3. The method of claim 2, wherein, The construction of the second prompt text based on the target video includes: Get at least one example of the descriptive text for a historical video; The second prompt text is constructed based on the target video and the example descriptive text.

4. The method of claim 1, wherein, The large-scale descriptive text generation model includes an image descriptive text generation model, and the step of obtaining the initial descriptive text of the target video includes: Extract keyframes from the target video; A third prompt text is constructed based on the keyframes; The third prompt text and the keyframe are input into the image description text generation model, so that the image description text generation model outputs the initial description text based on the third prompt text and the keyframe.

5. The method of claim 4, wherein, The construction of the third prompt text based on the keyframe includes: Get at least one example of descriptive text for a keyframe in a historical video; The third prompt text is constructed based on the keyframe and the example descriptive text of the keyframe.

6. The method of claim 1, wherein, The large-scale model for generating descriptive text includes a video visual question answering (VQA) model. The process of obtaining the initial descriptive text for the target video includes: The first question text is constructed based on the target video; The first question text and the target video are input into the video VQA big model, so that the video VQA big model outputs the corresponding first answer text based on the first question text on the basis of the target video, and the first answer text is used as the initial description text corresponding to the video VQA big model.

7. The method of claim 1, wherein, The large-scale model for generating descriptive text includes an image VQA large-scale model, and the process of obtaining the initial descriptive text of the target video includes: Extract keyframes from the target video; Construct the second question text based on the keyframes; The second question text and the keyframe are input into the image VQA large model, so that the image VQA large model outputs the corresponding second answer text based on the second question text on the basis of the keyframe, and the second answer text is used as the initial description text corresponding to the image VQA large model.

8. A video description text generating apparatus characterized by comprising: include: The initial text acquisition module is used to acquire the initial description text of the target video. The initial description text is obtained by a large model that generates multiple description texts based on the output of the target video. The prompt text construction module is used to construct a first prompt text based on the initial description text; The target text fusion module is used to input the initial description text and the first prompt text into the target text generation model, so that the target text generation model fuses the initial description text with the first prompt text to obtain the output target text.

9. An electronic device, comprising: include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.