Video text generation method and device, electronic equipment and storage medium

Abstract

The invention provides a video text generation method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a first text generation instruction of a target video, and determining a video frame sequence of the target video and an audio text of the target video; generating a second text generation instruction based on the first text generation instruction, the timestamp text of the video frame sequence and the audio text; replacing an image placeholder in the second text generation instruction based on a video frame region feature and a video time sequence feature of the video frame sequence to obtain a third text generation instruction; and generating a target text corresponding to the target video through the multi-modal large model according to the third text generation instruction. Therefore, according to the scheme, the image placeholder is reserved in the second text generation instruction, and the video features are used for replacing the image placeholder, so that information such as texts, images, audios and the like can be deeply fused, the understanding ability of a multi-modal large model to videos is improved, and the adaptation degree of the texts and the videos is improved.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and large model technology, and in particular to a method, apparatus, electronic device and storage medium for generating video text. Background Technology

[0002] Currently, video text creation typically relies on manual processes, including text writing, script design, material selection, and post-editing. Pre-designed templates can be used to generate video text. While template-based text generation simplifies the process, it still requires manual adjustments to adapt the templates to specific needs, resulting in low efficiency, high labor and hardware costs, and difficulty in quickly responding to market changes.

[0003] While computer vision methods can identify keyframes, scenes, objects, and people from user-uploaded short video files, and NLP techniques can be used to generate video summaries or descriptions based on the video content analysis results, or large language models can be used to parse dialogue and pre-set video text descriptions, directly generating the video text from the dialogue or video descriptions, neither of these methods can integrate multimodal information beyond text, such as speech, video frames, and video timing, and therefore cannot flexibly adapt to different videos. Summary of the Invention

[0004] The purpose of this application is to at least partially solve one of the technical problems in the related art.

[0005] Therefore, the first objective of this application is to propose a method for generating video text, which can achieve deep integration of information such as text, images and audio, so as to improve the understanding ability of multimodal large models of videos, make the generated target text more consistent with the content of the target video, and improve the adaptability of text and video.

[0006] The second objective of this application is to provide a device for generating video text.

[0007] The third objective of this application is to propose an electronic device.

[0008] The fourth objective of this application is to provide a computer-readable storage medium.

[0009] To achieve the above objectives, a first aspect of this application proposes a method for generating video text, comprising: obtaining a first text generation instruction for a target video, and determining the video frame sequence of the target video and the audio text of the target video; generating a second text generation instruction based on the first text generation instruction, the timestamp text of the video frame sequence, and the audio text; replacing image placeholders in the second text generation instruction based on the video frame region features and video temporal features of the video frame sequence to obtain a third text generation instruction; and generating target text corresponding to the target video using a trained multimodal large model according to the third text generation instruction.

[0010] To achieve the above objectives, a second aspect of this application provides a video text generation apparatus, comprising: an acquisition module, configured to acquire a first text generation instruction for a target video, and determine a video frame sequence of the target video, and an audio text of the target video; a first generation module, configured to generate a second text generation instruction based on the first text generation instruction, the timestamp text of the video frame sequence, and the audio text; a replacement module, configured to replace image placeholders in the second text generation instruction based on video frame region features and video temporal features of the video frame sequence, to obtain a third text generation instruction; and a second generation module, configured to generate target text corresponding to the target video using a trained multimodal large model according to the third text generation instruction.

[0011] To achieve the above objectives, a third aspect of this application provides an electronic device, including: a processor; and a memory communicatively connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to enable the processor to perform the video text generation method described in the first aspect of the application.

[0012] To achieve the above objectives, a fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, the computer instructions being used to cause the computer to execute the video text generation method described in the above aspect of the embodiment.

[0013] The video text generation method, apparatus, electronic device, and storage medium provided in this application, by determining a first text generation instruction for the target video, a video frame sequence, and the audio text of the target video, can generate a second text generation instruction based on the first text generation instruction, the video frame sequence, and the video text. Furthermore, based on the video frame region features and video temporal features of the video frame sequence, image placeholders in the second text generation instruction are replaced to obtain a third text generation instruction. This allows a multimodal large-scale model to generate the target text corresponding to the target video based on the third text generation instruction. Therefore, this scheme reserves image placeholders in the second text generation instruction and replaces the image placeholders using video features, enabling deep fusion of text, image, and audio information. This improves the multimodal large-scale model's understanding of the video, making the generated target text more consistent with the content of the target video and improving the text-video compatibility.

[0014] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0015] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart illustrating a method for generating video text provided in an embodiment of this application; Figure 2 A flowchart illustrating another method for generating video text provided in an embodiment of this application; Figure 3 This is a schematic diagram illustrating the determination of spatial features of a video frame sequence according to an embodiment of this application; Figure 4 A schematic diagram of a region encoder provided in an embodiment of this application; Figure 5 This is a schematic diagram of a video timing encoder provided in an embodiment of this application; Figure 6 A flowchart illustrating another method for generating video text provided in an embodiment of this application; Figure 7 This is a schematic diagram illustrating the generation of target text using a multimodal large model provided in an embodiment of this application. Figure 8 This is a flowchart illustrating the training process of a multimodal large model in a video text generation method provided in this application embodiment. Figure 9 A schematic diagram illustrating the training of a large multimodal model provided in an embodiment of this application; Figure 10This is a schematic diagram illustrating the process of generating target text for a target video, as provided in an embodiment of this application. Figure 11 This is a schematic diagram of the structure of a video text generation device provided in an embodiment of this application. Detailed Implementation

[0016] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0017] The method and apparatus for generating video text according to embodiments of this application are described below with reference to the accompanying drawings.

[0018] Figure 1 This is a flowchart of a video text generation method provided according to an embodiment of this application, such as... Figure 1 As shown, the video text generation method of this application embodiment includes, but is not limited to, the following steps: S101, obtain the first text generation instruction of the target video, and determine the video frame sequence of the target video and the audio text of the target video.

[0019] It should be noted that the execution subject of the video text generation method provided in this application embodiment is an electronic device, which can be a terminal device. Optionally, the terminal device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be personal computers (PCs), televisions, etc. This application embodiment does not impose specific limitations.

[0020] In some embodiments, interaction with a client can be performed to obtain input information from the client, wherein the input information includes a target video and a first text generation instruction. The first text generation instruction is used to instruct the generation of target text corresponding to the target video.

[0021] Optionally, the target text is used to convey information about the target video, thereby attracting different audiences and achieving goals such as video promotion through precise and attractive textual expression.

[0022] In some embodiments, the audio in the target video can be converted into text based on the Automatic Speech Recognition (ASR) function to obtain the audio text of the target video. A video frame sequence is obtained by sampling the video frames of the target video and arranging the sampled video frames in timestamp order.

[0023] In some embodiments, the target video is decoded to obtain the corresponding video frames, and the video frames are uniformly sampled at set time intervals to obtain a video frame sequence. Optionally, the scale of the video frames can also be adjusted to achieve video frame standardization.

[0024] S102, based on the first text generation instruction, the timestamp text of the video frame sequence, and the audio text, generate the second text generation instruction.

[0025] In some embodiments, a second text generation instruction is obtained by acquiring a template of a text generation instruction and filling the template with a first text generation instruction, the timestamp text of the video frame sequence, and the audio text.

[0026] In some embodiments, the template for the text generation instruction includes an image placeholder, which reserves a position for subsequent image feature fusion, so that the final generated third text generation instruction can fuse text features and image features, thereby improving the accuracy of the multimodal model's understanding of video content.

[0027] For example, the second text generation instruction is: "The target video has a total of 96 frames sampled, and the timestamps of the video frame samples are 1.5s, 2.5s.......4.5s."

[0028] [IMG], [IMG], ..., [IMG].

[0029] Audio text: [Textual information of video dialogue transcribed via ASR].

[0030] Promotion goal: [Users actively input relevant trending topics].

[0031] First text generation instruction: Please generate the corresponding target text based on the target video and promotion goal.

[0032] [IMG] represents an image placeholder.

[0033] In some embodiments, after generating the second text generation instruction, the second text generation instruction can be encoded to convert it into discrete numerical units that the model can process. That is, the second text generation instruction is a text string.

[0034] Optionally, a word segmenter can be used to convert text strings into discrete numerical units that the model can process. The word segmenter uses a pre-defined vocabulary that contains the correspondence between words and numerical values ​​in the text. In other words, the vocabulary maps words in the text to unique numerical IDs. These numerical IDs serve as input to the model, enabling it to process text data in numerical form.

[0035] Optionally, the word segmenter in this embodiment can be implemented using any word segmenter in related technologies, without further limitations.

[0036] S103, based on the video frame region features and video temporal features of the video frame sequence, the image placeholder in the second text generation instruction is replaced to obtain the third text generation instruction.

[0037] In some embodiments, feature extraction is performed on the video frame sequence to obtain video frame region features and video temporal features, and the image placeholder of the second text generation instruction Zhang Gong is replaced with the video frame region features and video temporal features to obtain a third text generation instruction containing text features and image features.

[0038] In some embodiments, spatial features of the video frame sequence can be extracted first, and then the spatial features can be region-encoded by a region encoder to determine the location of the region containing the target as the video frame region feature. In addition, the spatial features can be temporally encoded by a temporal encoder to determine the temporal feature as the video temporal feature.

[0039] It is understandable that the image placeholder is placed in advance in the second text generation instruction to reserve the position of video frame region features and video temporal features. Then, the image placeholder is replaced based on the video frame region features and video temporal features, so that the image features are integrated into the third text generation instruction, thereby enabling the third text generation instruction to contain multimodal information.

[0040] It should be noted that image placeholders are symbols or markers used to temporarily replace actual images. In the second text generation instruction, image placeholders are used to indicate the location where an image will be inserted in the future.

[0041] S104 uses a trained multimodal large model to generate the target text corresponding to the target video according to the third text generation instruction.

[0042] In some embodiments, by inputting a third text generation instruction into a trained multimodal large model, the multimodal large model fuses multimodal information based on the text features and image features in the third text generation instruction, enabling the multimodal large model to better understand the target video and the first text generation instruction, thereby outputting target text that better matches the target video and the first text generation instruction.

[0043] Among them, the text features in the third text generation instruction refer to the features of the first text generation instruction, the features of the timestamp text, and the features of the audio text; the image features in the third text generation instruction refer to the video frame region features and the video temporal features.

[0044] In other words, during the process of generating the third text generation instruction, the second text generation instruction can be vectorized and encoded first to determine the features of the first text generation instruction, the features of the timestamp text, and the features of the audio text. Then, the image placeholders are replaced according to the video frame region features and video temporal features to obtain the third text generation instruction.

[0045] In the video text generation method provided in this application embodiment, by determining the first text generation instruction, video frame sequence, and audio text of the target video, a second text generation instruction can be generated based on the first text generation instruction, video frame sequence, and video text. Then, based on the video frame region features and video temporal features of the video frame sequence, image placeholders in the second text generation instruction are replaced to obtain a third text generation instruction. This allows the multimodal large model to generate the target text corresponding to the target video based on the third text generation instruction. Therefore, this scheme reserves image placeholders in the second text generation instruction and replaces the image placeholders using video features, enabling deep fusion of text, image, and audio information. This improves the multimodal large model's understanding of the video, making the generated target text more consistent with the content of the target video and improving the text-video compatibility.

[0046] Figure 2 This is a flowchart of a video text generation method provided according to an embodiment of this application, such as... Figure 2 As shown, the video text generation method of this application embodiment includes, but is not limited to, the following steps: S201, obtain the first text generation instruction of the target video, and determine the video frame sequence of the target video and the audio text of the target video.

[0047] S202, based on the first text generation instruction, the timestamp text of the video frame sequence, and the audio text, generate the second text generation instruction.

[0048] In the embodiments of this application, steps S201-S202 can be implemented in any of the embodiments of this application, and no limitation is made here, nor will it be described in detail.

[0049] S203, determine the spatial characteristics of the video frame sequence.

[0050] In some embodiments, an image encoder can encode the video frame sequence to obtain the spatial features corresponding to the video frame sequence. By inputting each video frame in the video frame sequence into the image encoder, the image encoder performs a convolution operation on the video frames to encode the spatial information of the video frames, thereby obtaining the spatial features.

[0051] Figure 3 The diagram illustrates the determination of spatial features in a video frame sequence. For each video frame in the sequence, it is input into an image encoder, which performs convolutional encoding to obtain the spatial features of the video frame.

[0052] S204, target detection is performed based on spatial features to obtain video frame region features of the video frame sequence.

[0053] In some embodiments, video frame region features can refer to the features of key regions within spatial features, where the key region may be a region containing the target. By performing target detection on the spatial features, the key region can be determined from the spatial features, and the features of the key region can be used as video frame region features.

[0054] In some embodiments, key regions can be located based on the predicted bounding boxes used for object detection. By performing object detection on spatial features, candidate predicted bounding boxes of spatial features are obtained, and the candidate predictions are filtered to determine the target predicted bounding boxes of key regions. Thus, video frame region features are determined from spatial features based on the location of the target predicted bounding boxes.

[0055] In some embodiments, duplicate candidate bounding boxes can be filtered out from the candidate predicted bounding boxes using non-maximum suppression (NMS) to obtain the target predicted bounding box, and the coordinate information of the target predicted bounding box can be determined. The coordinate information can be the vertex coordinates of the target predicted bounding box. Further, video frame region features can be determined based on the coordinate information and spatial features.

[0056] In some embodiments, coordinate information can locate the specific position of a key region within a video frame, thereby allowing the extraction of features from the key region as video frame region features. Alternatively, the coordinate information of the target prediction box can be mapped to spatial features to determine the key region within the spatial features, and the features of this key region can be used as video frame region features.

[0057] In some embodiments, determining video frame region features can be accomplished using a region encoder, such as... Figure 4 The diagram shown is of a region encoder. Figure 4 In this context, the region encoder includes a bounding box predictor and a region alignment model.

[0058] By inputting spatial features into the target bounding box predictor, a set of predicted bounding boxes consisting of multiple candidate bounding boxes can be determined. Non-duplicated prediction of these candidate bounding boxes is then performed using Non-Maximum Suppression (NMS) to obtain the target predicted bounding box, and the coordinate information of the target predicted bounding box is determined. By inputting the coordinate information and spatial features into the region alignment model, the region features of the video frame can be determined.

[0059] S205 uses spatial features for temporal coding to obtain the video temporal features of the video frame sequence.

[0060] In some embodiments, the timestamp of a video frame can be obtained from the timestamp text corresponding to the video frame sequence, and the timestamp can be fused with spatial features to obtain the video temporal features of the video frame sequence from the fused features.

[0061] In some embodiments, the static fusion features of the video frame sequence are determined by obtaining the first semantic unit corresponding to the timestamp of the video frame sequence and the preset query semantic unit of the video frame sequence, and based on the first semantic unit, the query semantic unit and the spatial features.

[0062] Optionally, the first semantic unit and the query semantic unit can be concatenated to obtain a temporal semantic unit, and then the temporal semantic unit can be fused with spatial features to obtain a static fused feature containing the temporal semantic unit. For example, the temporal semantic unit and spatial features can be fused using an encoder.

[0063] Furthermore, a temporal sliding window can be used to extract the static fused features temporally to determine the video temporal features. Optionally, a temporal sliding window can be used to extract the query semantic units in the static fused features temporally to determine the video temporal features.

[0064] Understandably, a temporal sliding window is used to divide static fused features into multiple temporal segments. Static fused features can be divided according to the window size and stride of the temporal sliding window.

[0065] In some embodiments, the target number of time segments can be determined based on the window size and step size of the time-series sliding window. Alternatively, the target number of time segments can be determined by determining the number of video frames in the video frame sequence and combining this with the window size and step size of the time-series sliding window.

[0066] Furthermore, based on a temporal sliding window, the static fused features can be divided into a target number of temporal segments, and the temporal semantic units corresponding to the temporal segments can be determined. Thus, the temporal features of the video can be determined based on the temporal segments and the temporal semantic units.

[0067] Optionally, the temporal segments and their corresponding temporal semantic units can be encoded to obtain a target number of temporal features, and then the target number of temporal features can be concatenated to obtain video temporal features.

[0068] In some embodiments, spatial features can be temporally encoded by a video temporal encoder to obtain video temporal features of a video frame sequence. Figure 5 This is a schematic diagram of a video timing encoder. Figure 5 This includes a Querying Transformer (QFormer) encoder. Temporal semantic units are obtained by concatenating the first semantic unit and the query semantic unit. The QFormer encoder then fuses these temporal semantic units with spatial features to obtain static fused features. These static fused features are then divided into a target number of temporal segments using a temporal sliding window. The temporal segments and their corresponding temporal semantic units are encoded to obtain the target number of temporal features. Finally, these target number of temporal features are concatenated to obtain the video temporal features.

[0069] S206, based on the video frame region features and video temporal features of the video frame sequence, the image placeholder in the second text generation instruction is replaced to obtain the third text generation instruction.

[0070] S207 uses a trained multimodal large model to generate the target text corresponding to the target video according to the third text generation instruction.

[0071] In the embodiments of this application, steps S206-S207 can be implemented in any of the embodiments of this application, and no limitation is made here, nor will it be described in detail.

[0072] In the video text generation method provided in this application embodiment, based on the spatial characteristics of the video frame sequence, and based on the spatial characteristics, the video frame region characteristics and video frame temporal characteristics of the video frame sequence are determined. Explicit and implicit information can be obtained from the video frames to ensure that the generated text and video content are consistent in style, theme, and logic.

[0073] Figure 6 This is a flowchart of a video text generation method provided according to an embodiment of this application, such as... Figure 6 As shown, the video text generation method of this application embodiment includes, but is not limited to, the following steps: S601, obtain the first text generation instruction of the target video, and determine the video frame sequence of the target video and the audio text of the target video.

[0074] S602, based on the first text generation instruction, the timestamp text of the video frame sequence, and the audio text, generate the second text generation instruction.

[0075] In the embodiments of this application, steps S601-S602 can be implemented in any of the embodiments of this application, and no limitation is made here, nor will it be described in detail.

[0076] S603, obtain the fused features of video frame regional features and video temporal features.

[0077] S604, determine the encoding position of the image placeholder in the second text generation instruction.

[0078] S605, based on the encoded position, replaces image placeholders by fusing features to obtain the third text generation instruction.

[0079] In some embodiments, by fusing video frame region features and video temporal features, the fused features of the video frame region features and video temporal features can be determined. The fusion of video frame region features and video temporal features can be achieved using any feature fusion method in related technologies, and no further limitations are imposed here.

[0080] It should be noted that image placeholders are symbols or markers used to temporarily replace actual images. In the second text generation instruction, image placeholders are used to indicate the location where an image will be inserted in the future.

[0081] In some embodiments, the elements contained in the second text generation instruction and the position of each element are determined by analyzing the format and structure of the second text generation instruction. The encoded position of the image placeholder can be determined from the second text generation instruction by locating the image placeholder.

[0082] Optionally, the image placeholder can be located based on its identification information.

[0083] In some embodiments, after determining the encoding position of the image placeholder, the image placeholder at the encoding position can be replaced with a fusion feature to obtain a third text generation instruction, which can include text features and image features for multimodal information fusion.

[0084] S606 generates the target text corresponding to the target video based on the third text generation instruction using a trained multimodal large model.

[0085] In the embodiments of this application, step S606 can be implemented in any of the embodiments of this application, and no limitation is made here, nor will it be described in detail.

[0086] Figure 7 The diagram illustrates the generation of target text by a multimodal large model. By obtaining the vectorized representation of the second text generation instruction and fusing video frame region features and video temporal features, a fused feature is obtained. This fused feature is then used to replace the image placeholders in the vectorized second text generation instruction, resulting in a third text generation instruction. This third text generation instruction serves as the input to the multimodal large model, which then generates the target text based on the third text generation instruction.

[0087] In the video text generation method provided in this application embodiment, the video frame region features and video temporal features of the video frame sequence are fused, and the image placeholders in the second text generation instruction are replaced according to the fused features to obtain the third text generation instruction. This method can deeply fuse information such as text, images and audio to improve the understanding ability of the multimodal large model of video, so that the generated target text is more consistent with the content of the target video and the adaptability of text and video is improved.

[0088] Based on any of the above embodiments, the embodiments of this application can further explain and illustrate the training process of multimodal large models, such as... Figure 8 As shown, the training process of the multimodal large model in this application includes, but is not limited to, the following steps: S801, determine the sample training dataset corresponding to the multimodal large model to be trained.

[0089] In some embodiments, sample text generation instructions and their corresponding sample texts can be obtained and used as sample training data in the sample training dataset. Alternatively, a thought chain for generating sample text based on the sample text generation instructions can be obtained and associated with the sample text generation instructions and the sample texts, serving as sample training data in the sample training dataset.

[0090] Optionally, historical input and output information from a large multimodal model can be obtained as a sample training dataset. Alternatively, the sample training dataset can be manually configured.

[0091] For example, the format of sample training data is: [Sample text generation instruction]-[mind chain]-[sample text].

[0092] In some embodiments, the thought logic of the thought chain can be annotated to make the thought logic and thought process of the thought chain more reasonable.

[0093] In some embodiments, a thought chain template can be pre-set, and content can be filled into the thought chain template as a thought chain. For example, the thought chain template is: the key events that occur in the video from t1 to t2 are [Event 1, Event 2], which matches [Hotspot 1]; the key events that occur in the video from t3 to t4 are [Event 3, Event 4], which matches [Hotspot 2].

[0094] S802, the sample training dataset is divided into a first dataset and a second dataset, and the first dataset is input into the multimodal large model to be trained for fine-tuning training, so as to obtain a fine-tuned multimodal large model.

[0095] In some embodiments, the sample training dataset can be divided into a first dataset and a second dataset based on the number of sample training data points in the dataset. For example, the sample training dataset can be divided into a first dataset and a second dataset on an equal basis, based on the number of sample training data points.

[0096] In some embodiments, the first dataset and the second dataset are used for different stages of model training. In this example, the first dataset is used for the fine-tuning training stage of the model, and the second dataset is used for the scoring stage of the model.

[0097] In some embodiments, the second dataset may be used for the fine-tuning training phase of the model, and the first dataset may be used for the scoring phase of the model; no specific limitations are imposed on this.

[0098] In some embodiments, a first dataset is input into the multimodal large model to be trained, so that the multimodal large model generates output text according to the sample text generation instructions in the first dataset, and the multimodal large model is fine-tuned based on the output text and the sample text to obtain a fine-tuned multimodal large model.

[0099] Optionally, the multimodal large model can be fine-tuned based on the loss function of the output text and sample text, and training can continue until the data in the first dataset is trained, thus obtaining the fine-tuned multimodal large model.

[0100] S803, based on the second dataset, obtains the output results of the fine-tuned multimodal large model, and scores the output results based on the preset scoring rules to determine the score results of the output results.

[0101] In some embodiments, the output results are obtained by inputting the second dataset into the fine-tuned multimodal large model, and the output results are scored according to the preset scoring rules to determine the score of the output results.

[0102] In some embodiments, each sample text generation instruction in the second dataset can be repeatedly input into the fine-tuned multimodal large model to obtain multiple output results corresponding to a sample text instruction, and the multiple output results can be randomly sampled to score the randomly sampled output results.

[0103] For example, for sample text generation instruction 1, which corresponds to 5 output results, 3 output results can be determined from the 5 output results for scoring.

[0104] For example, the preset scoring rules are as follows: Rule 1. The output must include both "thought process" and "text". Add 1 point if both are included, and deduct 1 point if they are not. Rule 2. The generated text will be scored and evaluated using four dimensions, such as "readability, attractiveness, accuracy, and conciseness". If there is any deviation, points will be deducted; if there is no deviation, points will be added. The maximum score is 10 points. Rule 3. The text must include the above four dimensions. One point is added for each dimension included, and one point is deducted for each dimension missing. Rule 4. Check whether the thought process and the semantics of the text are similar, and score them from 1 to 10 according to the degree of similarity.

[0105] S804. Based on the scoring results, determine the optimization objective of the fine-tuned multimodal large model, and continue to optimize the fine-tuned multimodal large model based on the optimization objective to obtain the trained multimodal large model.

[0106] In some embodiments, defects in the output results can be identified from the scoring results, thereby determining the optimization objective of the fine-tuned multimodal large model. Optionally, scores corresponding to different scoring rules can be determined from the scoring results; a higher score indicates better performance of the output result under that scoring rule.

[0107] Taking the above scoring rules as an example, if the score corresponding to scoring result indicator rule 2 is 5 points, it can be determined that the output result has deviations in readability, attractiveness, correctness, and conciseness. Then, increasing readability, attractiveness, correctness, and conciseness can be used as the optimization goal of the fine-tuned multimodal large model.

[0108] In some embodiments, the finely tuned multimodal large model can be optimized using Group Relative Policy Optimization (GRPO) according to the optimization objective to obtain a trained multimodal large model. The trained multimodal large model is then used to generate the target text corresponding to the target video.

[0109] In some embodiments, to minimize the deviation between the output of the trained multimodal large model and the output of the fine-tuned multimodal large model, relative entropy divergence (KL divergence) can be used to constrain the generated text, thereby maintaining a certain degree of similarity between the text and avoiding performance degradation or output deviation from expectations due to optimization.

[0110] Figure 9 The diagram illustrates the training of a large multimodal model. A sample training dataset is obtained and divided into a first dataset and a second dataset. The first dataset is used to fine-tune the large multimodal model, and the second dataset is used to optimize the fine-tuned model.

[0111] By obtaining the output results corresponding to the fine-tuned multimodal large model and scoring the output results according to the preset scoring rules, the scoring results of the output results are determined. Then, the GRPO method can be used to optimize the fine-tuned multimodal large model based on the scoring results, thereby obtaining a trained multimodal large model.

[0112] The video text generation method provided in this application involves supervised fine-tuning of a multimodal large model to equip it with basic text generation capabilities. The output of the fine-tuned multimodal large model is then scored, and an optimization algorithm is used to further refine the model, making the generated text more aligned with human preferences. By introducing a scoring mechanism, the accuracy and contextual appropriateness of the generated text can be significantly improved, and the content of the generated text can be dynamically adjusted to reduce model illusions during text generation.

[0113] Figure 10 This diagram illustrates the process of generating target text for the target video. Figure 10 Includes the following steps: S1. Data Preprocessing: Receive the target video and the first text generation instruction. Simultaneously, use ASR technology to transcribe the audio content from the video into text, obtaining audio text. Concatenate the audio text and the first text generation instruction. Sample the video frames of the target video to obtain the timestamp text of the video frames. Finally, concatenate the first text generation instruction, the timestamp text, and the audio text to obtain the second text generation instruction. When concatenating the second text generation instruction, reserve image placeholders within the second text generation instruction.

[0114] S2. Tokenizer: Performs text segmentation, vocabulary mapping, and other processing on the second text generation instruction to obtain a vector representation that can be processed by the model.

[0115] S3. Image Encoder: Converts the input video frame sequence into a feature representation that the model can understand. This involves extracting the spatial features of the video frame sequence. Commonly used image encoders include Contrastive Language-Image Pre-training (CLIP) based on Vision Transformer (ViT) and open-source CLIP models. These models are trained by aligning image and text embeddings to capture rich information from the visual content.

[0116] S4. Region Encoder: Identifies key regions in spatial features and obtains candidate prediction boxes. Then, duplicate candidate prediction boxes are removed to obtain target prediction boxes, and the coordinate information of the target prediction boxes is determined. Based on this coordinate information, the video frame region features are determined from the spatial features.

[0117] S5. Video Timing Encoder: Used to capture inter-frame timing information of consecutive video frames. The video timing encoder uses a timing sliding window to obtain the timing characteristics between consecutive video frames as video timing features.

[0118] S6. Multimodal Information Fusion Unit: This is a vision-to-language multimodal information adapter that enables the fusion and interaction of information from different modalities within the same model. It obtains the third text generation instruction by fusing video frame region features and video temporal features, and then replacing image placeholders in the second text generation instruction with the fused features.

[0119] S7. Multimodal Large Model: Input the third text generation instruction into the multimodal large model, and the multimodal large model outputs the target text of the target video according to the third text generation instruction.

[0120] Corresponding to the video text generation methods proposed in the above embodiments, an embodiment of this application also proposes a video text generation apparatus. Since the video text generation apparatus proposed in this application corresponds to the video text generation methods proposed in the above embodiments, the implementation methods of the above video text generation methods are also applicable to the video text generation apparatus proposed in this application, and will not be described in detail in the following embodiments.

[0121] Figure 11 This is a schematic diagram of the structure of a video text generation device provided in an embodiment of this application.

[0122] like Figure 11 As shown, the video text generation device 1100 includes: The acquisition module 1101 is used to acquire the first text generation instruction of the target video, and determine the video frame sequence of the target video and the audio text of the target video; The first generation module 1102 is used to generate a second text generation instruction based on the first text generation instruction, the timestamp text of the video frame sequence, and the audio text; The replacement module 1103 is used to replace the image placeholder in the second text generation instruction based on the video frame region features and video temporal features of the video frame sequence, so as to obtain the third text generation instruction. The second generation module 1104 is used to generate the target text corresponding to the target video according to the third text generation instruction using a trained multimodal large model.

[0123] In one possible implementation of this application, the replacement module 1103 is further configured to: determine the spatial features of the video frame sequence; perform target detection based on the spatial features to obtain video frame region features of the video frame sequence; and perform temporal encoding based on the spatial features to obtain video temporal features of the video frame sequence.

[0124] In one possible implementation of this application, the replacement module 1103 is further configured to: perform target detection on the spatial features to obtain candidate prediction boxes of the spatial features; filter out duplicate candidate prediction boxes from the candidate prediction boxes to obtain target prediction boxes, and determine the coordinate information of the target prediction boxes; and determine the video frame region features based on the coordinate information and the spatial features.

[0125] In one possible implementation of this application, the replacement module 1103 is further configured to: obtain a first semantic unit corresponding to the timestamp of the video frame sequence, and a preset query semantic unit of the video frame sequence; determine the static fusion feature of the video frame sequence based on the first semantic unit, the query semantic unit and the spatial feature; and perform temporal extraction on the static fusion feature through a temporal sliding window to determine the video temporal feature.

[0126] In one possible implementation of this application embodiment, the temporal sliding window is used to divide the static fusion feature into multiple temporal segments, and the replacement module 1103 is further used to: determine the target number of temporal segments based on the window size and step size of the temporal sliding window; divide the static fusion feature into the target number of temporal segments based on the temporal sliding window, and determine the temporal semantic unit corresponding to the temporal segment; and determine the video temporal feature based on the temporal segment and the temporal semantic unit.

[0127] In one possible implementation of this application, the replacement module 1103 is further configured to: obtain the fusion features of the video frame region features and video temporal features; determine the encoding position of the image placeholder in the second text generation instruction; and, based on the encoding position, replace the image placeholder with the fusion features to obtain the third text generation instruction.

[0128] In one possible implementation of this application, the second generation module 1104 is further configured to: determine the sample training dataset corresponding to the multimodal large model to be trained; divide the sample training dataset into a first dataset and a second dataset, and input the first dataset into the multimodal large model to be trained for fine-tuning training to obtain a fine-tuned multimodal large model; based on the second dataset, obtain the output result of the fine-tuned multimodal large model, and score the output result based on a preset scoring rule to determine the score result of the output result; determine the optimization target of the fine-tuned multimodal large model based on the score result, and continue to optimize the fine-tuned multimodal large model based on the optimization target to obtain a trained multimodal large model, wherein the multimodal large model is used to generate the target text corresponding to the target video.

[0129] In the video text generation apparatus provided in this application embodiment, by determining a first text generation instruction for the target video, a video frame sequence, and the audio text of the target video, a second text generation instruction can be generated based on the first text generation instruction, the video frame sequence, and the video text. Then, based on the video frame region features and video temporal features of the video frame sequence, image placeholders in the second text generation instruction are replaced to obtain a third text generation instruction. This allows the multimodal large model to generate the target text corresponding to the target video based on the third text generation instruction. Therefore, this scheme reserves image placeholders in the second text generation instruction and replaces the image placeholders using video features, enabling deep fusion of text, image, and audio information. This improves the multimodal large model's understanding of the video, making the generated target text more consistent with the content of the target video and improving the text-video compatibility.

[0130] It should be noted that the foregoing explanation of the video text generation method embodiment also applies to the video text generation apparatus of this embodiment, and will not be repeated here.

[0131] To implement the above embodiments, this application also proposes an electronic device, including: a processor and a memory communicatively connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the method provided in the foregoing embodiments. To implement the above embodiments, this application also proposes a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods provided in the foregoing embodiments.

[0132] The collection, storage, use, processing, transmission, provision, and application of user personal information involved in this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0133] It should be noted that personal information collected from users should be used for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. Furthermore, such collection / sharing should only be conducted after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign an agreement / authorization that includes authorization of relevant user information before the user uses the function. In addition, any necessary steps must be taken to protect and safeguard access to such personal information data and ensure that others with access to personal information data comply with their privacy policies and procedures.

[0134] This application is intended to provide an implementation scheme for users to selectively prevent the use or access to their personal information data. Specifically, this application is intended to provide hardware and / or software to prevent or block access to such personal information data. Once personal information data is no longer needed, risks can be minimized by restricting data collection and deleting data. Furthermore, where applicable, such personal information is de-identified to protect user privacy.

[0135] In the foregoing descriptions of the embodiments, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0136] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0137] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0138] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0139] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0140] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0141] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0142] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for generating a video text, characterized by, The method comprises: obtaining a first text generation instruction of a target video, determining a video frame sequence of the target video, and an audio text of the target video; generating a second text generation instruction based on the first text generation instruction, a timestamp text of the video frame sequence, and the audio text; replacing an image placeholder in the second text generation instruction based on a video frame region feature and a video timing feature of the video frame sequence to obtain a third text generation instruction; generating a target text corresponding to the target video according to the third text generation instruction through a trained multi-modal large model.

2. The method of claim 1, wherein, Before the replacement of the image placeholder in the second text generation instruction based on the video frame region feature and the video timing feature of the video frame sequence to obtain the third text generation instruction, the method comprises: determining a spatial feature of the video frame sequence; performing target detection based on the spatial feature to obtain a video frame region feature of the video frame sequence; performing timing coding based on the spatial feature to obtain a video timing feature of the video frame sequence.

3. The method of claim 2, wherein, The target detection based on the spatial feature to obtain the video frame region feature of the video frame sequence comprises: performing target detection on the spatial feature to obtain a candidate prediction box of the spatial feature; screening out repeated candidate prediction boxes from the candidate prediction box to obtain a target prediction box, and determining coordinate information of the target prediction box; determining the video frame region feature based on the coordinate information and the spatial feature.

4. The method of claim 2, wherein, The timing coding based on the spatial feature to obtain the video timing feature of the video frame sequence comprises: obtaining a first semantic unit corresponding to a timestamp of the video frame sequence, and a preset query semantic unit of the video frame sequence; determining a static fusion feature of the video frame sequence based on the first semantic unit, the query semantic unit, and the spatial feature; performing timing extraction on the static fusion feature through a timing sliding window to determine the video timing feature.

5. The method of claim 4, wherein, The timing sliding window is used to divide the static fusion feature into a plurality of timing segments, and the timing extraction on the static fusion feature through the timing sliding window to determine the video timing feature comprises: determining a target number of the timing segments based on a window size and a step length of the timing sliding window; dividing the static fusion feature into the target number of timing segments based on the timing sliding window, and determining a timing semantic unit corresponding to the timing segment; determining the video timing feature based on the timing segment and the timing semantic unit.

6. The method according to any one of claims 1-5, characterized in that, The replacement of the image placeholder in the second text generation instruction based on the video frame region feature and the video timing feature of the video frame sequence to obtain the third text generation instruction comprises: obtaining a fusion feature of the video frame region feature and the video timing feature; determining an encoding position of the image placeholder in the second text generation instruction; replacing the image placeholder through the fusion feature based on the encoding position to obtain the third text generation instruction.

7. The method according to any one of claims 1-5, characterized in that, The training process of the multi-modal large model comprises: determining a sample training data set corresponding to the multi-modal large model to be trained; dividing the sample training data set into a first data set and a second data set, and inputting the first data set into the multi-modal large model to be trained for fine-tuning training to obtain a fine-tuned multi-modal large model; based on the second data set, obtaining the output result of the fine-tuned multi-modal large model, and scoring the output result based on a preset scoring rule to determine the scoring result of the output result; based on the scoring result, determining the optimization target of the fine-tuned multi-modal large model, and based on the optimization target, continuing to optimize the fine-tuned multi-modal large model to obtain a trained multi-modal large model, the multi-modal large model being used to generate a target text corresponding to the target video.

8. An apparatus for generating a video text, characterized by comprising: The device comprises: an acquisition module configured to acquire a first text generation instruction of a target video, determine a video frame sequence of the target video, and determine an audio text of the target video; a first generation module configured to generate a second text generation instruction based on the first text generation instruction, a timestamp text of the video frame sequence, and the audio text; a replacement module configured to replace an image placeholder in the second text generation instruction based on a video frame region feature and a video timing feature of the video frame sequence to obtain a third text generation instruction; a second generation module configured to generate a target text corresponding to the target video by a trained multi-modal large model according to the third text generation instruction.

9. An electronic device, comprising: comprise: a processor, and a memory connected in communication with the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the method of any one of claims 1-7.

Images