A script generation method and related apparatus

Abstract

The application discloses a script generation method and related device, and relates to the technical field of video processing, and comprises the following steps: obtaining an original video to be processed, performing shot processing on the original video to obtain a plurality of shot videos, tracking and marking characters in the plurality of shot videos to obtain a plurality of character role marked videos, performing scene description processing on the plurality of character role marked videos to obtain a target script containing character roles. The application can avoid losing tracking targets in the character tracking process through shot processing, improve the accuracy and reliability of tracking and marking, and make the content understanding degree of the character role marked videos higher through shot processing, improve the accuracy of scene description processing, and further improve the quality of the target script. Meanwhile, the characters can be integrated into the target script through character tracking and marking, the quality of the target script is further improved, and the consistency between the target script and the original video is ensured.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a script generation method and related apparatus. Background Technology

[0002] In some scenarios, it is necessary to generate a complete script based on a given video. For example, a script can be generated based on a short video published on a short video creation platform for use in short video derivative works, screenwriting, and other fields.

[0003] However, most current script generation methods are text-to-script generation methods, with very few methods involving video-to-script generation. Summary of the Invention

[0004] In view of the above problems, this application provides a script generation method and related apparatus to achieve the purpose of generating a target script based on the original video. The specific solution is as follows:

[0005] The first aspect of this application provides a script generation method, including:

[0006] Obtain the original video to be processed;

[0007] The original video is split into multiple segments to obtain multiple split-scene videos;

[0008] The characters in the multiple storyboard videos are tracked and labeled to obtain multiple character-labeled videos;

[0009] The video clips featuring the various characters are processed with scene descriptions to obtain a target script containing the characters.

[0010] In one possible implementation, the original video is processed into multiple split-scene videos, including:

[0011] Detect scene change information in the original video;

[0012] Based on the scene change information, the original video is divided into multiple segmented videos, each of which belongs to a different scene.

[0013] In one possible implementation, the tracking and tagging of characters in the multiple storyboard videos to obtain multiple character-tagged videos includes:

[0014] Character tracking is performed on the multiple storyboard videos to obtain the character trajectories corresponding to the multiple storyboard videos;

[0015] Based on the character trajectories corresponding to the multiple storyboard videos, the characters in the multiple storyboard videos are tagged, resulting in the multiple character-tagged videos.

[0016] In one possible implementation, the step of tagging the characters in the multiple storyboard videos according to the character trajectories corresponding to the multiple storyboard videos respectively, to obtain the multiple character-tagged videos, includes:

[0017] Obtain a pre-configured character face mapping table, wherein the character face mapping table includes the correspondence between the character identifiers and facial images of all characters in the original video;

[0018] For each of the multiple storyboard videos:

[0019] Based on the character trajectories corresponding to the storyboard video, obtain the facial image of each character from the storyboard video;

[0020] The facial image of each character is matched with the facial images in the character facial mapping table to obtain the matching result, and the character identifier of each character is determined based on the matching result;

[0021] Based on the character identifier of each character, each character in the storyboard video is tagged, resulting in a character-tagged video corresponding to the storyboard video.

[0022] The character tag videos corresponding to the multiple storyboard videos are obtained as the multiple character tag videos.

[0023] In one possible implementation, the step of performing scenario description processing on the multiple character-tagged videos to obtain a target script containing the characters includes:

[0024] Contextual understanding is performed on the multiple character-tagged videos to obtain video content description information corresponding to each of the multiple character-tagged videos, wherein the video content description information includes the character identifier of the character in the corresponding character-tagged video;

[0025] The target script is obtained by integrating the video content description information corresponding to the multiple character-marked videos.

[0026] In one possible implementation, the step of performing context understanding on the multiple character-tagged videos to obtain video content description information corresponding to each of the multiple character-tagged videos includes:

[0027] Obtain a first text prompt word, wherein the first text prompt word includes the basic script elements required to generate the target script;

[0028] The multiple character-labeled videos and the first text prompt are input together into a pre-trained video understanding multimodal large model to obtain video content description information corresponding to the multiple character-labeled videos;

[0029] The training data used in the training phase of the video understanding multimodal large model includes: multiple character tagging training videos and corresponding video content description information. The multiple character tagging training videos are obtained by tracking and tagging characters in multiple training storyboard videos.

[0030] In one possible implementation, the process of integrating the video content description information corresponding to the multiple character-tagged videos to obtain the target script includes:

[0031] Obtain the second text prompt word, wherein the second text prompt word is used to prompt the large language model to integrate the video content description information corresponding to the multiple character tag videos;

[0032] The video content description information corresponding to the multiple character-tagged videos and the second text prompt words are input into the large language model to obtain the target script output by the large language model.

[0033] A second aspect of this application provides a script generation apparatus, comprising:

[0034] The video acquisition module is used to acquire the original video to be processed;

[0035] The video segmentation module is used to perform segmentation processing on the original video to obtain multiple segmented videos;

[0036] The tracking and labeling module is used to track and label the characters in the multiple storyboard videos to obtain multiple character-labeled videos;

[0037] The script generation module is used to perform scenario description processing on the multiple character-tagged videos to obtain a target script containing the characters.

[0038] A third aspect of this application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0039] The memory is used to store computer programs;

[0040] The processor is used to execute the computer program so that the electronic device can implement the script generation method of the first aspect or any implementation thereof.

[0041] A fourth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the script generation method of the first aspect or any implementation thereof.

[0042] By employing the aforementioned technical solution, the script generation method provided in this application, considering that shorter video clips are more conducive to understanding video content and character tracking compared to complete videos, performs segmentation processing on the original video to obtain multiple segmented videos. Then, it tracks and tags the characters in these multiple segmented videos to obtain multiple character-tagged videos. Furthermore, it performs scenario description processing on these character-tagged videos to obtain a target script containing the characters. Through segmentation, this application avoids losing the tracking target during character tracking, improving the accuracy and reliability of tracking and tagging. Furthermore, segmentation makes the content of the character-tagged videos more comprehensible, improving the accuracy of scenario description processing, and thus enhancing the quality of the target script. Simultaneously, through character tracking and tagging, the characters can be integrated into the target script, further improving the quality of the target script and ensuring consistency between the target script and the original video. Attached Figure Description

[0043] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0044] Figure 1 A schematic diagram of a system architecture is provided for this application;

[0045] Figure 2 A schematic diagram of an optional hardware structure for the terminal 100 provided in this application;

[0046] Figure 3 This application provides a schematic diagram of the structure of a server 200;

[0047] Figure 4 A flowchart illustrating a script generation method provided in this application;

[0048] Figure 5 A schematic diagram illustrating the marking of a person's head within a video frame, as provided in this application;

[0049] Figure 6 A schematic diagram of the structure of a script generation device provided in this application;

[0050] Figure 7 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0051] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0052] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0053] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0054] See Figure 1 , Figure 1 A schematic diagram of a system architecture is shown. The system may include a terminal 100 and a server 200. The server 200 may include one or more servers (…). Figure 1 (The example includes a server), and the server 200 can provide the method provided in the embodiments of this application to one or more terminals.

[0055] The terminal 100 may have an application installed on it. The application and webpage can provide an interface. The terminal 100 can receive relevant parameters input by the user on the interface and send the parameters to the server 200. The server 200 can obtain the processing result based on the received parameters and return the processing result to the terminal 100.

[0056] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the received parameters on its own, without the need for the server to cooperate. This application embodiment is not limited to this.

[0057] The following description Figure 1 The product form of the mid-terminal 100;

[0058] The terminal 100 in this application embodiment can be a mobile phone, tablet computer, wearable device, vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc., and this application embodiment does not impose any restrictions on it.

[0059] Figure 2 A schematic diagram of an optional hardware structure for terminal 100 is shown.

[0060] refer to Figure 2 As shown, the terminal 100 may include a radio frequency unit 110, a first memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), a headphone jack 163 (optional), a first processor 170, an external interface 180, a power supply 190, and other components. Those skilled in the art will understand that... Figure 2 These are merely examples of terminals or multi-functional devices and do not constitute a limitation on terminals or multi-functional devices. They may include more or fewer components than shown in the illustration, or combine certain components, or use different components.

[0061] The input unit 130 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the portable multi-functional device. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect touch operations performed by the user on or near it (such as operations performed by the user using fingers, knuckles, styluses, or any suitable object on or near the touch screen), and drive the corresponding connection devices according to a pre-set program. The touch screen can detect the user's touch actions, convert the touch actions into touch signals and send them to the first processor 170, and can receive and execute commands sent by the first processor 170; the touch signal includes at least touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, various types of touch screens, such as resistive, capacitive, infrared, and surface acoustic wave, can be used to implement the touch screen. In addition to the touch screen 131, the input unit 130 may also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0062] Among them, the input device 132 can receive input data, etc.

[0063] The display unit 140 can be used to display information input by the user or information provided to the user, various menus of the terminal 100, interactive interfaces, file display, and / or playback of any multimedia file.

[0064] The first memory 120 can be used to store instructions and data. The first memory 120 may mainly include an instruction storage area and a data storage area. The data storage area can store various types of data, such as multimedia files and text; the instruction storage area can store software units such as operating systems, applications, and instructions required for at least one function, or subsets or extensions thereof. It may also include non-volatile random access memory; providing the first processor 170 with hardware, software, and data resources for managing the computing processing device, supporting control software and applications. It is also used for storing multimedia files, as well as storing running programs and applications.

[0065] The first processor 170 is the control center of the terminal 100. It connects various parts of the terminal 100 via various interfaces and lines. By running or executing instructions stored in the first memory 120 and calling data stored in the first memory 120, it performs various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the first processor 170 may include one or more processing units; preferably, the first processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the first processor 170. In some embodiments, the first processor 170 and the first memory 120 may be implemented on a single chip; in some embodiments, they may also be implemented on separate chips. The first processor 170 can also be used to generate corresponding operation control signals, send them to corresponding components of the computing processing device, read and process data in the software, especially read and process data and programs in the first memory 120, so that each functional module therein performs its corresponding function, thereby controlling the corresponding components to act according to the instructions.

[0066] The first memory 120 can be used to store software code related to the script generation method, and the first processor 170 can execute the steps of the script generation method, and can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to achieve the corresponding functions.

[0067] The radio frequency unit 110 (optional) can be used for receiving and transmitting signals during information transmission or calls. For example, it can receive downlink information from the base station and process it for the first processor 170; additionally, it can transmit uplink data to the base station. Typically, the RF circuit includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, the radio frequency unit 110 can also communicate wirelessly with network devices and other devices. This wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0068] In this embodiment of the application, the radio frequency unit 110 can send data to the server 200 and receive the processing results sent by the server 200.

[0069] It should be understood that the radio frequency unit 110 is optional and can be replaced with other communication interfaces, such as a network port.

[0070] The terminal 100 also includes a power supply 190 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the first processor 170 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0071] Terminal 100 also includes an external interface 180, which can be a standard Micro USB interface or a multi-pin connector, which can be used to connect terminal 100 to other devices for communication or to connect a charger to charge terminal 100.

[0072] Although not shown, terminal 100 may also include a flash, a Wireless Fidelity (WiFi) module, a Bluetooth module, sensors with various functions, etc., which will not be described in detail here. Some or all of the methods described below can be applied to, for example... Figure 2 In the terminal 100 shown.

[0073] The following description Figure 1 The product form of the mid-range server 200;

[0074] Figure 3 A structural diagram of a server 200 is provided, as follows: Figure 3 As shown, server 200 includes a first bus 201, a second processor 202, a communication interface 203, and a second memory 204. The second processor 202, the second memory 204, and the communication interface 203 communicate with each other via the first bus 201.

[0075] The first bus 201 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The first bus 201 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0076] The second processor 202 can be any one or more of the following processors: a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0077] The second memory 204 may include volatile memory, such as random access memory (RAM). The second memory 204 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0078] The second memory 204 can be used to store software code related to the script generation method, and the second processor 202 can execute the steps of the script generation method of the chip, and can also schedule other units to achieve corresponding functions.

[0079] It should be understood that the terminal 100 and server 200 can be centralized or distributed devices. The first processor 170 in the terminal 100 and the second processor 202 in the server 200 can be hardware circuits (such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, DSPs, microprocessors, or microcontrollers, etc.) or combinations of these hardware circuits. For example, the first processor 170 and the second processor 202 can be hardware systems with instruction execution capabilities, such as CPUs, DSPs, etc., or hardware systems without instruction execution capabilities, such as ASICs, FPGAs, etc., or combinations of the aforementioned hardware systems without instruction execution capabilities and hardware systems with instruction execution capabilities.

[0080] This application provides a script generation method. The script generation method of this application according to embodiments is described in detail below with reference to the accompanying drawings.

[0081] Reference Figure 4 , Figure 4 This is a flowchart illustrating a script generation method provided in an embodiment of this application. The method may include:

[0082] Step S401: Obtain the original video to be processed.

[0083] Here, the original video refers to the video for which scriptwriting is to be carried out.

[0084] Optionally, the original video can be a video on a video creation platform, a video on the Internet, or a video shot on a mobile phone or other electronic device.

[0085] Of course, the original video source can be other sources as well, and this application does not impose specific restrictions.

[0086] Step S402: Perform scene segmentation on the original video to obtain multiple scene videos.

[0087] Considering that video data is generally large, video content is difficult to understand and process, and long videos are prone to target tracking loss, this embodiment can perform segmentation processing on the original video to obtain multiple segmented videos contained in the original video in order to facilitate subsequent processing.

[0088] There are various basis for storyboarding, and this application provides, but is not limited to, the following.

[0089] The first method is to process the video into segments according to a fixed number of video frames. For example, if the number of video frames is x, the original video is divided into video segments of every x frames, thus obtaining multiple video segments.

[0090] The second method is to process the video into segments based on the scenes. For example, if the original video contains y scenes, then the original video will be divided into y video segments, resulting in y segmented videos.

[0091] Based on this, optionally, this embodiment can detect scene change information in the original video and divide the original video into multiple segmented videos according to the detected scene change information, wherein each of the multiple segmented videos belongs to a different scene.

[0092] Optionally, scene change information can be the scene's start timecode and / or end timecode.

[0093] Optionally, the scenedetect library can be used to detect scene change information in the original video, allowing the video to be segmented into multiple sub-videos based on this information. Here, the scenedetect library is a Python library for video scene detection that analyzes visual differences between video frames and utilizes various algorithms to detect scene changes. It can identify different scenes in the video and return the start and end timecodes for each scene, enabling users to segment the original video into multiple sub-videos based on these timecodes.

[0094] Of course, the basis for storyboarding can be other things, such as storyboarding according to the theme, etc., and this application does not impose specific limitations.

[0095] Step S403: Track and label the characters in multiple storyboard videos to obtain multiple character-labeled videos.

[0096] Considering that the script is generated based on the content understanding of the storyboard video, there may be situations where the script only objectively describes the video content, but the video content is not associated with the characters. For example, the generated script may be "A man in a white shirt gave an apple to a woman in a blue suit". Since it is not clear which character the "man in a white shirt" and the "woman in a blue suit" correspond to, the generated script is unusable.

[0097] To avoid the aforementioned problems, this application embodiment can track and label the characters in each storyboard video to obtain a character-labeled video corresponding to that storyboard video, thereby obtaining multiple character-labeled videos.

[0098] It should be noted that when tagging, the same character in multiple storyboard videos is given the same tag; that is, the same character in multiple character tagging videos is given the same tag, while different characters are given different tags.

[0099] For example, storyboard video 1 contains character 1 and character 2, and storyboard video 2 contains character 3 and character 4. If character 1 and character 3 are the same character, then in the character tagging video corresponding to storyboard video 1, character 1 is tagged with tag 1 and character 2 is tagged with tag 2. In the character tagging video corresponding to storyboard video 2, character 3 is tagged with tag 1 and character 4 is tagged with tag 3.

[0100] It should also be noted that the person being tracked and tagged in this embodiment can be either a person or an object, and the specific target can be determined according to the actual application scenario. This application does not impose any specific limitations.

[0101] Step S404: Perform scenario description processing on multiple character-tagged videos to obtain a target script containing the characters.

[0102] Specifically, this embodiment can understand the video content of multiple character-tagged videos and perform scene descriptions based on the tagged tags to obtain a target script containing characters.

[0103] The script generation method provided in this application considers that short video clips are more helpful for understanding video content and character tracking than complete videos. This application performs segmentation processing on the original video to obtain multiple segmented videos. Then, it tracks and tags the characters in these multiple segmented videos to obtain multiple character-tagged videos. Furthermore, it performs scene description processing on these character-tagged videos to obtain a target script containing the characters. By using segmentation, this application can avoid losing the tracking target during character tracking, improving the accuracy and reliability of tracking and tagging. Furthermore, segmentation makes the content of the character-tagged videos more comprehensible, improving the accuracy of scene description processing, and thus improving the quality of the target script. Simultaneously, through character tracking and tagging, the characters can be integrated into the target script, further improving the quality of the target script and ensuring consistency between the target script and the original video.

[0104] In some embodiments of this application, the process of step S403, "tracking and labeling the characters in multiple storyboard videos to obtain multiple character-labeled videos", is described.

[0105] In one possible implementation, this embodiment can use a storyboard-based character tagging system to track and tag characters in multiple storyboard videos, thereby obtaining multiple character-tagged videos.

[0106] Specifically, the process begins by tracking the characters in each of the multiple storyboard videos to obtain the character trajectories corresponding to each storyboard video.

[0107] Optionally, a pre-trained head detection model can be used to track people in each segment of the video, obtaining the person's trajectory for each segment. Here, the pre-trained head detection model is trained using training videos labeled with person trajectory tags as training data.

[0108] Optionally, the pre-trained head detection model can be a YOLOv8 model. Here, the YOLOv8 model can employ the DeepSORT multi-object tracking algorithm to simultaneously track multiple people in a storyboard video. This allows for the acquisition of multiple person trajectories when a storyboard video contains multiple people. The DeepSORT algorithm is primarily based on the SORT (Simple Online and Realtime Tracking) algorithm and incorporates deep learning techniques to improve tracking accuracy.

[0109] Furthermore, based on the character trajectories corresponding to multiple storyboard videos, character tags can be assigned to the characters in the multiple storyboard videos to obtain multiple character-tagged videos.

[0110] For example, characters belonging to the same character trajectory can be given the same identification (ID). Optionally, the identification can be a character identifier, such as an actor's name, or a predefined character number, etc. This application does not impose any restrictions.

[0111] In one optional embodiment, the process of "labeling characters in multiple storyboard videos according to the character trajectories corresponding to each of the multiple storyboard videos to obtain multiple character-labeled videos" may include: obtaining a pre-configured character face mapping table, wherein the character face mapping table includes the correspondence between the character identifiers (such as actor names) of all characters in the original video and their facial images; for each storyboard video in the multiple storyboard videos, obtaining the facial image of each character from the storyboard video according to the character trajectories corresponding to the storyboard video; performing similarity matching between the facial image of each character and the facial images in the character face mapping table to obtain a matching result; determining the character identifier of each character according to the matching result; labeling each character in the storyboard video according to the character identifier of each character to obtain the character-labeled video corresponding to the storyboard video, so as to obtain character-labeled videos corresponding to multiple storyboard videos respectively, which serve as multiple character-labeled videos.

[0112] Optionally, the process of "matching the facial image of each person with the facial images in the character face mapping table" may include: using a face recognition model to perform cosine similarity matching between the facial image of each person and the facial images in the character face mapping table. Here, the face recognition model is trained using training facial images labeled with cosine similarity and the corresponding character face mapping table as training data.

[0113] Taking the character face mapping table, which includes four mapping relationships (character identifier 1 - face image 1, character identifier 2 - face image 2, character identifier 3 - face image 3, and character identifier 4 - face image 4), as an example, the similarity of each character's face image (taking character a's face image as an example) with face image 1 to face image 4 can be calculated. Assuming the similarity is 0.5, 0.6, 0.7, and 0.8 respectively, then the character identifier of character a can be determined to be character identifier 4.

[0114] Optionally, the process of “labeling each character in the storyboard video according to each character’s character identifier” may include: printing each character’s character identifier as a label onto the head area of ​​the corresponding character in the storyboard video.

[0115] For example, see Figure 5 The diagram illustrates the labeling of a person's head within a video frame. Taking the actor's name as the character identifier as an example, this embodiment can label the actor's name onto the corresponding head area. For example, "Zhang San" can be labeled... Figure 5 The head area of ​​the figure on the left, "Li Si" is... Figure 5 The head area of ​​the figure on the right.

[0116] Of course, the character can be tagged in any area other than the head; this application does not impose any restrictions.

[0117] In summary, this application embodiment tracks and labels the characters in each storyboard video, so that the labeled video corresponding to each character in each storyboard video is marked with the character identifier of each character, such as the actor's name, which improves the character recognition and helps to distinguish the characters in the subsequent scene description processing, thereby improving the generation effect of the target script.

[0118] In some other embodiments of this application, the process of step S404, "performing scenario description processing on multiple character-tagged videos to obtain a target script containing characters," is described.

[0119] In one possible implementation, this embodiment can use a scenario description system based on a multimodal large model that incorporates character information to perform scenario description processing on multiple character-tagged videos to obtain a target script containing the characters.

[0120] Specifically, the process begins by understanding the context of each of the multiple character-tagged videos to obtain video content description information corresponding to each of the multiple character-tagged videos. The video content description information includes the character identifiers of the characters in the corresponding character-tagged videos.

[0121] Optionally, the process of "performing contextual understanding on multiple character-labeled videos to obtain video content description information corresponding to each character-labeled video" can include: implementation using a pre-trained video understanding multimodal large-scale model. Here, the training data used by the video understanding multimodal large-scale model during the training phase includes: multiple character-labeled training videos and corresponding video content description information. The multiple character-labeled training videos are obtained by tracking and labeling characters in multiple training storyboard videos.

[0122] Specifically, in this embodiment, a certain number of videos can be obtained from various channels such as the network and databases, and then multiple training storyboard videos can be obtained according to the storyboard processing method provided above. For example, the scenedetect library can be used to process the obtained videos to obtain 1000 training storyboard videos.

[0123] Multiple training storyboard videos are fed into a storyboard-based character tagging system to track and tag the characters in the multiple training storyboard videos, resulting in multiple character tagging training videos.

[0124] Next, this application allows for manual description of the video content of multiple character tagging training videos, resulting in video content description information corresponding to each of the multiple character tagging training videos. It is worth noting that the manually generated video content description information must include the character identifiers from the character tagging training videos, for example, Figure 5 The corresponding video content description is "Zhang San and Li Si are talking".

[0125] After the above preprocessing, this embodiment obtains multiple character tagging training videos and corresponding video content description information, which can be used as a training set to fine-tune and pre-train the video understanding multimodal base model to obtain the above-mentioned pre-trained video understanding multimodal model (the video understanding multimodal model trained by the training set can already meet the requirement of this embodiment to output video content description information containing character tags, so it is not necessary to fine-tune and pre-train the video understanding multimodal model again based on a new training set).

[0126] Optionally, the large-scale video understanding multimodal foundation model can be the Qwen2-VL large-scale video understanding multimodal model. Here, Qwen2-VL is a large visual language model with three models of different parameter sizes: 2 billion, 7 billion, and 72 billion. Qwen2-VL is optimized for dialogue scenarios and outperforms other open-source dialogue models in most human evaluations of usefulness and safety. Optionally, this embodiment selects the 7 billion (7B) model Qwen2-VL-7B-Instruct as the large-scale video understanding multimodal foundation model.

[0127] Optionally, when fine-tuning the large multimodal foundation model for video understanding, the LORA (Low-Rank Adaptation) method in PEFT (Parameter Efficient Fine-tuning) can be used for fine-tuning.

[0128] Optional Lora configuration parameters include: matrix rank r=64, scaling factor lora_alpha=16, lora_dropout=0.05 (lora_dropout is used to randomly "drop out" (i.e. temporarily remove) some neurons in the network during training to prevent overfitting), training epochs=20, and learning rate=0.001.

[0129] After training the video understanding multimodal large model as described above, this embodiment can obtain the first text prompt word, which includes the basic elements of the script required to generate the target script; inputting multiple character-tagged videos and the first text prompt word into the video understanding multimodal large model together, the video content description information corresponding to the multiple character-tagged videos is obtained.

[0130] Optionally, the basic elements of a script may include one or more of the following: dialogue information, environmental information, and descriptions of character actions.

[0131] Of course, the basic elements of a script can be other than those specified in this application.

[0132] Furthermore, this embodiment can integrate the video content description information corresponding to multiple character-marked videos to obtain the target script.

[0133] Optionally, the process of "integrating the video content description information corresponding to the videos tagged with multiple characters to obtain the target script" can be achieved through a large language model.

[0134] Specifically, in this embodiment, a second text prompt word can be obtained. The second text prompt word is used to prompt the large language model to integrate the video content description information corresponding to the multiple character tag videos. The video content description information corresponding to the multiple character tag videos and the second text prompt word are input together into the large language model to obtain the target script output by the large language model.

[0135] Optionally, the second text prompt may include prompts that integrate the video content description information corresponding to each of the storyboard videos. The second text prompt may also include other optimized prompts; this application does not impose any limitations on this.

[0136] Optionally, the aforementioned large language model can be GPT-4 (Generative Pre-trained Transformer 4). Of course, other large language models are also possible, and this application does not impose any limitations.

[0137] In summary, this embodiment employs a multimodal video understanding model to generate video content description information. Compared to other large models, it can more accurately understand the video content of character-tagged videos, thereby generating more accurate video content description information. Furthermore, a large language model is used to integrate the video content description information corresponding to multiple storyboard videos, ensuring that the generated target script includes character identifiers for each character, thus enhancing the usability of the target script.

[0138] Experiments have verified that the script generation method provided in this embodiment can accurately integrate the characters in the original video into the script content description (by tagging the characters), and automatically generate a high-quality, highly relevant, content-rich, and professional target script. This target script does not require secondary manual calibration and its effect is comparable to that of a manually produced script, greatly saving manpower and improving the generation effect from video to script.

[0139] The above describes a script generation method provided by the embodiments of this application. The following will describe the apparatus for performing the above script generation method.

[0140] Please see Figure 6 , Figure 6 This is a schematic diagram of a script generation device provided in an embodiment of this application. Figure 6 As shown, the device may include:

[0141] Video acquisition module 501 is used to acquire the original video to be processed;

[0142] The video storyboard module 502 is used to process the original video into multiple storyboard videos.

[0143] The tracking and labeling module 503 is used to track and label characters in multiple storyboard videos to obtain multiple character-labeled videos;

[0144] The script generation module 504 is used to process the scene description of multiple character-tagged videos to obtain the target script containing the characters.

[0145] In one possible implementation, the above video storyboard module may include: a scene detection module and a scene storyboard module;

[0146] The scene detection module is used to detect scene change information in the original video;

[0147] The scene segmentation module is used to divide the original video into multiple segmented videos based on scene change information, with each segmented video belonging to a different scene.

[0148] In one possible implementation, the above tracking and marking module may include: a person tracking module and a character marking module;

[0149] The character tracking module is used to track characters in multiple storyboard videos separately, and obtain the character trajectories corresponding to each storyboard video.

[0150] The character tagging module is used to tag characters in multiple storyboard videos based on the character trajectories corresponding to each video, resulting in multiple character-tagged videos.

[0151] In one possible implementation, the aforementioned character tagging module may include: a mapping table acquisition module and a face matching module;

[0152] The mapping table acquisition module is used to acquire a pre-configured character face mapping table, which includes the correspondence between the character identifiers and facial images of all characters in the original video.

[0153] The face matching module is used for each of the multiple storyboard videos: based on the character trajectory corresponding to the storyboard video, it obtains the face image of each character in the storyboard video; it performs similarity matching between the face image of each character and the face image in the character face mapping table to obtain the matching result, and determines the character identifier of each character based on the matching result; it tags each character in the storyboard video according to the character identifier, and obtains the character-tagged video corresponding to the storyboard video; thus obtaining the character-tagged videos corresponding to multiple storyboard videos, which are used as multiple character-tagged videos.

[0154] In one possible implementation, the script generation module described above may include: a scenario understanding module and an information integration module;

[0155] The context understanding module is used to perform context understanding on multiple character-tagged videos to obtain video content description information corresponding to each character-tagged video. The video content description information includes the character identifier of the corresponding character in the character-tagged video.

[0156] The information integration module is used to integrate the video content description information corresponding to multiple character-marked videos to obtain the target script.

[0157] In one possible implementation, the above-mentioned scenario understanding module may include: a first prompt acquisition module and a video description module;

[0158] The first prompt acquisition module is used to acquire the first text prompt word, wherein the first text prompt word includes the basic script elements required to generate the target script;

[0159] The video description module is used to input multiple character-labeled videos and the first text prompt word into the pre-trained video understanding multimodal large model to obtain video content description information corresponding to each of the multiple character-labeled videos;

[0160] The training data used in the training phase of the video understanding multimodal large model includes: multiple character-labeled training videos and corresponding video content description information. The multiple character-labeled training videos are obtained by tracking and labeling the characters in multiple training storyboard videos.

[0161] In one possible implementation, the information integration module mentioned above may include: a second prompt acquisition module and a video description integration module;

[0162] The second prompt acquisition module is used to acquire the second text prompt words, wherein the second text prompt words are used to prompt the large language model to integrate the video content description information corresponding to the multiple character tag videos together;

[0163] The video description integration module is used to input the video content description information and second text prompts corresponding to the videos tagged with multiple characters into the large language model, and obtain the target script output by the large language model.

[0164] The script generation apparatus provided in this application corresponds to the script generation method provided above. For details, please refer to the above description, which will not be repeated here.

[0165] This application also provides an electronic device in its embodiments. (See reference...) Figure 7The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 7 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0166] like Figure 7 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing unit 601, ROM 602, and RAM 603 are interconnected via a second bus 604. An input / output (I / O) interface 605 is also connected to the second bus 604.

[0167] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, memory cards, hard drives, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0168] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the script generation methods provided in this application.

[0169] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the script generation methods provided in this application.

[0170] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0171] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0172] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0173] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A script generation method, characterized in that, include: Obtain the original video to be processed; The original video is split into multiple segments to obtain multiple split-scene videos; The characters in the multiple storyboard videos are tracked and tagged to obtain multiple character-tagged videos. The tracking and tagging is used to identify the characters in the character-tagged videos using tags. Based on the tags of the characters in the multiple character tagging videos, the multiple character tagging videos are subjected to scenario description processing to obtain a target script containing the characters, so as to integrate the characters into the target script. The scenario description processing is used to obtain the character identifiers of the characters from the character tagging videos, so that the target script contains the character identifiers of each character. The step of performing scenario description processing on the multiple character-tagged videos to obtain a target script containing the characters includes: performing scenario understanding on each of the multiple character-tagged videos to obtain video content description information corresponding to each of the multiple character-tagged videos, wherein the video content description information includes the character identifiers of the characters in the corresponding character-tagged videos; and integrating the video content description information corresponding to each of the multiple character-tagged videos to obtain the target script. The step of performing contextual understanding on the multiple character-tagged videos to obtain video content description information corresponding to each of the multiple character-tagged videos includes: obtaining a first text prompt word, wherein the first text prompt word includes the basic elements of the script required to generate the target script; and inputting the multiple character-tagged videos and the first text prompt word into a pre-trained video understanding multimodal large model to obtain video content description information corresponding to each of the multiple character-tagged videos.

2. The script generation method according to claim 1, characterized in that, The original video is processed into multiple split-scene videos, including: Detect scene change information in the original video; Based on the scene change information, the original video is divided into multiple segmented videos, each of which belongs to a different scene.

3. The script generation method according to claim 1, characterized in that, The process of tracking and tagging characters in the multiple storyboard videos to obtain multiple character-tagged videos includes: Character tracking is performed on the multiple storyboard videos to obtain the character trajectories corresponding to the multiple storyboard videos; Based on the character trajectories corresponding to the multiple storyboard videos, the characters in the multiple storyboard videos are tagged, resulting in the multiple character-tagged videos.

4. The script generation method according to claim 3, characterized in that, The step of tagging characters in the multiple storyboard videos according to the character trajectories corresponding to each of the multiple storyboard videos, to obtain the multiple character-tagged videos, includes: Obtain a pre-configured character face mapping table, wherein the character face mapping table includes the correspondence between the character identifiers and facial images of all characters in the original video; For each of the multiple storyboard videos: Based on the character trajectories corresponding to the storyboard video, obtain the facial image of each character from the storyboard video; The facial image of each character is matched with the facial images in the character facial mapping table to obtain the matching result, and the character identifier of each character is determined based on the matching result; Based on the character identifier of each character, each character in the storyboard video is tagged, resulting in a character-tagged video corresponding to the storyboard video. The character tag videos corresponding to the multiple storyboard videos are obtained as the multiple character tag videos.

5. The script generation method according to any one of claims 1 to 4, characterized in that, The training data used in the training phase of the video understanding multimodal large model includes: multiple character-labeled training videos and corresponding video content description information. The multiple character-labeled training videos are obtained by tracking and labeling characters in multiple training storyboard videos.

6. The script generation method according to claim 5, characterized in that, The process of integrating the video content description information corresponding to the tagged videos of the multiple characters to obtain the target script includes: Obtain the second text prompt word, wherein the second text prompt word is used to prompt the large language model to integrate the video content description information corresponding to the multiple character tag videos; The video content description information corresponding to the multiple character-tagged videos and the second text prompt words are input into the large language model to obtain the target script output by the large language model.

7. A script generation device, characterized in that, include: The video acquisition module is used to acquire the original video to be processed; The video segmentation module is used to perform segmentation processing on the original video to obtain multiple segmented videos; The tracking and labeling module is used to track and label the characters in the multiple storyboard videos to obtain multiple character-labeled videos. The tracking and labeling is used to identify the characters in the character-labeled videos using tags. The script generation module is used to perform scenario description processing on the multiple character tag videos based on the tags of the characters in the multiple character tag videos, to obtain a target script containing the characters, so as to integrate the characters into the target script. The scenario description processing is used to obtain the character identifiers of the characters from the character tag videos, so that the target script contains the character identifiers of each character. The step of performing scenario description processing on the multiple character-tagged videos to obtain a target script containing the characters includes: performing scenario understanding on each of the multiple character-tagged videos to obtain video content description information corresponding to each of the multiple character-tagged videos, wherein the video content description information includes the character identifiers of the characters in the corresponding character-tagged videos; and integrating the video content description information corresponding to each of the multiple character-tagged videos to obtain the target script. The step of performing contextual understanding on the multiple character-tagged videos to obtain video content description information corresponding to each of the multiple character-tagged videos includes: obtaining a first text prompt word, wherein the first text prompt word includes the basic elements of the script required to generate the target script; and inputting the multiple character-tagged videos and the first text prompt word into a pre-trained video understanding multimodal large model to obtain video content description information corresponding to each of the multiple character-tagged videos.

8. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the script generation method as described in any one of claims 1 to 6.

9. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the script generation method as described in any one of claims 1 to 6.

Images