Video generation method and device based on large model, electronic equipment and medium

Abstract

The invention provides a video generation method and device based on a large model, electronic equipment and a medium, and relates to the technical field of artificial intelligence, in particular to the fields of large models, multimedia, voice technologies, video technologies and the like. According to the specific implementation scheme of the method, an initial text is analyzed through a first large model, subtitle information is generated, and the subtitle information comprises information used for plot deduction for all roles in the initial text; synthesizing the audio information of each role according to the preset audio and the subtitle information; performing mirror splitting processing on the subtitle information to generate mirror splitting picture information; and generating the subtitle information, the audio information and the split picture information into a target video for explaining the initial text according to a development sequence of plots in the initial text.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the fields of large models, multimedia, voice technology, video technology, and specifically to video generation methods, devices, electronic devices, and media based on large models. Background Art

[0002] The large model is based on deep learning technology, can capture the complex relationships between words, accurately understand user instructions, and is widely used in the field of artificial intelligence.

[0003] With the development of digital media, interpreting the plots of popular novels in the form of commentary videos can present the novel text to users in a more vivid and vivid way by combining visual and auditory means, which is more in line with the diversified needs of current users. Summary of the invention

[0004] The present disclosure provides a large model-based video generation method, device, electronic device and medium.

[0005] According to one aspect of the present disclosure, a large model-based video generation method is provided, comprising: parsing an initial text using a first large model to generate subtitle information, wherein the subtitle information includes information for interpreting a plot for each character in the initial text; synthesizing audio information of each character according to predetermined audio and subtitle information; storyboarding the subtitle information to generate storyboard image information; and generating a target video for explaining the initial text using the subtitle information, audio information, and storyboard image information in the order of development of the plot in the initial text.

[0006] According to another aspect of the present disclosure, a video generation device based on a large model is provided, comprising: a parsing module, a synthesis module, a storyboard module and a generation module. The parsing module is used to parse the initial text using the large model and generate subtitle information, wherein the subtitle information includes information for each character in the initial text for interpreting the plot. The synthesis module is used to synthesize the audio information of each character according to predetermined audio and subtitle information. The storyboard module is used to storyboard the subtitle information and generate storyboard picture information. The generation module is used to generate a target video for explaining the initial text according to the development sequence of the plot in the initial text using the subtitle information, audio information and storyboard picture information.

[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0008] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method as described above.

[0009] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the method described above when executed by a processor.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0012] Figure 1 An exemplary system architecture to which the video generation method and apparatus according to an embodiment of the present disclosure can be applied is schematically shown;

[0013] Figure 2 The flowchart of the video generation method according to the embodiment of the present disclosure is schematically shown;

[0014] Figure 3 A schematic diagram schematically shows a video generation method according to an embodiment of the present disclosure;

[0015] Figure 4 A schematic diagram schematically shows a method of synthesizing the audio of each character according to subtitle information according to an embodiment of the present disclosure;

[0016] Figure 5A A schematic diagram schematically shows how to perform storyboard processing on subtitle information to obtain storyboard images according to an embodiment of the present disclosure;

[0017] Figure 5B A schematic diagram schematically shows a schematic diagram of performing storyboard processing on subtitle information to obtain storyboard images according to another embodiment of the present disclosure;

[0018] Figure 5C A schematic diagram schematically shows a schematic diagram of performing storyboard processing on subtitle information to obtain storyboard images according to another embodiment of the present disclosure;

[0019] Figure 6 A schematic diagram schematically shows a video generation method according to another embodiment of the present disclosure;

[0020] Figure 7 A block diagram schematically shows a video generating device according to an embodiment of the present disclosure; and

[0021] Figure 8A block diagram of an electronic device suitable for implementing a video generating method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0022] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0023] Novels mainly describe the plot in text form, while film and television works mainly present the plot through scenes, voice dialogues between characters, and a combination of vision and hearing. This change in presentation requires a series of complex production processes to achieve, such as: refining and summarizing the text of the novel; dubbing the characters in the novel; drawing video images for the scenes described in the novel, etc. These complex productions lead to a long production cycle for the commentary video.

[0024] As users pay more and more attention to commentary videos of popular novels, it is imperative to speed up the production efficiency of commentary videos.

[0025] Therefore, the disclosed embodiment provides a video generation method based on a large model, which uses the semantic understanding and generation capabilities of the first large model to parse the initial text, and on the basis of fully summarizing and refining the plot in the initial text, can also accurately capture the wonderful plot, and generate subtitle information for interpreting the dialogue of the characters in these wonderful plots, providing rich interpretation content for the commentary video. Then, based on the subtitle information, the audio and storyboard images of each character are synthesized respectively, thereby shortening the production cycle of the commentary video and improving the generation efficiency of the commentary video.

[0026] Figure 1 An exemplary system architecture to which the video generation method and apparatus according to an embodiment of the present disclosure can be applied is schematically shown.

[0027] It should be noted that Figure 1 The examples shown are only examples of system architectures to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. For example, in another embodiment, the exemplary system architecture to which the video generation method and apparatus can be applied may include a terminal device, but the terminal device may implement the video generation method and apparatus provided by the embodiments of the present disclosure without interacting with the server.

[0028] like Figure 1As shown, the system architecture 100 according to this embodiment may include a terminal device 101, a network 102 and a server 103. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired and / or wireless communication links, etc.

[0029] The terminal device 101 may be any electronic device having a display screen and supporting web browsing, including but not limited to a smart phone, a tablet computer, a laptop computer, a desktop computer, and the like.

[0030] The server 103 may be a server that provides various services, such as a background management server (only as an example) that provides support for the content browsed by the user using the terminal device 101. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device.

[0031] The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system. It solves the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server can also be a server for a distributed system, or a server combined with blockchain.

[0032] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc.

[0033] For example, a user may input an initial text 110 into the terminal device 101, and the terminal device 101 may send the initial text 110 to the server 103, which executes the video generation method provided by the embodiment of the present disclosure and returns the target video to the terminal device 101 for display to the user.

[0034] The video generation method provided in the embodiment of the present disclosure may also be generally performed by the server 103. Accordingly, the video generation device provided in the embodiment of the present disclosure may generally be arranged in the server 103. The video generation method provided in the embodiment of the present disclosure may also be performed by a server or server cluster that is different from the server 103 and can communicate with the terminal device 101 and / or the server 103. Accordingly, the video generation device provided in the embodiment of the present disclosure may also be arranged in a server or server cluster that is different from the server 103 and can communicate with the terminal device 101 and / or the server 103.

[0035] It should be noted that the video generation method provided in the embodiment of the present disclosure can generally be executed by the terminal device 101. Correspondingly, the video generation apparatus provided in the embodiment of the present disclosure can also be set in the terminal device 101.

[0036] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0037] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0038] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0039] Figure 2 The flowchart of the video generation method according to the embodiment of the present disclosure is schematically shown.

[0040] like Figure 2 As shown, the method 200 includes operations S210 to S240.

[0041] In operation S210, the initial text is parsed using the first large model to generate subtitle information.

[0042] In operation S220, audio information of each character is synthesized based on predetermined audio and subtitle information.

[0043] In operation S230, storyboard processing is performed on the subtitle information to generate storyboard picture information.

[0044] In operation S240, the subtitle information, the audio information, and the storyboard information are combined in the order of the plot development in the initial text to generate a target video for explaining the initial text.

[0045] According to an embodiment of the present disclosure, the first large model may be a large language model (LLM). The initial text may be a manuscript of a popular novel, and the subtitle information may include information for each character in the initial text for interpreting the plot. The information for each character to interpret the plot includes but is not limited to the action information and dialogue information of each character.

[0046] For example, the initial text can be "Sure enough, after a brief moment of ecstasy, A immediately revealed a hideous smile. He held a huge ball of light in both hands and slowly flew towards an ancient castle in the distance." The initial text and the parsing requirements can be input into the large language model as prompts. The parsing requirements can be, for example, "Please split the manuscript into segments in sequence, and split them according to who said the original words. If it is not said by the original character, mark it as narration. Each line in the split result is presented in the format of character + original content." The subtitle information output by the large language model may include two lines. The first line can be "Narration: Sure enough, after a brief moment of ecstasy, A immediately revealed a hideous smile." The second line can be "A: He held a huge ball of light in both hands and slowly flew towards an ancient castle in the distance."

[0047] According to an embodiment of the present disclosure, the predetermined audio may be a reference sound distinguished according to attributes such as gender and age of a person, for example, the voice of a young man, the voice of a girl, the voice of an old woman, and the like.

[0048] According to the embodiments of the present disclosure, reference voices can be matched according to the gender, age and other attributes of each character in the initial text. For example, if Person A in the initial text is a young man, then the subtitle information of Person A can be matched to the young man's voice and generate the audio of Person A by using the text-to-speech (TTS) technology.

[0049] According to an embodiment of the present disclosure, the speech synthesis technology may adopt a speech synthesis technology using zero-shot TTS.

[0050] In the subtitle information, the action information and language information of each character can be listed in separate lines, and two adjacent lines can also be action information and language information for the same character. The order of the subtitle information of each character is determined according to the order of the lines in the initial text.

[0051] According to an embodiment of the present disclosure, subtitle information may be storyboarded according to a preset storyboard rule, which includes but is not limited to the relevance of the actions and languages ​​of the characters, or the relevance of the emotions expressed in the actions and languages, or the relevance of the scenes.

[0052] For example: Lines 3 to 5 in the subtitle information are all descriptions of A performing a certain action in a certain place. Therefore, the subtitle information of lines 3 to 5 can be processed as a storyboard.

[0053] According to an embodiment of the present disclosure, subtitle information corresponding to each storyboard can be matched to a storyboard picture used to describe the storyboard scene from a pre-configured picture material library.

[0054] For example: A storyboard describes a scene of a young man practicing in a dark valley. You can match a picture of a young man wearing martial arts clothing in a dim environment from the picture material library as the picture for the storyboard.

[0055] According to the embodiments of the present disclosure, the subtitle information, audio information and storyboard information can be matched according to a pre-set alignment rule, and the alignment rule includes but is not limited to the audio of each character corresponding to the subtitle content, the scene described by the narration and the content of the storyboard corresponding to the content of the storyboard, etc. Then, using the video synthesis technology, the aligned and matched subtitle information, audio information and storyboard information are generated according to the development order of the plot in the initial text to generate a target video for explaining the initial text.

[0056] It should be noted that the speech synthesis technology and video synthesis technology described above are relatively mature technologies and will not be elaborated here.

[0057] According to the embodiments of the present disclosure, the semantic understanding and generation capabilities of the first large model are used to parse the initial text. On the basis of fully summarizing and refining the plot in the initial text, the wonderful plot can be accurately captured, and subtitle information for interpreting the dialogues of the characters in these wonderful plots can be generated, providing rich interpretation content for the commentary video. Then, the audio and storyboard images of each character are synthesized based on the subtitle information, thereby shortening the production cycle of the commentary video and improving the generation efficiency of the commentary video.

[0058] Reference below Figure 3~Figure 6 , combined with specific embodiments Figure 2 The method shown is further explained.

[0059] According to an embodiment of the present disclosure, the above operation S210 may include the following operations: using the first large model to perform plot analysis on the initial text to generate a deductive text; and using the first large model to perform role separation on the deductive text to generate subtitle information.

[0060] Figure 3 A schematic diagram of a video generating method according to an embodiment of the present disclosure is schematically shown.

[0061] like Figure 3 As shown, in this embodiment 300, first, the original text 301 and Prompt A are input into LLM 310 for plot summary and content optimization to generate interpretation text 302. Then, the interpretation text 302 and Prompt B are input into LLM 310 for role separation to obtain subtitle information 303.

[0062] Next, the reference audio 304 is matched based on the subtitle information 303, and the target audio 305 of each character is synthesized using speech synthesis technology. In addition, the subtitle information 303 is storyboarded to generate a storyboard screen 306.

[0063] Finally, the storyboard image 306, the target audio 305 and the subtitle information 303 are synthesized into a commentary video 307 using video synthesis technology.

[0064] According to an embodiment of the present disclosure, the interpretation text includes the plot of the initial text and information of each character used to interpret the plot. Compared with the original text, the interpretation text adds interpretation content associated with the plot on the basis of the original plot, for example, dialogues, actions, expressions and other content can be added to make the characters in the commentary video more vivid and lively.

[0065] According to the embodiments of the present disclosure, when the large language model is used to split the roles of the interpretation text, the roles can be split according to the roles, for example: narration, protagonist, supporting role, etc. The narration's task is mainly to introduce the story background, the protagonist's task is mainly to interpret the story plot, and the supporting role's task may be to promote the development of the plot.

[0066] According to the embodiments of the present disclosure, the large language model can also be used to split the interpretation text according to the dialogue content. For example, a certain segment of the interpretation text can be "This is the legendary mysterious land. After carefully inspecting the surroundings, B did not show any inferiority. He said to C: Wait for me here, and then strode towards the center of the mysterious land." The dialogue content "Wait for me here" between B and C can be used as the subtitle for the character B, and other action descriptions and environmental descriptions are all subtitles of the narration.

[0067] According to the embodiments of the present disclosure, by utilizing the semantic understanding and generation capabilities of a large language model and fully refining the storyline of the initial text, the characters' dialogues, actions, expressions and other contents are enhanced to make the characters' images more three-dimensional and vivid, thereby further improving the quality of the commentary video.

[0068] The semantic understanding and generation capabilities of the large language model are stimulated based on Prompt. In this embodiment 300, Prompt A may be parsing prompt information, and the parsing prompt information includes examples of parsing text requirements and parsing reference texts.

[0069] According to an embodiment of the present disclosure, the initial text and parsing prompt information may be input into the first large model, and the deduced text may be output.

[0070] According to an embodiment of the present disclosure, the text analysis requirements may be pre-set according to the requirements of the application scenario. For example, for a martial arts novel, the text analysis requirements may include: skipping the straightforward literary plots and introducing the fighting plots in detail.

[0071] According to an embodiment of the present disclosure, an example of parsing a reference text may include a reference text segment and a parsing result of the reference text segment. For example, the reference text segment may be a text segment in "Water Margin" in which Lu Zhishen punches Zhen Guanxi. The parsing result may only briefly summarize the cause of the story, and focus on describing the action + dialogue plot part in the original text segment in which Lu Zhishen punches Zhen Guanxi.

[0072] According to an embodiment of the present disclosure, the parsing prompt information may also include parsing roles and parsing tasks, etc. The parsing role may be a creator of a video explaining a popular novel. The parsing task may be to perform a video interpretation task for a given original novel segment and generate a deductive text.

[0073] According to the embodiments of the present disclosure, the user's parsing requirements for the initial text can be clearly indicated to the large language model based on the parsing prompt information, and the standards and reference examples for the large language model in performing text parsing tasks are clarified so that the large language model can efficiently complete the text parsing task according to the standards and reference examples.

[0074] In addition, in this embodiment 300, a splitting prompt information Prompt B can be constructed for the role splitting task, and the interpretation text and the splitting prompt information are input into the first large model to generate subtitle information.

[0075] According to an embodiment of the present disclosure, the splitting prompt information may include examples of role splitting requirements and splitting reference texts.

[0076] According to the embodiments of the present disclosure, the role splitting requirement can also be pre-set according to the requirements of the application scenario. For example, the attribution splitting is performed according to who said the words in the original text, and if the words are not said by the original character, they are marked as narration. The role splitting requirement can also include the output format of the splitting result, for example, each line of the splitting result is output in the format of "explanator + original content".

[0077] According to an embodiment of the present disclosure, an example of splitting a reference text may include a reference text segment and a result of role splitting of the reference text segment. For example, a reference text segment may be "A and B are participating in a debate, and A, as the first debater of the affirmative side, speaks first: xxx. B is not to be outdone, and as the first debater of the negative side, he loudly expounds the negative viewpoint: yyy." The result of role splitting may include at least 4 lines, and the first line may be "Narration: A and B are participating in a debate, and A, as the first debater of the affirmative side, speaks first." The second line may be "A: xxx." The third line may be "Narration: B is not to be outdone, and as the first debater of the negative side, he loudly expounds the negative viewpoint." The fourth line may be "B: yyy."

[0078] According to an embodiment of the present disclosure, based on the role prompt information, the user's parsing requirements for the interpreted text can be clearly indicated to the large language model, and the standards and reference examples for the large language model in performing the role splitting task are clarified, so that the large language model can efficiently complete the role splitting task according to the standards and reference examples.

[0079] The explanation video allows users to experience the plot of the novel in an immersive way from the perspective of vision and hearing. People are usually more sensitive to sound than vision. Therefore, in film and television works, the higher the degree of match between the character's voice and the character's personality, the easier it is to immerse people in the scenes rendered in the film and television works, and the higher the user satisfaction.

[0080] Therefore, in the embodiment of the present disclosure, multiple candidate reference audios can be preconfigured, and these candidate reference audios are not only for different ages and different genders, but also for different character styles, such as a graceful lady in the boudoir, a bohemian knight, and so on.

[0081] According to an embodiment of the present disclosure, synthesizing audio information of each character based on predetermined audio and subtitle information may include the following operations: parsing the subtitle information using a first large model to generate style information of each character; matching the target reference audio of each character from the candidate reference audio based on the style information of each character; and synthesizing the audio information of each character based on the target reference audio and subtitle information of each character.

[0082] According to an embodiment of the present disclosure, the style of a character can be comprehensively determined based on the dialogue content between the character and other characters in the subtitle information, the action description content surrounding the character, the scene in which the character appears, etc.

[0083] For example, when a character first appears, the narrator describes it as "Before seeing the person, we first ask about his voice, and his hearty laughter accompanied by the mature scent of cosmetics rushes towards everyone." "Mature scent of cosmetics" can determine that the character is a mature female character, and "hearty laughter" can determine that the character has a hearty personality. Combined with "rushing towards everyone", it can be inferred that the character is hearty and a little bit aggressive.

[0084] The big model can be used to analyze the semantics of the subtitle information of the narration, and the character's style should be "a mature woman with a cheerful personality and a bit of acrimony". Then, the target reference audio of each character can be matched from the candidate reference audio according to the character's style. The character's style is similar to that of Wang Xifeng in "A Dream of Red Mansions". The dubbing of Wang Xifeng in the film and television work "A Dream of Red Mansions" can be used as the target reference audio to synthesize the audio of the character.

[0085] According to the embodiments of the present disclosure, by analyzing the character style, audio that conforms to the character style is synthesized for different characters, which further improves the adaptability of the character audio and the character style, enhances the user's sense of immersion from the auditory level, and improves the user's auditory experience.

[0086] Similar to the parsing task and role splitting task, for the style parsing task, it is also necessary to build corresponding prompts to stimulate the semantic understanding and generation capabilities of the large language model in order to efficiently complete the role style parsing task.

[0087] According to an embodiment of the present disclosure, subtitle information is parsed using the first large model to generate style information of each character, which may include the following operations: extracting information of each character from the subtitle information respectively; and inputting the information of each character and style prompt information into the first large model, and outputting the style information of each character.

[0088] Figure 4 A schematic diagram schematically illustrates a method of synthesizing the audio of each character based on subtitle information according to an embodiment of the present disclosure.

[0089] like Figure 4 As shown, in this embodiment 400, first, the subtitle information "A: xxx; A: sss" 422 of the character A is extracted from the subtitle information 421. It should be noted that the information of the character A includes but is not limited to the dialogue and actions of A, and may also include information for describing the scene in which A appears.

[0090] Then, the subtitle information of character A "A: xxx; A: sss" 422 and Prompt C are input into LLM 310, and the character style of A 423 is output. Prompt C can be style prompt information, and the style prompt information can include: determining a reference example of the character style of each reference character based on the information of each reference character. The reference example can include information such as the language, action, and appearance scene of the reference character, as well as the character style of the reference character, for example: a smart and alert child.

[0091] Next, the character style 423 of Person A and the subtitle information 424 of Person A are input into the speech synthesis model 320 to obtain the audio 425 of Person A. The speech synthesis model 320 has a plurality of candidate reference audios built in.

[0092] For example, the character style 423 of A can be a smart and alert child. The language synthesis model 320 can identify the character style of A and match it to the voice of a child that can express the style of being smart and alert. Then, the child's voice can be used in combination with A's subtitle information 424 for speech synthesis to generate A's audio 425.

[0093] According to an embodiment of the present disclosure, based on style hint information, a reference example for determining the character style based on the language and actions of each character can be provided to the large language model, so that the large language model can efficiently complete the character style parsing task based on the reference example, thereby improving the ability of the large language model in performing the character style parsing task.

[0094] The matching degree between the visual images and the scenes described in the original novel can visually draw users into the plot of the novel. Usually, the similarity of scene descriptions between different novels is low. Due to the limited image materials, using existing materials to adapt the novel scenes can easily cause visual fatigue to users and affect their viewing experience.

[0095] Therefore, the embodiment of the present disclosure first uses the first large model to split the subtitle information to generate storyboard description text; and based on the storyboard description text, generates storyboard picture information.

[0096] Figure 5A A schematic diagram schematically shows a storyboard screen obtained by storyboard processing of subtitle information according to an embodiment of the present disclosure.

[0097] like Figure 5A As shown, in this embodiment 500A, first, the subtitle information 421 and Promp D are input into LLM 310, and the storyboard description text 531 is output. The storyboard description text 531 may include the scene description of each storyboard and the dialogue information of each character in each storyboard. For example: Storyboard 1: [scene description] A: xxx; B: yyy.

[0098] According to an embodiment of the present disclosure, Promp D may be storyboard prompt information. The storyboard prompt information may include a reference example of storyboard description reference text obtained by storyboarding based on the interpretation text of each reference character. Subtitle information and storyboard prompt information may be input into the first model to output the storyboard description text.

[0099] According to an embodiment of the present disclosure, the storyboard prompt information may also include storyboard role information, for example: suppose you are a senior editor who is very good at storyboard insight and storyboard splitting. The storyboard prompt information may also include storyboard requirements, for example: the manuscript is sequentially storyboarded according to the rationality of the plot, and it can be output according to "storyboard identification-[scene description]-number of lines of storyboard description content in the subtitle information". [Scene description] is obtained by refining the scene content in the interpretation text, and may include the name of the place, and may also include a picture description of the place. For example: the storyboard description text may be: Storyboard 1-[A mysterious altar in a dark valley, occasionally emitting an inky blue flame]-the narration subtitles of lines 3-5.

[0100] According to the embodiments of the present disclosure, the editing ability of the large language model for the novel plot can be stimulated based on the storyboard prompt information, thereby obtaining the picture scene description information for each plot, so as to generate storyboard pictures corresponding to the picture scenes of each plot, improve the adaptability of the storyboard pictures to the novel plot, reduce user visual fatigue, increase the user's visual immersion, and further enhance the user experience.

[0101] Then, the storyboard description text and Prompt E may be input into the drawing master model 330, and the storyboard screen 532 may be output. The drawing master model may be, for example, a SDXL (DreamShaper XL) model.

[0102] According to an embodiment of the present disclosure, Prompt E may be a drawing prompt information, and the drawing prompt information may include a drawing role, for example: you are a senior illustrator. The drawing prompt information may also include drawing requirements, for example: according to the storyboard plot description, the scene storyboard drawing is performed, and the drawing content needs to express the plot tension so that the audience can intuitively feel the plot content.

[0103] According to the embodiments of the present disclosure, a large language model is used to storyboard subtitle information, and at least one storyboard description text can be provided for each plot. Storyboard images are then generated based on the storyboard description text, so that each plot is presented to the user in a unique visual manner, further enhancing the user's viewing experience.

[0104] Figure 5B A schematic diagram schematically shows a storyboard screen obtained by storyboard processing subtitle information according to another embodiment of the present disclosure.

[0105] like Figure 5B As shown, in embodiment 500B, the process of generating storyboard description text 531 using LLM 310 is the same as that in embodiment 500A, and will not be described in detail here.

[0106] Since the characters in the original novel are named after characters, such as Zhang San, Li Si, etc., they are also named after characters in the storyboard description text. However, during the storyboard drawing process, such character names may interfere with the drawing model. For example, if a character's name is Liang Moumou, the drawing model may recognize the character's surname as a beam of a building, resulting in an abrupt beam appearing in the storyboard.

[0107] Therefore, the interference with the large drawing model can be reduced by optimizing the storyboard description text. For example, the storyboard description text is optimized using the first large model to generate storyboard screen prompt information; and the storyboard screen prompt information is processed using the second large model to generate storyboard screen information.

[0108] For example: input the storyboard description text 531 and Prompt F into LLM310 to optimize Prompt F and generate optimized storyboard prompt information 532.

[0109] According to an embodiment of the present disclosure, Prompt F may be optimization prompt information, and the optimization prompt information includes a reference example for determining the character attributes of each reference character based on the name of each reference character. The storyboard description text and the optimization prompt information may be input into the first large model, and the storyboard screen prompt information may be output.

[0110] According to an embodiment of the present disclosure, the optimization prompt information may include optimizing roles, for example: You are an expert in Chinese culture. The optimization prompt information may also include optimization requirements, for example: The gender and age status of the corresponding named character will be accurately displayed through the character name. For example: Replace the character name with [Man], [Woman], [Child], etc. The reference example for determining the character attributes of each reference character based on the name of each reference character may include the name of the reference character and the replaced attribute name of the reference character. For example: Before optimization, it may be: [Zhang] playing with [Xiao Ming], and after optimization, it may be [Man] playing with [Child].

[0111] According to the embodiments of the present disclosure, by generalizing the character names and optimizing the storyboard prompt information, the interference of the character names on the large painting model can be reduced, and the accuracy of the storyboard content output by the large painting model can be improved.

[0112] When processing subtitle information, some descriptions with unclear plot features may be omitted, resulting in discontinuity in the storyboard content. For example, storyboard 1 corresponds to the descriptions of lines 1-3 in the subtitle information, and storyboard 2 corresponds to the descriptions of lines 5-8 in the subtitle information. It can be seen that the descriptions of line 4 do not appear in either storyboard 1 or storyboard 2. This will cause the storyboard images, audio, and subtitles to not be completely aligned, affecting the quality of video synthesis.

[0113] Therefore, the embodiment of the present disclosure further includes the following operations after obtaining the storyboard description text and before performing storyboard drawing: extracting mapping information between each storyboard and each description text from the storyboard description text; and based on the mapping relationship, in response to the presence of a missing mark between the description text in the i-th storyboard and the description text for the (i+1)-th storyboard, adding subtitle information corresponding to the missing mark to the description text of the i-th storyboard, where i is an integer greater than 1 and equal to 1.

[0114] Figure 5C A schematic diagram schematically shows a storyboard screen obtained by storyboard processing subtitle information according to another embodiment of the present disclosure.

[0115] like Figure 5C As shown, in this embodiment 500C, the process of using LLM310 to process subtitle information 533 to generate storyboard description text 534 is the same as that in embodiment 500B and will not be repeated here.

[0116] Then, the mapping information between each storyboard and each description text can be obtained by structuring the storyboard description text 534. The mapping information represents the corresponding relationship between the first identifier and the second identifier; the first identifier represents the order of each description text in the subtitle information; and the second identifier represents the order of each storyboard.

[0117] For example, in the mapping information, storyboard 1 corresponds to the description content of lines 1-2 in the subtitle information, and storyboard 2 corresponds to the description content of lines 4-5 in the subtitle information. At this time, the gap between storyboard 1 and storyboard 2 is marked as line 3 in the subtitle information, so the subtitle information of line 3 can be added to storyboard 1 to obtain the optimized storyboard description text 536: storyboard 1 corresponds to the description content of lines 1-3.

[0118] Next, the optimized storyboard description text 536 and the optimized prompt information Prompt F are input into LLM310 to optimize the prompt information, and the optimized storyboard prompt information 537 is obtained. Finally, the optimized storyboard prompt information 537 is input into the drawing model 330 to generate the storyboard screen 538.

[0119] According to the embodiments of the present disclosure, by extracting the mapping information between each storyboard and each description text, detecting whether there is any description text omission, and promptly adding the omitted description text to the corresponding storyboard, the impact of the description text omission on the alignment of the storyboard images, audio, and subtitles before video synthesis is effectively reduced, thereby improving the efficiency and quality of video synthesis.

[0120] Before synthesizing the storyboard images, the image style information of each character may also be obtained; and the storyboard image prompt information and the image style information may be processed using the second largest model to generate the storyboard image information of each storyboard.

[0121] According to an embodiment of the present disclosure, the picture style information may be input by a user through an interactive interface, and the input method includes but is not limited to selection of a predetermined picture style and input of customized picture style information.

[0122] For example, the interactive interface can display predetermined picture style information, such as cute anime style, simple and fresh style, etc. After the user selects a certain picture style information, the predetermined picture style information can be converted into a picture style prompt that can be recognized by the large model by calling the prompt template. Then, the storyboard picture prompt information and the picture style prompt are input into the painting large model to generate the storyboard picture information of each storyboard.

[0123] According to the embodiments of the present disclosure, the picture style determined by the user is integrated into the storyboard picture prompt information, so that the large model can perform the storyboard picture generation task according to the picture style specified by the user and output the storyboard picture that meets the user's intention, thereby reducing the number of repeated interactions between the user and the large model and improving the interaction efficiency.

[0124] Figure 6 A schematic diagram of a video generating method according to another embodiment of the present disclosure is schematically shown.

[0125] like Figure 6 As shown, in the embodiment 600, first, the LLM is used to perform plot analysis and character segmentation on the initial text 601 to generate subtitle information 602 of each character. Then, the LLM is used to perform storyboard segmentation on the subtitle information 602 to obtain the storyboard description text 603 of each storyboard. Then, the painting model is used to generate the corresponding storyboard screen 604 for each storyboard. Then, according to the subtitle information 402 of each character and the reference audio of each character, the audio 605 of each character is synthesized using speech synthesis technology. Then, the subtitles, audio, and storyboard screen of each character are aligned, and the video 607 is synthesized according to the plot development order in the initial text 601.

[0126] According to the embodiments of the present disclosure, the semantic understanding and generation capabilities of the large language model are used to extract the plot and split the characters of the novel, so that the key plots can be accurately captured and subtitles for each character can be generated. The speech synthesis model is used to generate audio for each character to improve the auditory expressiveness of each character. Then, the interpretation text is storyboarded, and a series of storyboards closely related to the plot are generated using the large drawing model. Finally, the subtitles, audio, and storyboards are aligned to synthesize the commentary video. A highly integrated automatic production process of the storyline, pictures, character sounds, and subtitles is achieved, which shortens the production cycle of the commentary video and improves the production efficiency of the commentary video.

[0127] Figure 7The block diagram of a video generating device according to an embodiment of the present disclosure is schematically shown.

[0128] like Figure 7 As shown, in embodiment 700 , the video generating device 700 may include: a narration module 710 , a synthesis module 720 , a storyboard module 730 and a generating module 740 .

[0129] The parsing module 710 is used to parse the initial text using the large model to generate subtitle information, wherein the subtitle information includes information for interpreting the plot for each character in the initial text.

[0130] Synthesis module 720, for synthesizing the audio information of each character according to the predetermined audio and subtitle information

[0131] The storyboard module 730 is used to perform storyboard processing on the subtitle information to generate storyboard picture information.

[0132] The generation module 740 is used to generate a target video for explaining the initial text according to the development sequence of the plot in the initial text by using the subtitle information, audio information and storyboard information.

[0133] According to an embodiment of the present disclosure, the storyboard module may include: a splitting submodule and a first generating submodule. The splitting submodule is used to split the subtitle information using the first large model to generate storyboard description text. The first generating submodule is used to generate storyboard picture information based on the storyboard description text.

[0134] According to an embodiment of the present disclosure, the first generating submodule may include: an optimizing unit and a first generating unit.

[0135] The optimization unit is used to optimize the storyboard description text by using the first large model to generate storyboard screen prompt information.

[0136] The first generating unit is used to generate storyboard picture information by calling the second large model based on the storyboard picture prompt information.

[0137] According to an embodiment of the present disclosure, the optimization unit may include: an input subunit. The input subunit is used to input the storyboard description text and optimization prompt information into the first large model, and output the storyboard screen prompt information; wherein the optimization prompt information includes a reference example for determining the character attributes of each reference character based on the name of each reference character.

[0138] According to an embodiment of the present disclosure, the first generation submodule further includes: an acquisition unit and a second generation unit. The acquisition unit is used to acquire the picture style information of each character. The second generation unit is used to generate the storyboard picture information of each storyboard by calling the second large model based on the storyboard picture prompt information and the picture style information.

[0139] According to an embodiment of the present disclosure, the storyboard module may include: an input submodule. The input submodule is used to input subtitle information and storyboard prompt information into a first model and output a storyboard description text; wherein the storyboard prompt information includes a reference example of a storyboard description reference text obtained by storyboarding based on the interpretation text of each reference role.

[0140] According to an embodiment of the present disclosure, the storyboard module further includes: an extraction submodule and an addition submodule. The extraction submodule is used to extract mapping information between each storyboard and each description text from the storyboard description text; the mapping information represents the corresponding relationship between the first identifier and the second identifier; the first identifier represents the order of each description text in the subtitle information; the second identifier represents the order of each storyboard. The addition submodule is used to add the subtitle information corresponding to the missing identifier to the description text of the i-th storyboard based on the mapping relationship in response to the presence of a missing identifier between the description text in the i-th storyboard and the description text for the i+1-th storyboard.

[0141] According to an embodiment of the present disclosure, the predetermined audio includes at least two candidate reference audios; the synthesis module may include: a parsing submodule, a matching submodule and a synthesis submodule.

[0142] The parsing submodule is used to parse the subtitle information using the first model to generate style information of each character.

[0143] The matching submodule is used to match the target reference audio of each character from the candidate reference audio according to the style information of each character.

[0144] The synthesis submodule is used to synthesize the audio information of each character based on the target reference audio and subtitle information of each character.

[0145] According to an embodiment of the present disclosure, the parsing submodule includes: an extraction unit and a first input unit. The extraction unit is used to extract information of each character from the subtitle information. The first input unit is used to output the information of each character and the style prompt information to the large model, and output the style information of each character; wherein the style prompt information includes a reference example of the character style of each reference character determined based on the information of each reference character.

[0146] According to an embodiment of the present disclosure, the parsing module includes: a plot parsing submodule and a role splitting submodule. The plot parsing submodule is used to perform plot parsing on the initial text using the first large model to generate a deductive text; wherein the deductive text includes the plot of the initial text and information about each role used to deduce the plot. The role splitting submodule is used to perform role splitting on the deductive text using the first large model to generate subtitle information.

[0147] According to an embodiment of the present disclosure, the plot parsing submodule includes: a second input unit, which is used to input the initial text and parsing prompt information into the first large model and output the deductive text; wherein the parsing prompt information includes examples of parsing text requirements and parsing reference texts.

[0148] According to an embodiment of the present disclosure, the role splitting submodule includes: a third input unit, used to input the deductive text and splitting prompt information into the first large model, and output the deductive text; wherein the splitting prompt information includes examples of role splitting requirements and splitting reference texts.

[0149] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0150] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0151] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method as described above.

[0152] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the method as described above.

[0153] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0154] like Figure 8As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 to a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0155] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0156] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as a video generation method. For example, in some embodiments, the video generation method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the XXX method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the video generation method in any other appropriate manner (e.g., by means of firmware).

[0157] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0158] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0159] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0160] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0161] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0162] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0163] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0164] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A video generation method based on a large model, include: Parsing the initial text using the first model to generate subtitle information, wherein the subtitle information includes information for interpreting the plot for each character in the initial text; Synthesizing audio information of each of the characters according to the predetermined audio and the subtitle information; Performing storyboard processing on the subtitle information to generate storyboard picture information; and The subtitle information, the audio information and the storyboard information are combined in the order of the plot development in the initial text to generate a target video for explaining the initial text.

2. The method according to claim 1, in, The performing storyboard processing on the subtitle information to generate storyboard picture information comprises: Using the first large model to split the subtitle information to generate storyboard description text; and Based on the storyboard description text, storyboard picture information is generated.

3. The method according to claim 2, wherein the storyboard information is generated based on the storyboard description text. include: Optimizing the storyboard description text using the first large model to generate storyboard screen prompt information; as well as The storyboard prompt information is processed using the second largest model to generate the storyboard information.

4. The method according to claim 3, in, The step of optimizing each of the description texts by using the first large model to generate storyboard prompt information includes: Inputting the storyboard description text and optimization prompt information into the first large model, and outputting the storyboard screen prompt information; The optimization prompt information includes reference examples for determining character attributes of each reference character based on the name of each reference character.

5. The method according to claim 3 or 4, further comprising: include: Obtaining the graphic style information of each of the characters; as well as The second large model is used to process the storyboard picture prompt information and the picture style information to generate storyboard picture information of each storyboard.

6. The method according to claim 2, in, The method of using the first large model to split the subtitle information and generate storyboard description text includes: Input the subtitle information and storyboard prompt information into the first model, and output the storyboard description text; The storyboard prompt information includes reference examples of storyboard description reference texts obtained by storyboarding based on the interpretation texts of each reference character.

7. The method according to claim 6, further comprising: include: Extracting mapping information between each storyboard and each description text from the storyboard description text; The mapping information represents the corresponding relationship between the first identifier and the second identifier; the first identifier represents the order of each of the description texts in the subtitle information; the second identifier represents the order of each of the storyboards; as well as Based on the mapping relationship, in response to the presence of a missing mark between the description text in the i-th storyboard and the description text for the i+1-th storyboard, the subtitle information corresponding to the missing mark is added to the description text of the i-th storyboard, where i is an integer greater than 1 and equal to 1.

8. The method according to claim 1, in, The predetermined audio includes at least two candidate reference audios; The step of synthesizing the audio information of each character according to the predetermined audio and the subtitle information includes: Parsing the subtitle information using the first model to generate style information of each character; According to the style information of each of the characters, matching the target reference audio of each of the characters from the candidate reference audios; and The audio information of each of the characters is synthesized according to the target reference audio of each of the characters and the subtitle information.

9. The method according to claim 8, in, The using the first large model to parse the subtitle information to generate style information of each character includes: Extracting information of each of the characters from the subtitle information; and Inputting the information and style prompt information of each of the characters into the first large model, and outputting the style information of each of the characters; The style prompt information includes reference examples for determining the character style of each reference character based on information of each reference character.

10. The method according to claim 1, in, The method of parsing the initial text using the first large model to generate subtitle information includes: Analyzing the plot of the initial text using the first model to generate a derived text; wherein the derived text includes the plot of the initial text and information of each character used to deduce the plot; and The first large model is used to perform role separation on the deductive text to generate the subtitle information.

11. The method according to claim 10, in, The step of using the first model to perform plot analysis on the initial text to generate a deductive text includes: Inputting the initial text and parsing prompt information into the first large model, and outputting the deduced text; The parsing prompt information includes parsing text requirements and examples of parsing reference texts.

12. The method according to claim 10, in, The step of using the first large model to perform role splitting on the deductive text to generate the subtitle information includes: Input the deductive text and the splitting prompt information into the first large model, and output the subtitle information; The splitting prompt information includes examples of role splitting requirements and splitting reference texts.

13. A video generation device based on a large model, include: A parsing module, used to parse the initial text using a large model to generate subtitle information, wherein the subtitle information includes information for interpreting the plot for each character in the initial text; A synthesis module, used for synthesizing the audio information of each character according to the predetermined audio and the subtitle information; A storyboard module, used for performing storyboard processing on the subtitle information to generate storyboard picture information; and A generation module is used to generate a target video for explaining the initial text according to the development sequence of the plot in the initial text by combining the subtitle information, the audio information and the storyboard information.

14. The device according to claim 13, in, The split mirror module comprises: a splitting submodule, configured to split the subtitle information using the first large model to generate storyboard description text; and The first generating submodule is used to generate storyboard picture information based on the storyboard description text.

15. The apparatus according to claim 14, wherein the first generating submodule comprises: include: An optimization unit, configured to optimize the storyboard description text by using the first large model to generate storyboard screen prompt information; as well as The first generating unit is used to generate the storyboard picture information by calling the second large model based on the storyboard picture prompt information.

16. The device according to claim 15, in, The optimization unit comprises: An input subunit, used for inputting the storyboard description text and optimization prompt information into the first large model, and outputting the storyboard screen prompt information; The optimization prompt information includes reference examples for determining character attributes of each reference character based on the name of each reference character.

17. The device according to claim 15 or 16, wherein the first generating submodule further comprises: include: An acquisition unit, used for acquiring the picture style information of each character; as well as The second generating unit is used to generate the storyboard picture information of each storyboard by calling the second large model based on the storyboard picture prompt information and the picture style information.

18. The device according to claim 14, wherein the split mirror module include: An input submodule, used for inputting the subtitle information and storyboard prompt information into the first model, and outputting the storyboard description text; The storyboard prompt information includes reference examples of storyboard description reference texts obtained by storyboarding based on the interpretation texts of each reference character.

19. The device according to claim 18, wherein the mirror module further comprises: include: An extraction submodule, used for extracting mapping information between each storyboard and each description text from the storyboard description text; The mapping information represents the corresponding relationship between the first identifier and the second identifier; the first identifier represents the order of each of the description texts in the subtitle information; the second identifier represents the order of each of the storyboards; as well as The adding submodule is used to add the subtitle information corresponding to the vacancy mark to the description text of the i-th storyboard based on the mapping relationship in response to the presence of a vacancy mark between the description text in the i-th storyboard and the description text for the i+1-th storyboard.

20. The device according to claim 13, in, The predetermined audio includes at least two candidate reference audios; The synthesis module comprises: A parsing submodule, used for parsing the subtitle information using the first large model to generate style information of each character; a matching submodule, configured to match the target reference audio of each character from the candidate reference audios according to the style information of each character; and The synthesis submodule is used to synthesize the audio information of each character according to the target reference audio of each character and the subtitle information.

21. The device according to claim 19, in, The parsing submodule includes: an extraction unit, configured to extract information of each of the characters from the subtitle information; and A first input unit, used for inputting the information and style prompt information of each character into the first large model, and outputting the style information of each character; The style prompt information includes reference examples for determining the character style of each reference character based on information of each reference character.

22. The device according to claim 13, in, The parsing module comprises: a plot analysis submodule, configured to use the first large model to perform plot analysis on the initial text to generate a deductive text; wherein the deductive text includes the plot of the initial text and information of each character used to deduce the plot; and The role splitting submodule is used to use the first large model to perform role splitting on the deductive text to generate the subtitle information.

23. The device according to claim 21, in, The plot analysis submodule includes: A second input unit, used for inputting the initial text and parsing prompt information into the first large model, and outputting the deduced text; The parsing prompt information includes parsing text requirements and examples of parsing reference texts.

24. The device according to claim 21, in, The role splitting submodule includes: A third input unit, used for inputting the deductive text and the splitting prompt information into the first large model, and outputting the subtitle information; The splitting prompt information includes examples of role splitting requirements and splitting reference texts.

25. An electronic device, include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

26. A non-transitory computer-readable storage medium storing computer instructions, in, The computer instructions are used to cause the computer to execute the method according to any one of claims 1-12.

27. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images