Video generation method and device, video processing method and device, equipment and storage medium

Abstract

The invention provides a video generation method, and relates to the technical field of artificial intelligence, in particular to the technical field of large models, video processing and virtual digital humans. According to the specific implementation scheme, multiple initial prompt texts are determined according to an initial text input by a user, and the multiple initial prompt texts comprise an initial content prompt text and an initial material prompt text; determining a video content text and at least one piece of initial object action driving data corresponding to the video content text according to the initial content prompt text; and generating an initial video according to the at least one piece of initial object action driving data and at least one piece of initial material corresponding to the at least one piece of initial material prompt text. The invention further provides a video processing method and device, electronic equipment and a storage medium.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the technical fields of large models, video processing and virtual digital humans, and can be applied to a variety of video production scenarios such as social media videos, marketing videos, education and training videos, news reporting videos, entertainment and leisure videos, e-commerce videos and stylized animation videos. More specifically, the present disclosure provides a video generation method, a video processing method, a device, an electronic device and a storage medium. Background Art

[0002] With the development of artificial intelligence technology, the application scenarios of large models are constantly expanding. Based on artificial intelligence technology, videos can be generated based on the text entered by the user. Summary of the invention

[0003] The present disclosure provides a video generation method, a video processing method, an apparatus, a device, and a storage medium.

[0004] According to one aspect of the present disclosure, a video generation method is provided, the method comprising: determining a plurality of initial prompt texts according to an initial text input by a user, wherein the plurality of initial prompt texts include an initial content prompt text and an initial material prompt text; determining a video content text and at least one initial object action driving data corresponding to the video content text according to the initial content prompt text; and generating an initial video according to the at least one initial object action driving data and at least one initial material corresponding to the at least one initial material prompt text.

[0005] According to another aspect of the present disclosure, a video processing method is provided, the method comprising: determining at least one adjustment prompt text and at least one attribute adjustment information based on an adjustment text corresponding to a video to be processed, wherein the video to be processed corresponds to at least one material to be adjusted; adjusting the attribute information of at least one material to be adjusted corresponding to the at least one adjustment prompt text based on at least one attribute adjustment information corresponding to the at least one adjustment prompt text to obtain at least one adjusted material; and obtaining a processed video based on the at least one adjusted material.

[0006] According to another aspect of the present disclosure, a video generating device is provided, which includes: a first determining module, used to determine a plurality of initial prompt texts according to an initial text input by a user, wherein the plurality of initial prompt texts include an initial content prompt text and an initial material prompt text; a second determining module, used to determine a video content text and at least one initial object action driving data corresponding to the video content text according to the initial content prompt text; and a generating module, used to generate an initial video according to the at least one initial object action driving data and at least one initial material corresponding to the at least one initial material prompt text.

[0007] According to another aspect of the present disclosure, a video processing device is provided, which includes: a third determination module, used to determine at least one adjustment prompt text and at least one attribute adjustment information based on the adjustment text corresponding to the video to be processed, wherein the video to be processed corresponds to at least one material to be adjusted; an adjustment module, used to adjust the attribute information of at least one material to be adjusted corresponding to the at least one adjustment prompt text based on the at least one attribute adjustment information corresponding to the at least one adjustment prompt text, so as to obtain at least one adjusted material; and an acquisition module, used to obtain a processed video based on the at least one adjusted material.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided according to the present disclosure.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided. The computer instructions are used to cause a computer to execute the method provided according to the present disclosure.

[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method provided according to the present disclosure is implemented.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1 is a schematic diagram of an exemplary system architecture to which a video generation method, a video processing method, and an apparatus can be applied according to an embodiment of the present disclosure;

[0014] Figure 2 is a flowchart of a video generation method according to an embodiment of the present disclosure;

[0015] Figure 3 is a schematic diagram of script data according to one embodiment of the present disclosure;

[0016] Figure 4 is a schematic diagram of multiple tasks corresponding to multiple prompt texts according to an embodiment of the present disclosure;

[0017] Figure 5 is a schematic diagram of a video generation method according to an embodiment of the present disclosure;

[0018] Fig. 6A is a schematic diagram of a visual interface according to one embodiment of the present disclosure;

[0019] Figure 6B and Figure 6C is a schematic diagram of different video frames of an initial video according to an embodiment of the present disclosure;

[0020] Figure 7 is a flowchart of a video processing method according to another embodiment of the present disclosure;

[0021] Figure 8 is a block diagram of a video generating apparatus according to an embodiment of the present disclosure;

[0022] Fig. 9 is a block diagram of a video processing device according to another embodiment of the present disclosure; and

[0023] Fig.10 is a block diagram of an electronic device to which a video generating method and / or a video processing method can be applied according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0025] In order to design and produce an exquisite video that meets the requirements, one or more designers, directors and actors with excellent aesthetics and rich experience need to spend a lot of time to complete it, which leads to high labor costs in the video production process, especially the labor costs of professionals, and also leads to high time costs required for video production.

[0026] In some embodiments, a conversational model may be used to generate a video based on natural language input by a user.

[0027] However, when using large models to generate videos, the controllability and editability of the videos are insufficient. When generating complex scenes, large models (such as Sora) may find it difficult to accurately simulate physical behaviors, resulting in the generated videos being inaccurate in details. For example, the interactions between objects are unnatural and do not follow the laws of physics. Even though some large models (such as RunwayGen3) can generate high-quality videos, they have difficulty handling complex character and object interactions, and the generated videos are difficult to meet user needs.

[0028] In addition, when using artificial intelligence technology to generate videos, the duration of the generated videos is relatively short. As the duration of the generated videos increases, the errors and unreasonableness in the videos also increase. For example, if artificial intelligence technology is used to generate a video of several minutes in length, it may take a long time to generate, and the quality of the generated video is also low, making it difficult to use artificial intelligence-based video generation technology for the generation of long videos.

[0029] In addition, in the process of generating videos using artificial intelligence technology, general data can be used for automated video generation, resulting in the generated videos lacking the user's personalized characteristics, having weak recognition, and being unable to meet the user's personalized needs.

[0030] In addition, AI-based video generation technology has low maturity and stability, and its performance in different scenarios varies greatly, especially in complex scenarios where it is difficult to generate continuous, high-quality videos. AI-based video generation technology has brought many conveniences and possibilities to video creation, but the controllability, editability, compliance, and personalization levels still need to be improved, and the hardware resource overhead is large and still needs to be further optimized.

[0031] Therefore, in order to efficiently generate videos, the present disclosure provides a video generation method and a video processing method, and the system architecture applying the method will be described below.

[0032] Figure 1 is a schematic diagram of an exemplary system architecture to which a video generation method, a video processing method and an apparatus can be applied according to an embodiment of the present disclosure. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0033] like Figure 1As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0034] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptops, desktop computers, etc.

[0035] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0036] It should be noted that the video generation method and the video processing method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the video generation device and the video processing device provided in the embodiment of the present disclosure can generally be set in the server 105. The video generation method and the video processing method provided in the embodiment of the present disclosure can also be executed by a server or a server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the video generation device and the video processing device provided in the embodiment of the present disclosure can also be set in a server or a server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0037] It can be understood that the above describes the system architecture of the present disclosure, and the following will describe the method of the present disclosure.

[0038] Figure 2 is a flowchart of a video generation method according to an embodiment of the present disclosure.

[0039] like Figure 2 As shown, the method 200 may include operations S210 to S230.

[0040] In operation S210, a plurality of initial prompt texts are determined according to an initial text input by a user.

[0041] In the disclosed embodiment, the initial prompt text may be determined in various ways. For example, the initial text may be segmented. Based on the segmentation result, the initial prompt text is determined. If the initial text is "a host is introducing news", the initial prompt text may be "host" and "news".

[0042] In the disclosed embodiment, the multiple initial prompt texts include initial content prompt texts and initial material prompt texts. For example, the initial prompt text "news" can be the initial content prompt text. The initial prompt text "host" can be the initial material prompt text.

[0043] In operation S220, a video content text and at least one initial object motion driving data corresponding to the video content text are determined according to the initial content prompt text.

[0044] In the disclosed embodiment, the initial object action driving data includes driving data corresponding to the lip shape of the object. For example, based on the initial prompt text "news", one or more news of a certain day can be obtained as the video content text. Based on the text-to-speech (TTS) technology, one or more lip shape data corresponding to the video content text can be obtained. Based on the one or more lip shape data, one or more initial object action driving data can be determined.

[0045] In operation S230, an initial video is generated according to at least one initial object motion driving data and at least one initial material corresponding to at least one initial material prompt text.

[0046] For example, the initial material corresponding to the initial prompt text "host" may be a virtual image. The virtual image includes a head and lips. The lips of the virtual image may be driven to perform one or more lip movements using the initial object action driving data to achieve the above one or more lip shapes, thereby obtaining one or more video frames. Based on these video frames, an initial video may be obtained.

[0047] Through the disclosed embodiments, the initial prompt text is used to realize video generation, reducing the manpower cost and time cost required for video generation. The initial content prompt text is used to generate the video content text, and the action driving data is determined, so that the video text content and the action displayed by the object in the video can be more consistent and coordinated, thereby improving the quality of the video.

[0048] In addition, through the embodiments of the present disclosure, when the scene in the video is a three-dimensional scene, the production cost can be greatly reduced. When the materials required for video generation are prepared, a video with a three-dimensional scene can be quickly generated through text description. For low- and medium-requirement projects that require short-term delivery, such as virtual human live broadcasts and promotional videos, the production delivery cycle can be greatly shortened to within a few days. In addition, the energy of three-dimensional scene designers can be focused on material optimization and overall scene concept design, minimizing the repetitive work of designers and improving the overall quality of the video and the efficiency of video generation.

[0049] It can be understood that the above describes the method of the present disclosure, and the following will describe the initial prompt text of the present disclosure.

[0050] In some embodiments, in some implementations of the above operation S210, determining multiple initial prompt texts according to the initial text input by the user includes: determining initial script data according to the initial text and attribute information of the user. Determining multiple initial prompt texts according to the initial script data.

[0051] In the disclosed embodiments, the user's attribute information includes information such as the industry in which the user is located and the actual application scenario. For example, when a user first uses an artificial intelligence product, he or she may enter the text "Help me generate a video using this digital human" in the input box of the visual interface of the product. Next, after the user authorizes, the user's authorized attribute information may be obtained.

[0052] In the embodiment of the present disclosure, according to the initial text and the attribute information of the user, a large model can be used to determine multiple initial prompt texts. The large model can be obtained by fine-tuning multiple sample texts and multiple preset prompt texts. The initial prompt text is determined from multiple preset prompt texts using the large model. The large model can be a large language model (LLM). The preset prompt text can be a standardized prompt text (prompt). The sample text can be a historical text input into the large model by a user, or a historical text input into the large model by multiple users with similar attributes, or a text generated from the historical text input by the user with a high similarity, and the present disclosure does not limit this. The large model can be a conversational large model such as Wenxin Yiyan. Through the embodiment of the present disclosure, using the conversational large model, a short natural language text prompt word can be used to quickly generate a video based on the produced material in the material production platform. The large model is fine-tuned using the preset prompt text, and the preset prompt text can correspond to the identification text of the material, so that the fine-tuned large model can quickly determine the material corresponding to the initial prompt text or the adjustment prompt text from multiple materials.

[0053] In the embodiment of the present disclosure, the initial script data includes at least one of the initial script outline data, the initial script storyboard data and the initial script reference picture data. Figure 3 Provide explanation.

[0054] Figure 3 is a schematic diagram of script data according to an embodiment of the present disclosure.

[0055] like Figure 3 As shown, user user30 can input the natural language text "A host is introducing the news" into the large model llm30. The large model llm30 can disassemble the natural language text to obtain multiple disassembly results. The multiple disassembly results may include "host", "news", etc. According to the multiple disassembly results, the script structure is determined from the script structure library stru30 using the script framework fw30 corresponding to the user's attribute information. The script structure may, for example, include at least one of an outline, a storyboard, and a reference screen. Accordingly, the script data determined using the script structure may include at least one of the script outline data sc30, the script storyboard description data b30, and the script reference screen data p30.

[0056] The script outline data sc30 is equivalent to the script outline provided by the screenwriter. The script outline data sc30 may include a script outline text "a host is introducing the news".

[0057] The script storyboard description data b30 may correspond to the script storyboard description text "the lens advances from the panoramic view, switches to the close-up view, and then cuts to the medium view to fix the shot". The script storyboard description text may correspond to the camera lens movement method commonly used in news scenes. The script storyboard description data may also correspond to at least one storyboard data. Figure 3 As shown, at least one storyboard data may include storyboard data bd31, storyboard data bd32, storyboard data bd33 and storyboard data bd34, etc. Storyboard data bd31 may include at least one of scene description data scene31, lens indication data lens31, action data action31, audio data audio31, lighting data light31 and duration data dur31. Scene description data scene31 may indicate scene type, peripheral layout, etc. Lens indication data lens31 may indicate lens type and lens movement mode. Action data action31 may indicate head movement and body movement of an object. Audio data audio31 may indicate sound effects and background music. Lighting data light31 may indicate character light and scene light. Duration data dur31 may indicate the duration of a storyboard.

[0058] The script reference picture data p30 includes video size data, focal length data, depth of field data and action rhythm data. The action rhythm data 31 may indicate the speed at which the subject performs the above-mentioned body movements. The script reference picture data p30 may correspond to the script reference picture description text "16:9 composition, overall slow rhythm, etc."

[0059] It can be understood that if the above-mentioned natural language text is used as the initial text, the above-mentioned script data can be used as the initial script data. The script outline data sc30, the script storyboard description data b30 and the script reference picture data p31 can be used as the initial script outline data, the initial script storyboard description data and the initial script reference picture data respectively. The initial script storyboard description data corresponds to at least one initial storyboard data, and the initial storyboard data includes at least one of initial scene description data, initial lens indication data, initial action data, initial audio data, initial lighting data and initial duration data. The initial script reference picture data includes at least one of initial video size data, initial focal length data and initial depth of field data.

[0060] Through the embodiments of the present disclosure, a short natural language text can be used to generate a highly professional script (script outline, reference pictures, storyboards, etc.) that matches the user's attributes, so as to quickly generate a video based on the script.

[0061] It can be understood that the above is an explanation of the initial script data of the present disclosure, and the following will be combined with Figure 4 Description of multiple initial prompt text lines.

[0062] Figure 4 is a schematic diagram of multiple tasks corresponding to multiple prompt texts according to an embodiment of the present disclosure.

[0063] like Figure 4 As shown, user 40 may provide a natural language text 40. The natural language text 40 may be "a host is introducing the news".

[0064] In the disclosed embodiment, multiple initial prompt texts can be determined based on the initial script data. For example, based on the above-mentioned script outline text, script storyboard description text and script reference screen description text, the script text st40 can be obtained by processing using a large model. The script text st40 can be "A female host in a blue business suit stands in a science fiction-style studio, introducing the international current affairs news broadcast that happened that day. The camera advances from the panoramic view, switches to the close-up advance and then cuts to the medium shot. The total length is 30 seconds, the composition is 16:9, the overall rhythm is smooth, etc." It can be understood that on the basis of the script outline text, descriptive texts related to scenes, objects, etc. are added to obtain the above-mentioned script text. Based on the script text, multiple prompt texts can be determined. Multiple prompt texts can correspond to multiple tasks. Multiple tasks may include content generation task m401, action generation task m402, object determination task m403, scene determination task m404, lens determination task m405, lighting determination task m406, and synthesis output task m407.

[0065] In the disclosed embodiment, the plurality of prompt texts may include a content prompt text. The content prompt text may be "international current affairs news occurring on the day". The content prompt text may be used as the initial content prompt text.

[0066] In the disclosed embodiment, a large model can be used to generate an initial content text based on the initial content prompt text. For example, in order to perform the content generation task m401, the large model can search for international current affairs news according to the date and generate a content text tn40. The content text tn40 can be "This afternoon, the returner of the lunar exploration project held an unpacking event, which brought the implementation phase of the lunar exploration mission project to a successful conclusion. According to a report by Country A's Broadcasting Company yesterday, the Director of a country's Health Bureau announced that there are currently no major public health incidents. The President of Country B signed a decree approving Unit C's decision on the use of drones, and the decree has been published on the website of the Presidential Palace of Country B." This content text can be used as the initial content text.

[0067] For another example, after executing the action generation task m402, the object's limb action driving data and head action driving data can be obtained. The limb action driving data can correspond to the object's overall action, the action above the waist, the action of the feet, and the action of the hands. After executing the object determination task m403, for example, a digital person can be determined as the object. The limbs and head of the object can be driven by the above-mentioned limb action driving data and head action driving data respectively to perform related actions. After executing the scene determination task m404, the scene in which the object is located can be determined. In order to execute the lens determination task m405, the lens description information can be determined based on the position of the object. The lens description information is related to the movement mode of the lens. The position of the object includes the correspondence of the object's head, eyes, mouth, waist, left hand, right hand, left foot, right foot, left leg and right leg.

[0068] The lighting determination task m406 may include one or more of the fill light task m4061, the lighting style determination task m4062, the color temperature determination task m4063, the hue determination task m4064, and the like. The execution result of the fill light task m4061 may correspond to fill light methods such as front light, back light, side light, side front light, side back light, top light, and edge light. The execution result of the lighting style determination task m4062 may correspond to multiple lighting styles. Multiple lighting styles may include flat light style, horror style, 1920s style, cyber style, and the like. The execution result of the color temperature determination task m4063 is related to warm white temperature, pure white temperature, and cold white temperature. The execution result of the hue determination task m4064 corresponds to hues such as red and blue.

[0069] The video output task m407 may include one or more of the following tasks: an audio synthesis task m4071, a color adjustment task m4072, a subtitle synthesis task m4073, a size configuration task m4074, a rendering output task m4075, etc. Based on the audio synthesis task m4071, audios such as background sounds, environmental sounds, and sound effects may be synthesized into an audio file. Based on the color adjustment task m4072, the overall tone of the video may be adjusted. Based on the subtitle synthesis task m4073, one or more of subtitle encapsulation, font setting, and subtitle special effect encapsulation processing may be performed. The size configuration task m4074 corresponds to multiple video sizes. Multiple video sizes include 16:9, 4:3, etc. The rendering output task m4075 corresponds to one or more video frames, and also corresponds to the format of the video. The file format of the video frame may be the Joint Photographic Experts Group (JPEG) format, the Portable Network Image (PNG) format, etc.

[0070] It can be understood that the above describes multiple initial prompt texts, and the method of generating a video will be further described below.

[0071] Figure 5 is a schematic diagram of a video generation method according to an embodiment of the present disclosure.

[0072] like Figure 5 As shown, the user can input natural language text as the initial text text50. Based on the initial text text50, the initial script data can be determined. The initial script data can include an initial script text st50. According to the initial script text st50, multiple initial prompt texts can be determined. The multiple initial prompt texts can include an initial content prompt text. It can be understood that the above description of the natural language text text40 and the script text st40 is also applicable to the initial text text50 and the initial script text st50. Some methods for determining object action driving data will be described below.

[0073] In an embodiment of the present disclosure, in some implementations of the above-mentioned operation S220, determining the video content text and at least one initial object action driving data corresponding to the video content text according to the initial content prompt text includes: generating the initial content text according to the initial content prompt text. Determining the initial audio content data, time information and at least one initial object action driving data corresponding to the initial content text. For example, the above-mentioned text-to-speech technology can be used to determine the initial audio content data audio501 corresponding to the initial content text tn50. The initial audio content data audio501 can correspond to a time information. The time information is related to the duration of the initial audio content. Some methods for determining the object action driving data will be described below. It can be understood that the above description of the initial content text tn40 is also applicable to the initial content text tn50, and the present disclosure will not repeat them here.

[0074] In some embodiments, determining the initial audio content data, time information, and at least one initial object motion drive data corresponding to the initial content text includes: determining the initial head motion drive data according to the initial audio content data. For example, the initial head motion drive data hdrive50 may correspond to the lip motion of the initial audio content data.

[0075] In some embodiments, determining initial audio content data, time information, and at least one initial object motion driving data corresponding to the initial content text includes: determining initial body motion driving data according to the initial content text.

[0076] In the disclosed embodiment, according to the initial content text, determining the initial body motion driving data includes: determining at least one initial content subtext of the initial content text according to the text structure of the initial content text. Determining at least one first initial body motion driving subdata corresponding to at least one initial content subtext. For example, the text structure may indicate the segmentation information of the text. At least one content subtext of the content text tn50 may include "This afternoon, the returner of the lunar exploration project held an opening event, which brought a successful conclusion to the implementation phase of the lunar exploration mission project.", "According to a report by the broadcasting company of Country A yesterday, the director of the health bureau of a certain country announced that there are currently no major public health events." and "The president of Country B signed a decree approving the decision of Unit C on the use of drones, which has been published on the website of the Presidential Palace of Country B". Different content subtexts may correspond to different actions. Thus, at least one action material may be determined from a plurality of action materials as at least one first action material corresponding to the content subtext. The driving data corresponding to the first action material may be used as the first initial body motion driving subdata body510.

[0077] In the disclosed embodiment, determining the initial body movement driving data according to the initial content text includes: determining at least one second initial body movement driving sub-data corresponding to the initial content text according to the time information. For example, according to the time information of the above-mentioned initial audio content data audio501, at least one second action material corresponding to the entire initial content text can be determined from multiple action materials. The second action material can increase the richness of the object's action. The driving data corresponding to the at least one second action material can be used as the second initial body movement driving sub-data body520.

[0078] In the embodiment of the present disclosure, determining the initial body movement driving data according to the initial content text includes: determining at least one initial body movement driving data according to at least one first initial body movement driving sub-data and at least one second initial body movement driving sub-data. Figure 5 As shown, at least one first initial limb motion driving sub-data body510 and at least one second initial limb motion driving sub-data body520 are merged to determine at least one initial limb motion driving data bdrive50.

[0079] In some embodiments, determining the initial audio content data, time information, and at least one initial object motion driving data corresponding to the initial content text includes: determining at least one initial object motion driving data according to the initial head motion driving data and the initial limb motion driving data. Figure 5 As shown, initial object motion drive data can be determined based on initial head motion drive data hdrive50 and at least one initial limb motion drive data bdrive50.

[0080] It can be understood that the above describes some methods for determining initial object motion driving data, and the following describes some methods for generating videos.

[0081] In some embodiments, the at least one initial material prompt text includes an initial style material prompt text. In some implementations of the above operation S220, generating the initial video according to the at least one initial object motion driving data and the at least one initial material corresponding to the at least one initial material prompt text includes: determining the initial object material according to the initial lighting material corresponding to the initial style material prompt text and the at least one initial object motion driving data. Figure 5 As shown, the multiple initial prompt texts determined according to the initial script text st50 may include an initial style material prompt text. Based on the initial style material prompt text, the initial lighting material sm501 may be determined. According to the initial lighting material sm501, the virtual image corresponding to the initial prompt text "host" and at least one initial object action driving data, the initial object material sm502 may be determined. According to the initial object material, an initial video may be generated.

[0082] In the disclosed embodiment, the multiple initial prompt texts also include at least one of the initial scene material prompt text and the initial shot description prompt text. For example, the initial scene material prompt text may be "studio". The multiple scene materials may include indoor scene materials and outdoor scene materials. The outdoor scene materials may correspond to scenes such as mountains, grasslands, city rooftops, and tree houses in the woods, and may also correspond to different weather materials. Different weather materials include sunny outdoor scene materials, rainy outdoor scene materials, and snowy outdoor scene materials. Indoor scene materials may include office materials, studio materials, and classroom materials, etc. The initial scene material sm503 corresponding to the initial scene material prompt text "studio" may be the above-mentioned studio material. The initial shot description prompt text may correspond to the shot description information shot50. The shot description information may indicate the motion type, focal length, and angle of the shot, etc.

[0083] In the embodiment of the present disclosure, generating an initial video according to the initial object material includes: obtaining a plurality of first initial video frames according to the initial object material, the initial illumination material, and the initial scene material corresponding to the initial scene material prompt text. For example, obtaining a plurality of first initial video frames according to the initial object material, the initial illumination material, the initial scene material, and the initial shot description information corresponding to the initial shot description prompt text. Figure 5As shown, the initial object material sm502 can correspond to multiple different actions. According to the initial object material sm502, the initial scene material sm503, and the shot description information shot50, multiple synthesis results r50 corresponding to multiple actions are determined. According to the multiple synthesis results r50 and the initial lighting material sm501, multiple first initial video frames f51 can be obtained. According to the multiple first initial video frames f51, an initial video can be generated. It can be understood that the initial video can be directly determined based on the multiple first initial video frames. It is also possible to select one or more from the multiple first initial video frames to generate an initial video, which will be explained below.

[0084] In the embodiment of the present disclosure, generating the initial video according to the plurality of first initial video frames includes: determining the initial video editing information according to at least one of the initial audio content data corresponding to the initial content text, the initial background sound effect material corresponding to the initial style material prompt text, the initial editing style material corresponding to the initial style material prompt text, and the initial transition style material corresponding to the initial style material prompt text. Figure 5 As shown, according to the initial style material prompt text, the initial background sound effect material sm504, the initial editing style material sm505 and the initial transition style material sm506 can be determined, and the initial video clip information clip50 can be determined. The initial video clip information can include one or more clip conditions. The clip condition can be that the actions of different frames are the same. One of the two video frames that meet the clip condition can be deleted. It can be understood that the clip condition can also be other conditions, and the present disclosure does not limit this.

[0085] In the embodiment of the present disclosure, generating an initial video according to multiple first initial video frames includes: obtaining at least one second initial video frame according to the initial video clip information and the multiple first initial video frames. For example, at least one second video frame f52 can be obtained according to the initial video clip information clip50 and the multiple first initial video frames f51. Next, generating an initial video according to the at least one second initial video frame.

[0086] In the embodiment of the present disclosure, generating the initial video according to at least one second initial video frame includes: packaging at least one second initial video frame according to the initial video packaging material corresponding to the initial style material prompt text to generate the initial video. Figure 5 As shown, the initial style material prompt text may correspond to the initial video packaging material sm507. The initial video packaging material sm507 may indicate the format of video packaging. According to the initial video packaging material package50, at least one second initial video frame f52 is packaged to obtain an initial video v50.

[0087] It can be understood that the above describes the method of the present invention, and the following will describe the post-processing method of the initial video of the present invention.

[0088] In some embodiments, the sequence tags of multiple video frames in the initial video may be determined to adjust the order of the video frames. For example, a content subtext may correspond to multiple video frames. The order of video content corresponding to different content subtexts in the video may be changed according to the order of the video frames.

[0089] In some embodiments, one or more video frames in the initial video that meet the deletion condition may be deleted. The deletion condition may include at least one of the presence of motion errors, repeated motion, and stillness.

[0090] In some embodiments, the playback rate of the initial video may be adjusted.

[0091] In some embodiments, an external application programming interface (API) may be called to render a tile image. The tile image may be an image such as an icon displayed in the video. An external audio generation application programming interface may also be called to adjust the audio or sound effects of the initial video. If there are subtitles in the initial video, the font format of the subtitles may also be adjusted.

[0092] It can be understood that the above describes the post-processing method of the video, and the following will describe the initial video.

[0093] In some embodiments, generating the initial video includes: displaying the initial video on a visual interface. FIG. 6A to FIG. 6C Provide explanation.

[0094] Fig. 6A is a schematic diagram of a visual interface according to an embodiment of the present disclosure.

[0095] like Fig. 6A As shown, the visual interface i60 may include an input box ib60. The user uses the input box ib60 to input an initial text "a host is introducing news". Next, an initial video may be generated based on the initial text and displayed on the visual interface i60.

[0096] Figure 6B and Figure 6C is a schematic diagram of different video frames of an initial video according to an embodiment of the present disclosure.

[0097] like Figure 6B As shown, at the first moment, the visual interface i60 displays the video frame f621 of the initial video. Figure 6C As shown, at the second moment, the visual interface i60 displays a video frame f622 of the initial video. The actions of the objects in the video frames f621 and f622 are different.

[0098] It can be understood that the video generation method of the present invention is described above, and the application scenario of the method will be further described below.

[0099] In some embodiments, the above method can be used to create videos for social media. For example, using the above method and the conversational model, users can quickly obtain interesting and attractive videos to display on social media platforms and conveniently share them in daily life.

[0100] In some embodiments, the above method can be used to produce corporate advertising and marketing videos. For example, using the above method and the conversational big model, relevant personnel of the company can easily produce promotional videos to showcase the characteristics of the company's products and brand concepts, thereby improving advertising effects and attracting more potential customers.

[0101] In some embodiments, the above method can be used to produce educational training videos. For example, using the above method and the conversational model, teachers can transform teaching content into vivid videos to improve students' learning mood and enable students to learn knowledge more intuitively.

[0102] In some embodiments, the above method can be used to produce news report videos. For example, using the above method and the conversational big model, reporters, directors, etc. can quickly produce news report videos to increase the speed and influence of news dissemination, allowing viewers to understand news events more intuitively.

[0103] In some embodiments, the above method can be used to make entertainment videos. For example, using the above method and the conversational model, in entertainment and leisure occasions, users can make home movies, travel records, etc., and record their lives in video form.

[0104] In some embodiments, the above method can be used to produce e-commerce operation videos. For example, in the e-commerce industry, using the above method and the conversational big model, anchors, merchants, etc. can produce product display videos to attract more customers to buy goods and improve efficiency.

[0105] In some embodiments, the above method can be used to create stylized animation videos. For example, using the above method and the conversational large model, animation creators can quickly create animation clips, greatly improve animation production efficiency, and bring rich visual experience to the audience.

[0106] Through the disclosed embodiments, in the video production process, starting from the user's perspective, in accordance with the general video commercial production process, in each key link, by relying on the big model to input text prompt words to assist the user to quickly obtain the desired effect in each link. In the disclosed embodiments, the big model can be Wenxinyiyan, which can support the user's full Chinese input and more accurately understand the Chinese semantics.

[0107] It can be understood that the above describes the application scenario of the video generation method of the present disclosure, and the following will describe the video processing method of the present disclosure.

[0108] Figure 7 is a flowchart of a video processing method according to another embodiment of the present disclosure.

[0109] like Figure 7 As shown, the method 700 may include operations S740 to S760.

[0110] In operation S740, at least one adjustment prompt text and at least one property adjustment information are determined according to the adjustment text corresponding to the video to be processed.

[0111] In the embodiment of the present disclosure, the video to be processed may be a generated video. For example, the video to be processed may be the initial video v50 mentioned above.

[0112] In the disclosed embodiment, the video to be processed corresponds to at least one material to be adjusted. For example, the video to be processed may be generated using at least one material. Any material in the image to be processed may be used as a material to be adjusted. The above-mentioned initial scene material may be used as a material to be adjusted. The attributes of the material may include size, position, color, content, etc.

[0113] In the disclosed embodiment, the adjustment text may be input by the user for the generated video. For example, the adjustment text may be “please help me change the scene to a blue one”.

[0114] In the disclosed embodiments, the adjustment prompt text and the attribute adjustment information can be determined in various ways. For example, the adjustment text can be segmented. Based on the result of the segmentation, the adjustment prompt text and the attribute adjustment information are determined. Taking the example of taking the first noun as the adjustment prompt text and determining the attribute adjustment information with the verb and the related words, "scene" can be taken as an adjustment prompt text, and the attribute adjustment information can be determined based on "changing to blue".

[0115] In operation S750, attribute information of at least one to-be-adjusted material corresponding to at least one adjustment prompt text is adjusted according to at least one attribute adjustment information corresponding to at least one adjustment prompt text to obtain at least one adjusted material.

[0116] In the disclosed embodiment, the adjustment prompt text may correspond to one or more attribute adjustment information. The adjustment prompt text may also correspond to one or more materials to be adjusted. For example, the adjustment prompt text "scene" may correspond to the scene material and also to the attribute adjustment information determined based on "change to blue color". The color of the scene material may be adjusted to blue to obtain the adjusted scene material.

[0117] In operation S760, a processed video is obtained according to the at least one adjusted material.

[0118] In the disclosed embodiment, a video can be generated based on the adjusted material and the unadjusted material to obtain a processed video. Alternatively, the adjusted material can be used to replace the material corresponding to the adjustment prompt text to obtain a processed video.

[0119] Through the disclosed embodiments, the adjustment prompt text and attribute adjustment information are used to adjust the material, thereby realizing the specified modification of the video and reducing the manpower cost and time cost required for video generation. In addition, the video can be modified through the adjustment text input by the user, which reduces the threshold for specified modification. Users who do not have professional artistic ability can also adjust beautiful videos, which can effectively improve the user experience.

[0120] In some embodiments, the video to be processed is obtained based on the initial video, and the attribute information of at least one material to be adjusted corresponding to at least one adjustment prompt text is adjusted: in response to hitting the material to be adjusted corresponding to the adjustment prompt text from at least one material to be adjusted, the identification text of the material to be adjusted corresponding to the adjustment prompt text is displayed in the visual interface.

[0121] In some embodiments, determining at least one adjustment prompt text and at least one attribute adjustment information according to the adjustment text corresponding to the video to be processed includes: determining at least one adjustment prompt text and at least one attribute adjustment information according to the adjustment text using a large model. For example, the adjustment prompt text "scene" and the attribute adjustment information "change to blue" can be determined using the large model llm30. Through the embodiments of the present disclosure, using a conversational large model, the style and tone of a video can be changed quickly and in real time according to the input text prompt words.

[0122] In some embodiments, the video to be processed is obtained based on the initial video, and the attribute information of at least one material to be adjusted corresponding to at least one adjustment prompt text is adjusted: in response to hitting the material to be adjusted corresponding to the adjustment prompt text from at least one material to be adjusted, the identification text of the material to be adjusted corresponding to the adjustment prompt text is displayed on the visual interface. For example, the material to be processed used when generating the initial video v50 can be used as the material to be adjusted. From multiple materials to be adjusted, the material to be adjusted corresponding to the adjustment prompt text "scene" can be determined. The material to be adjusted can be the above-mentioned initial scene material. The text "Successfully positioned the scene" can be displayed on the visual interface to display the identification text "scene" of the material to be adjusted. Through the embodiment of the present disclosure, the identification text of the material corresponding to the adjustment prompt text is displayed on the visual interface, so that the user can obtain the standard and normative text describing the material in the video, which helps to improve the efficiency of video adjustment and further improve the user experience.

[0123] In the disclosed embodiment, adjusting the attribute information of at least one to-be-adjusted material corresponding to at least one adjustment prompt text according to at least one attribute adjustment information corresponding to at least one adjustment prompt text includes: adjusting the attribute information of the to-be-adjusted material corresponding to the prompt text according to the attribute adjustment information corresponding to the adjustment prompt text. For example, according to the attribute adjustment information "Change to blue series" corresponding to the adjustment prompt text "scene", the scene material can be adjusted to the blue series, and a blue scene can be obtained as the adjusted material.

[0124] In some embodiments, obtaining a processed video according to at least one adjusted material may include: displaying the processed video on a visual interface. For example, the processed video may be displayed on a visual interface.

[0125] It can be understood that the method of the present disclosure is described above, and the device of the present disclosure will be described below.

[0126] Figure 8 is a block diagram of a video generating apparatus according to an embodiment of the present disclosure.

[0127] like Figure 8 As shown, the device 800 may include a first determination module 810 , a second determination module 820 and a generation module 830 .

[0128] The first determination module 810 is used to determine a plurality of initial prompt texts according to the initial text input by the user. The plurality of initial prompt texts include an initial content prompt text and an initial material prompt text.

[0129] The second determination module 820 is used to determine the video content text and at least one initial object action driving data corresponding to the video content text according to the initial content prompt text.

[0130] The generating module 830 is used to generate an initial video according to at least one initial object motion driving data and at least one initial material corresponding to at least one initial material prompt text.

[0131] In some embodiments, the first determination module includes: a first determination submodule, configured to determine initial script data according to the initial text and user attribute information; and a second determination submodule, configured to determine a plurality of initial prompt texts according to the initial script data.

[0132] In some embodiments, the initial script data includes at least one of initial script outline data, initial script storyboard description data, and initial script reference picture data. The initial script storyboard description data corresponds to at least one initial storyboard data, and the initial storyboard data includes at least one of initial scene description data, initial lens indication data, initial action data, initial audio data, initial lighting data, and initial duration data. The initial script reference picture data includes at least one of initial video size data, initial focal length data, and initial depth of field data.

[0133] In some embodiments, the second determination module includes: a first generation submodule for generating an initial content text according to the initial content prompt text; and a third determination submodule for determining initial audio content data, time information, and at least one initial object action driving data corresponding to the initial content text.

[0134] In some embodiments, the third determination submodule includes: a first determination unit, configured to determine initial head motion driving data according to initial audio content data; a second determination unit, configured to determine initial body motion driving data according to initial content text; and a third determination unit, configured to determine at least one initial object motion driving data according to the initial head motion driving data and the initial body motion driving data.

[0135] In some embodiments, the second determination unit includes: a first determination subunit, for determining at least one initial content subtext of the initial content text according to the text structure of the initial content text. A second determination subunit, for determining at least one first initial body motion driving subdata corresponding to the at least one initial content subtext. A third determination subunit, for determining at least one second initial body motion driving subdata corresponding to the initial content text according to time information. A fourth determination subunit, for determining at least one initial body motion driving data according to the at least one first initial body motion driving data and the at least one second initial body motion driving data.

[0136] In some embodiments, at least one initial material prompt text includes an initial style material prompt text. The generation module includes: a fourth determination submodule includes, for determining an initial object material according to an initial lighting material corresponding to the initial style material prompt text and at least one initial object motion driving data. A second generation submodule is used to generate an initial video according to the initial object material.

[0137] In some embodiments, the at least one initial material prompt text further includes an initial scene material prompt text. The second generation submodule includes: a first acquisition unit, configured to obtain a plurality of first initial video frames according to the initial object material, the initial illumination material, and the initial scene material corresponding to the initial scene material prompt text. A first generation unit, configured to generate an initial video according to the plurality of first initial video frames.

[0138] In some embodiments, the plurality of initial prompt texts further include initial shot description prompt texts. The first obtaining unit is further configured to obtain a plurality of first initial video frames according to initial object material, initial illumination material, initial scene material and initial shot description information corresponding to the initial shot description prompt texts.

[0139] In some embodiments, at least one initial material prompt text also includes an initial style material prompt text, and the first generating unit includes: a fifth determining subunit, which is used to determine the initial video editing information based on at least one of the initial audio content data corresponding to the initial content text, the initial background sound effect material corresponding to the initial style material prompt text, the initial editing style material corresponding to the initial style material prompt text, and the initial transition style material corresponding to the initial style material prompt text. A first obtaining subunit is used to obtain at least one second initial video frame based on the initial video editing information and multiple first initial video frames. A generating subunit is used to generate an initial video based on the at least one second initial video frame.

[0140] In some embodiments, the generating subunit is further configured to: encapsulate at least one second initial video frame according to the initial video encapsulation material corresponding to the initial style material prompt text to generate an initial video.

[0141] In some embodiments, the first determination module is further used to: determine multiple initial prompt texts using a large model based on the initial text and the user's attribute information. The large model is obtained by fine-tuning multiple sample texts and multiple preset prompt texts. The initial prompt text is determined from multiple preset prompt texts using the large model.

[0142] In some embodiments, the generation module includes: a first display submodule, which is used to display the initial video on the visual interface.

[0143] Fig. 9is a block diagram of a video processing device according to another embodiment of the present disclosure.

[0144] like Fig. 9 As shown, the device 900 may include a third determining module 940 , an adjusting module 950 , and an obtaining module 960 .

[0145] The third determination module 940 is used to determine at least one adjustment prompt text and at least one property adjustment information according to the adjustment text corresponding to the video to be processed. The video to be processed corresponds to at least one material to be adjusted.

[0146] The adjustment module 950 is used to adjust the attribute information of at least one to-be-adjusted material corresponding to at least one adjustment prompt text according to at least one attribute adjustment information corresponding to at least one adjustment prompt text, so as to obtain at least one adjusted material.

[0147] The obtaining module 960 is used to obtain a processed video according to at least one adjusted material.

[0148] In some embodiments, the video to be processed is obtained based on the initial video, and the adjustment module includes: a second display sub-module, which is used to display the identification text of the material to be adjusted corresponding to the adjustment prompt text in the visual interface in response to hitting the material to be adjusted corresponding to the adjustment prompt text from at least one material to be adjusted.

[0149] In some embodiments, the acquisition module includes: a third display submodule, used to display the processed video on a visual interface.

[0150] In some embodiments, the third determination module is further used to: determine at least one adjustment prompt text and at least one attribute adjustment information based on the adjustment text using a large model.

[0151] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0152] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0153] Fig.10A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0154] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 to a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0155] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0156] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as at least one of the video generation method and the video processing method. For example, in some embodiments, at least one of the video generation method and the video processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of at least one of the video generation method and the video processing method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform at least one of the video generation method and the video processing method in any other appropriate manner (e.g., by means of firmware).

[0157] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), system on chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0158] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0159] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories, read-only memories, erasable programmable read-only memories (EPROM) or flash memories, optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0160] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) display or a liquid crystal display (LCD)) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0161] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a Local Area Network (LAN), a Wide Area Network (WAN), and the Internet.

[0162] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0163] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0164] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A video generation method, comprising: Determine a plurality of initial prompt texts according to the initial text input by the user, wherein the plurality of initial prompt texts include an initial content prompt text and an initial material prompt text; Determining, according to the initial content prompt text, a video content text and at least one initial object action driving data corresponding to the video content text; An initial video is generated according to at least one of the initial object motion driving data and at least one initial material corresponding to at least one of the initial material prompt texts.

2. The method according to claim 1, wherein: Determining a plurality of initial prompt texts according to the initial text input by the user comprises: Determining initial script data according to the initial text and the attribute information of the user; According to the initial script data, a plurality of initial prompt texts are determined.

3. The method according to claim 2, wherein: The initial script data includes at least one of initial script outline data, initial script storyboard description data and initial script reference picture data. The initial script storyboard description data corresponds to at least one initial storyboard data, and the initial storyboard data includes at least one of initial scene description data, initial shot indication data, initial action data, initial audio data, initial lighting data and initial duration data; The initial script reference picture data includes at least one of initial video size data, initial focal length data, and initial depth of field data.

4. The method according to claim 1, wherein: The step of determining the video content text and at least one initial object action driving data corresponding to the video content text according to the initial content prompt text comprises: Generate the initial content text according to the initial content prompt text; Initial audio content data, time information and at least one initial object motion driving data corresponding to the initial content text are determined.

5. The method according to claim 4, wherein: The determining of the initial audio content data, time information and at least one initial object action driving data corresponding to the initial content text comprises: Determining initial head motion driving data according to the initial audio content data; Determining initial body movement driving data according to the initial content text; At least one of the initial object motion driving data is determined based on the initial head motion driving data and the initial limb motion driving data.

6. The method according to claim 5, wherein: Determining the initial body movement driving data according to the initial content text includes: Determining at least one initial content subtext of the initial content text according to the text structure of the initial content text; Determine at least one first initial body motion driving sub-data corresponding to at least one of the initial content sub-texts; Determine, according to the time information, at least one second initial body movement driving sub-data corresponding to the initial content text; At least one of the initial limb motion driving data is determined according to at least one of the first initial limb motion driving data and at least one of the second initial limb motion driving data.

7. The method according to claim 1, wherein: At least one of the initial material prompt texts includes an initial style material prompt text, The generating of the initial video according to at least one of the initial object action driving data and at least one initial material corresponding to at least one of the initial material prompt texts comprises: Determining an initial object material according to an initial lighting material corresponding to the initial style material prompt text and at least one of the initial object action driving data; The initial video is generated according to the initial object material.

8. The method according to claim 7, wherein: At least one of the initial material prompt texts also includes an initial scene material prompt text, The generating the initial video according to the initial object material comprises: Obtaining a plurality of first initial video frames according to the initial object material, the initial illumination material, and the initial scene material corresponding to the initial scene material prompt text; The initial video is generated according to a plurality of the first initial video frames.

9. The method according to claim 8, wherein: Multiple initial prompt texts also include initial shot description prompt text, The obtaining of a plurality of first initial video frames according to the initial object material, the initial illumination material and the initial scene material corresponding to the initial scene material prompt text comprises: A plurality of first initial video frames are obtained according to the initial object material, the initial lighting material, the initial scene material and the initial shot description information corresponding to the initial shot description prompt text.

10. The method according to claim 8, wherein: At least one of the initial material prompt texts also includes an initial style material prompt text, The generating the initial video according to the plurality of the first initial video frames comprises: Determine initial video editing information according to at least one of initial audio content data corresponding to the initial content text, initial background sound effect material corresponding to the initial style material prompt text, initial editing style material corresponding to the initial style material prompt text, and initial transition style material corresponding to the initial style material prompt text; Obtain at least one second initial video frame according to the initial video clipping information and the plurality of first initial video frames; The initial video is generated according to at least one of the second initial video frames.

11. The method according to claim 10, wherein: The generating the initial video according to at least one of the second initial video frames comprises: At least one of the second initial video frames is encapsulated according to the initial video encapsulation material corresponding to the initial style material prompt text to generate the initial video.

12. The method according to claim 1, wherein: Determining a plurality of initial prompt texts according to the initial text input by the user comprises: According to the initial text and the attribute information of the user, a plurality of the initial prompt texts are determined by using a large model, The large model is obtained by fine-tuning multiple sample texts and multiple preset prompt texts. The initial prompt text is determined from a plurality of preset prompt texts using the large model.

13. The method according to claim 1, wherein: Generating the initial video comprises: The initial video is displayed in a visual interface.

14. A video processing method, comprising: Determining at least one adjustment prompt text and at least one property adjustment information according to the adjustment text corresponding to the video to be processed, wherein the video to be processed corresponds to at least one material to be adjusted; According to at least one of the property adjustment information corresponding to at least one of the adjustment prompt texts, adjusting the property information of at least one of the to-be-adjusted materials corresponding to at least one of the adjustment prompt texts, to obtain at least one of the adjusted materials; A processed video is obtained according to at least one of the adjusted materials.

15. The method according to claim 14, wherein: The video to be processed is obtained according to the initial video, and the property information of at least one of the materials to be adjusted corresponding to at least one of the adjustment prompt texts is adjusted, including: In response to hitting the to-be-adjusted material corresponding to the adjustment prompt text from at least one of the to-be-adjusted materials, an identification text of the to-be-adjusted material corresponding to the adjustment prompt text is displayed on a visual interface.

16. The method according to claim 14, wherein: The step of obtaining a processed video according to at least one of the adjusted materials comprises: The processed video is displayed in a visual interface.

17. The method according to claim 14, wherein: The determining, according to the adjustment text corresponding to the video to be processed, at least one adjustment prompt text and at least one attribute adjustment information comprises: According to the adjustment text, at least one adjustment prompt text and at least one attribute adjustment information are determined using a large model.

18. A video generating device, comprising: A first determining module, configured to determine a plurality of initial prompt texts according to an initial text input by a user, wherein the plurality of initial prompt texts include an initial content prompt text and an initial material prompt text; A second determination module, configured to determine a video content text and at least one initial object action driving data corresponding to the video content text according to the initial content prompt text; A generating module is used to generate an initial video according to at least one of the initial object action driving data and at least one initial material corresponding to at least one of the initial material prompt texts.

19. A video processing device, comprising: A third determination module, configured to determine at least one adjustment prompt text and at least one property adjustment information according to the adjustment text corresponding to the video to be processed, wherein the video to be processed corresponds to at least one material to be adjusted; an adjustment module, configured to adjust the attribute information of at least one of the to-be-adjusted materials corresponding to at least one of the adjustment prompt texts according to at least one of the attribute adjustment information corresponding to at least one of the adjustment prompt texts, so as to obtain at least one of the adjusted materials; The obtaining module is used to obtain a processed video according to at least one of the adjusted materials.

20. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 17.

21. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 17.

22. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 17.

Images