Digital human video processing method and device, storage medium and system

Abstract

The application provides a digital person video processing method and device, a storage medium and a system. The method comprises the following steps: processing a first image and a first action description text by using a first text-to-video model to obtain a first video file, processing a second image and a second action description text by using a second text-to-video model to obtain a second video file, and storing the first video file and the second video file in a video library; obtaining a GPT interactive text, determining a target video file from the video library based on the GPT interactive text by using GPT technology, and playing the target video file. By considering real-time information and processing the generation of the digital person video file based on the GPT technology, the video file can better conform to the current location of the festival and the solar term, thereby solving the problem that the action of the digital person generated by the current scheme is relatively stiff, and the user experience is poor.

Description

Technical Field

[0001] This application relates to the field of digital human technology, and more specifically, to a method, apparatus, storage medium, and system for processing digital human videos. Background Technology

[0002] Most of the GPT-like assistant applications currently mounted on browsers are just icons, presenting users with the concept of a tool, rather than GPT-like assistant plugins that are displayed to users in the browser as specific virtual characters.

[0003] The current mainstream approach is to generate a digital human-shaped assistant for the user and then drive the digital human to interact with the user. The actions are generally limited to a number of actions that the user has set in advance. The appearance of the digital human is also not diverse enough, which makes the user feel that the other party is rather rigid when interacting with the digital human. Over time, the user will become tired of the digital human image and unwilling to interact with it.

[0004] Current solutions mainly focus on text-driven movement of humanoid virtual characters, controlling the movement of the virtual character by controlling the position of key points on the human body. When a non-humanoid virtual character appears, it may not be able to drive its movement normally, resulting in motion distortion.

[0005] The current solution generates digital human figures with rather stiff movements, resulting in a poor user experience. Summary of the Invention

[0006] The main objective of this application is to provide a method, apparatus, storage medium, and system for processing digital human videos, so as to at least solve the problem that the movements of digital human images generated by current solutions are relatively stiff, resulting in a poor user experience.

[0007] To achieve the above objectives, according to one aspect of this application, a method for processing digital human videos is provided, the method comprising:

[0008] Receive initial setting instructions, and based on the initial setting instructions, use GPT technology to generate a first image of the digital human, a first action description text, and an initial background story text. The initial setting instructions include the basic information of the digital human, including the age, gender, and personality traits of the digital human. The initial background story text is the text of the background story of the digital human.

[0009] Receive real-time information instructions, and based on the real-time information instructions, the first image, and the initial background story text, use GPT technology to generate a second image and a second action description text for the digital human. The real-time information instructions include real-time information, which includes the time information of the current location, the solar term of the current location, and the festival of the current location.

[0010] The first image and the first action description text are processed using a first text-to-video model to obtain a first video file, and the second image and the second action description text are processed using a second text-to-video model to obtain a second video file. The first video file and the second video file are then stored in a video library.

[0011] Obtain GPT interactive text, and based on the GPT interactive text, use GPT technology to determine the target video file from the video library and play the target video file. The target video file is either the first video file or the second video file. The GPT interactive text is the text in which the user interacts with GPT at the current moment.

[0012] Optionally, storing the first video file and the second video file in a video library includes: processing the first video file using a first GPT evaluation model to determine whether the limbs of the digital human in the first video file are distorted, and processing the second video file using a second GPT evaluation model to determine whether the limbs of the digital human in the second video file are distorted, wherein the first GPT evaluation model is used to detect whether the limbs of the digital human in the first video file are distorted, and the second GPT evaluation model is used to detect whether the limbs of the digital human in the second video file are distorted; generating alarm information to indicate whether the limbs of the digital human in the first video file are distorted, and / or to indicate whether the limbs of the digital human in the second video file are distorted; storing the first video file in the video library if the limbs of the digital human in the first video file are not distorted; and storing the second video file in the video library if the limbs of the digital human in the second video file are not distorted.

[0013] Optionally, based on the initial setting instructions, generating a first image, a first action description text, and an initial background story text for the digital human using GPT technology includes: processing the initial setting instructions using a first GPT model to obtain the initial background story text, wherein the first GPT model is used to generate the background story text of the digital human based on the initial setting instructions; processing the initial background story text using a second GPT model to obtain the first action description text, wherein the second GPT model is used to generate the first action description text of the digital human based on the initial background story text; and processing the initial setting instructions using a third GPT model to obtain the first image, wherein the third GPT model is used to generate the first image of the digital human based on the initial setting instructions.

[0014] Optionally, processing the initial background story text using the second GPT model to obtain the first action description text includes: upon receiving a background story modification instruction, modifying the initial background story text based on the background story modification instruction to obtain the target background story text; and processing the target background story text using the second GPT model to obtain the first action description text.

[0015] The method involves processing the first image and the first action description text using a first text-to-video model to obtain a first video file, including: upon receiving an action description modification instruction, modifying the first action description text based on the action description modification instruction to obtain a target action description text; and processing the first image and the target action description text using the first text-to-video model to obtain the first video file.

[0016] Optionally, based on the real-time information command, the first image, and the initial background story text, a second image and a second action description text of the digital human are generated using GPT technology. This includes: processing the real-time information command and the initial background story text using a fourth GPT model to obtain the second action description text, wherein the fourth GPT model is used to correct the initial background story text based on the real-time information command to obtain the second action description text; and processing the real-time information command and the first image using a fifth GPT model to obtain the second image, wherein the fifth GPT model is used to correct the first image based on the real-time information command to obtain the second image.

[0017] Optionally, obtaining GPT interactive text includes: receiving user instructions and using the sixth GPT model to interact with the user instructions to generate the GPT interactive text;

[0018] Based on the GPT interactive text, the target video file is determined from the video library using GPT technology, including: using the seventh GPT model based on the GPT interactive text to calculate the matching degree of all video files in the video library, obtaining a first matching degree and a second matching degree, wherein the first matching degree is the matching degree between the first video file and the GPT interactive text, and the second matching degree is the matching degree between the second video file and the GPT interactive text; if the first matching degree is greater than the second matching degree, the target video file is determined to be the first video file; if the first matching degree is less than or equal to the second matching degree, the target video file is determined to be the second video file.

[0019] Optionally, after determining the target video file from the video library using GPT technology, the method further includes: storing the mapping relationship between the target video file and the GPT interactive text in a video mapping relationship table of the database.

[0020] According to another aspect of this application, a digital human video processing apparatus is provided, the apparatus comprising:

[0021] The first receiving unit is used to receive an initial setting instruction and, based on the initial setting instruction, use GPT technology to generate a first image of the digital human, a first action description text, and an initial background story text. The initial setting instruction includes the basic information of the digital human, which includes the age, gender, and personality traits of the digital human. The initial background story text is the text of the background story of the digital human.

[0022] The second receiving unit is used to receive real-time information instructions, and based on the real-time information instructions, the first image, and the initial background story text, to generate a second image and a second action description text of the digital human using GPT technology. The real-time information instructions include real-time information, which includes the time information of the current location, the solar term of the current location, and the festival of the current location.

[0023] The first processing unit is configured to process the first image and the first action description text using a first text-to-video model to obtain a first video file, and to process the second image and the second action description text using a second text-to-video model to obtain a second video file, and to store the first video file and the second video file in a video library.

[0024] The second processing unit is used to acquire GPT interactive text, and based on the GPT interactive text, use GPT technology to determine a target video file from the video library and play the target video file. The target video file is either the first video file or the second video file. The GPT interactive text is the text in which the user interacts with GPT at the current moment.

[0025] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform any of the aforementioned digital human video processing methods.

[0026] According to another aspect of this application, a digital human video processing system is provided, the system comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including methods for performing any of the digital human video processing methods described above.

[0027] By applying the technical solution of this application, and taking into account real-time information, the generation of video files for digital humans is processed based on GPT technology. This makes the video files more consistent with the festival and solar term of the current location, and makes the movements of the digital humans in the video files more flexible than existing solutions, thereby improving the user experience and solving the problem that the movements of the digital human images generated by the current solutions are relatively stiff, resulting in a poor user experience. Attached Figure Description

[0028] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0029] Figure 1 A flowchart illustrating a method for processing digital human video according to an embodiment of this application is shown.

[0030] Figure 2 A flowchart illustrating another method for processing digital human video according to an embodiment of this application is shown.

[0031] Figure 3 A structural block diagram of a digital human video processing apparatus provided according to an embodiment of this application is shown. Detailed Implementation

[0032] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0033] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0034] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0035] As described in the background section, most current GPT-like assistant applications mounted on browsers are merely icons, presenting the user with the concept of a tool, rather than GPT-like assistant plugins that display a concrete virtual character image in the browser. Current mainstream solutions generate digital human-shaped assistants for users and then drive these digital humans to interact with them. The actions are generally limited to a few pre-set actions, and the digital human images lack diversity, making the interaction feel somewhat rigid. Over time, users become bored with the digital human image and unwilling to interact with it. Current solutions mainly focus on text-driven humanoid virtual character movement, controlling the virtual character's movement by controlling the position of key points on the human body. When non-humanoid virtual characters appear, their movement may not be properly driven, resulting in motion distortion. To address the problem of rigid movements and poor user experience caused by current solutions, embodiments of this application provide a method, apparatus, storage medium, and system for processing digital human videos.

[0036] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0037] This embodiment provides a method for processing digital human video. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0038] Figure 1 This is a schematic flowchart illustrating a digital human video processing method according to an embodiment of this application. Figure 1 As shown, the method includes the following steps:

[0039] Step S101: Receive initial setting instructions, and based on the initial setting instructions, use GPT technology to generate the first image of the digital human, the first action description text, and the initial background story text. The initial setting instructions include the basic information of the digital human, including the age, gender, and personality traits of the digital human. The initial background story text is the text of the background story of the digital human.

[0040] Specifically, it receives initial setting instructions from the user, and the personality traits can be such as cheerful or shy. Digital humans with different personality traits will perform actions in different ways.

[0041] Step S101, based on the aforementioned initial setting instructions, uses GPT technology to generate the first image of the digital human, the first action description text, and the initial background story text, including:

[0042] The first GPT model is used to process the above initial setting instructions to obtain the above initial background story text. The first GPT model is used to generate the background story text of the above digital human based on the above initial setting instructions.

[0043] Specifically, GPT technology is used to generate the background story text of the digital human based on the basic information of the digital human included in the initial setting instructions;

[0044] The initial background story text is processed using the second GPT model to obtain the first action description text. The second GPT model is used to generate the first action description text of the digital human based on the initial background story text.

[0045] Specifically, the process of processing the initial background story text using the second GPT model to obtain the first action description text includes: upon receiving a background story modification instruction, modifying the initial background story text based on the background story modification instruction to obtain a target background story text; and processing the target background story text using the second GPT model to obtain the first action description text.

[0046] The initial setting instructions are processed using the third GPT model to obtain the first image image. The third GPT model is used to generate the first image image of the digital human based on the initial setting instructions.

[0047] Step S102: Receive real-time information instructions, and based on the real-time information instructions, the first image, and the initial background story text, use GPT technology to generate the second image and second action description text of the digital human. The real-time information instructions include real-time information, which includes the time information of the current location, the solar term of the current location, and the festival of the current location.

[0048] Step S102, which generates the second image and second action description text of the digital human based on the aforementioned real-time information command, the aforementioned first image, and the aforementioned initial background story text using GPT technology, includes: processing the aforementioned real-time information command and the aforementioned initial background story text using a fourth GPT model to obtain the aforementioned second action description text; the aforementioned fourth GPT model is used to correct the aforementioned initial background story text based on the aforementioned real-time information command to obtain the aforementioned second action description text; and processing the aforementioned real-time information command and the aforementioned first image using a fifth GPT model to obtain the aforementioned second image; the aforementioned fifth GPT model is used to correct the aforementioned first image based on the aforementioned real-time information command to obtain the aforementioned second image.

[0049] Specifically, by taking real-time information into account, the subsequent second video file can better reflect the current solar term and festival atmosphere, thereby improving the user experience.

[0050] Step S103: The first image and the first action description text are processed using the first text-to-video model to obtain the first video file, and the second image and the second action description text are processed using the second text-to-video model to obtain the second video file. The first video file and the second video file are then stored in the video library.

[0051] The first and second text-to-video models can be large-scale video generation models open-sourced by Stability AI, namely the Stable Video Diffusion model.

[0052] The Stable Video Diffusion model is a large-scale video generation model open-sourced by Stability AI. It's a model for generating stable, high-quality videos. Utilizing deep learning technology, it can generate realistic and stable video content, producing smooth and continuous video effects. This model can be used in various video generation applications, including video effects, video editing, and film production. With the Stable Video Diffusion model, users can easily generate high-quality video content, providing powerful tools and technical support for video creation.

[0053] Step S103, which involves using a first text-to-video model to process the first image and the first action description text to obtain a first video file, includes: upon receiving an action description modification instruction, modifying the first action description text based on the action description modification instruction to obtain a target action description text; and using the first text-to-video model to process the first image and the target action description text to obtain the first video file.

[0054] Specifically, when a user needs to modify the first action description text, the system receives the user's input action description modification instruction, modifies the first action description text based on the action description modification instruction, obtains the target action description text, and then uses the first text to generate a video to process the first image and the target action description text to obtain the first video file, thereby achieving a further correction effect.

[0055] Step S103, storing the first video file and the second video file into the video library, includes:

[0056] The first video file is processed using a first GPT evaluation model to determine whether the limbs of the digital human in the first video file are distorted, and the second video file is processed using a second GPT evaluation model to determine whether the limbs of the digital human in the second video file are distorted. The first GPT evaluation model is used to detect whether the limbs of the digital human in the first video file are distorted, and the second GPT evaluation model is used to detect whether the limbs of the digital human in the second video file are distorted.

[0057] If it is determined that the limbs of the digital human in the first video file are distorted, and / or if it is determined that the limbs of the digital human in the second video file are distorted, an alarm message is generated to indicate that the limbs of the digital human in the first video file are distorted, and / or to indicate that the limbs of the digital human in the second video file are distorted; if it is determined that the limbs of the digital human in the first video file are not distorted, the first video file is stored in the video library.

[0058] If it is determined that the limbs of the digital human in the second video file are not distorted, the second video file is stored in the video library.

[0059] Specifically, by detecting whether the limbs of digital humans in video files are distorted, the video files can be made more comfortable for users to watch, thus improving the user experience.

[0060] Step S104: Obtain GPT interaction text, and based on the GPT interaction text, use GPT technology to determine the target video file from the video library and play the target video file. The target video file is either the first video file or the second video file. The GPT interaction text is the text in which the user interacts with GPT at the current moment.

[0061] In the above steps, by taking real-time information into account and processing the generation of digital human video files based on GPT technology, the video files are made more consistent with the festival and solar term of the current location. This makes the movements of the digital human in the video files more flexible compared to existing solutions, improving the user experience and solving the problem that the movements of the digital human images generated by the current solutions are relatively stiff, resulting in a poor user experience.

[0062] The step S104 of obtaining GPT interactive text includes: receiving user instructions and using the sixth GPT model to interact with the user instructions to generate the GPT interactive text.

[0063] Specifically, for example, if the user instruction is for the user to greet someone, then the GPT interactive text would be for someone to greet the user, such as adding New Year's greetings and actions during the Chinese New Year, thereby improving the user experience.

[0064] Step S104, which involves determining the target video file from the video library using GPT technology based on the aforementioned GPT interactive text, includes: using the seventh GPT model based on the aforementioned GPT interactive text to calculate the matching degree of all video files in the video library, obtaining a first matching degree and a second matching degree. The first matching degree is the matching degree between the first video file and the aforementioned GPT interactive text, and the second matching degree is the matching degree between the second video file and the aforementioned GPT interactive text. If the first matching degree is greater than the second matching degree, the target video file is determined to be the first video file. If the first matching degree is less than or equal to the second matching degree, the target video file is determined to be the second video file.

[0065] Specifically, based on the matching degree between GPT interactive text and each video file, the video file with the highest matching degree is found as the target video file, which improves the user experience.

[0066] In one embodiment of this application, after determining the target video file from the video library using GPT technology, the method further includes: storing the mapping relationship between the target video file and the GPT interactive text in a video mapping relationship table of the database.

[0067] Specifically, as the number of mapping relationships in the video mapping table gradually increases, the video file corresponding to the GPT interaction text in the video mapping table can be played directly based on the GPT interaction text, thus saving the processing time of the previous steps and providing users with a more efficient and comfortable interactive experience.

[0068] To enable those skilled in the art to better understand the technical solution of this application, the implementation process of the digital human video processing method of this application will be described in detail below with reference to specific embodiments.

[0069] This embodiment relates to a specific method for processing digital human videos, such as... Figure 2 As shown, it includes the following steps:

[0070] Step S1: Receive initial setup instructions. The initial setup instructions include the basic information of the digital human, including the digital human's age, gender, and personality traits. The initial background story text is the text of the digital human's background story.

[0071] Step S2: The initial setting instructions are processed using the first GPT model to obtain the initial background story text. The first GPT model is used to generate the background story text of the digital human based on the initial setting instructions. In the case of receiving a background story modification instruction, the initial background story text is modified based on the background story modification instruction to obtain the target background story text.

[0072] Step S3: The initial background story text is processed using the second GPT model to obtain the first action description text. The second GPT model is used to generate the first action description text of the digital human based on the initial background story text. Specifically, when a background story modification instruction is received, the target background story text is processed using the second GPT model to obtain the first action description text.

[0073] Step S4: The initial setting instructions are processed using the third GPT model to obtain the first image. The third GPT model is used to generate the first image of the digital human based on the initial setting instructions.

[0074] Step S5: Receive real-time information instructions and process the real-time information instructions and initial background story text using the fourth GPT model to obtain the second action description text. The fourth GPT model is used to correct the initial background story text based on the real-time information instructions to obtain the second action description text. Process the real-time information instructions and first image using the fifth GPT model to obtain the second image. The fifth GPT model is used to correct the first image based on the real-time information instructions to obtain the second image. The real-time information instructions include real-time information, which includes the time information of the current location, the solar term of the current location, and the festival of the current location.

[0075] Step S6: Use the first text-to-video model to process the first image and the first action description text to obtain the first video file, and use the second text-to-video model to process the second image and the second action description text to obtain the second video file;

[0076] Step S7: Process the first video file using the first GPT evaluation model to determine whether the limbs of the digital human in the first video file are distorted, and process the second video file using the second GPT evaluation model to determine whether the limbs of the digital human in the second video file are distorted. The first GPT evaluation model is used to detect whether the limbs of the digital human in the first video file are distorted, and the second GPT evaluation model is used to detect whether the limbs of the digital human in the second video file are distorted. If it is determined that the limbs of the digital human in the first video file are distorted, and / or if it is determined that the limbs of the digital human in the second video file are distorted, generate an alarm message to indicate that the limbs of the digital human in the first video file are distorted, and / or to indicate that the limbs of the digital human in the second video file are distorted. If it is determined that the limbs of the digital human in the first video file are not distorted, store the first video file in the video library. If it is determined that the limbs of the digital human in the second video file are not distorted, store the second video file in the video library.

[0077] Step S8: Receive user instructions and use the sixth GPT model to interact with the user instructions and generate GPT interactive text. Then, use the seventh GPT model based on the GPT interactive text to calculate the matching degree of all video files in the video library, obtaining a first matching degree and a second matching degree. The first matching degree is the matching degree between the first video file and the GPT interactive text, and the second matching degree is the matching degree between the second video file and the GPT interactive text. If the first matching degree is greater than the second matching degree, the target video file is determined to be the first video file. If the first matching degree is less than or equal to the second matching degree, the target video file is determined to be the second video file, and the target video file is played. The target video file is either the first video file or the second video file. The GPT interactive text is the text that the user interacts with GPT at the current moment.

[0078] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0079] This application also provides a digital human video processing apparatus. It should be noted that the digital human video processing apparatus of this application can be used to execute the digital human video processing method provided in this application. This apparatus is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0080] The following describes the digital human video processing apparatus provided in the embodiments of this application.

[0081] Figure 3 This is a structural block diagram of a digital human video processing apparatus provided according to an embodiment of this application. Figure 3 As shown, the device includes:

[0082] The first receiving unit 31 is used to receive an initial setting instruction and, based on the initial setting instruction, use GPT technology to generate a first image of the digital human, a first action description text, and an initial background story text. The initial setting instruction includes the basic information of the digital human, including the age, gender, and personality traits of the digital human. The initial background story text is the text of the background story of the digital human.

[0083] The second receiving unit 32 is used to receive real-time information instructions, and based on the real-time information instructions, the first image, and the initial background story text, to generate the second image and the second action description text of the digital human using GPT technology. The real-time information instructions include real-time information, which includes the time information of the current location, the solar term of the current location, and the festival of the current location.

[0084] The first processing unit 33 is used to process the first image and the first action description text using a first text-to-video model to obtain a first video file, and to process the second image and the second action description text using a second text-to-video model to obtain a second video file, and to store the first video file and the second video file in a video library.

[0085] The second processing unit 34 is used to acquire GPT interactive text, and based on the GPT interactive text, use GPT technology to determine the target video file from the video library and play the target video file. The target video file is either the first video file or the second video file. The GPT interactive text is the text in which the user interacts with GPT at the current moment.

[0086] In the aforementioned device, by taking into account real-time information and processing the generation of digital human video files based on GPT technology, the video files are made more consistent with the festival and solar term of the current location. This makes the movements of the digital human in the video files more flexible compared to existing solutions, improving the user experience and solving the problem that the movements of the digital human images generated by the current solutions are relatively stiff, resulting in a poor user experience.

[0087] In one embodiment of this application, the first processing unit includes a first processing module, a second processing module, a third processing module, and a fourth processing module. The first processing module is used to process the first video file using a first GPT evaluation model to determine whether the limbs of the digital human in the first video file are distorted, and to process the second video file using a second GPT evaluation model to determine whether the limbs of the digital human in the second video file are distorted. The first GPT evaluation model is used to detect whether the limbs of the digital human in the first video file are distorted, and the second GPT evaluation model is used to detect whether the limbs of the digital human in the second video file are distorted. The second processing module is used to generate alarm information to indicate that the limbs of the digital human in the first video file are distorted, and / or to indicate that the limbs of the digital human in the second video file are distorted, when it is determined that the limbs of the digital human in the first video file are distorted; the third processing module is used to store the first video file in the video library when it is determined that the limbs of the digital human in the first video file are not distorted; the fourth processing module is used to store the second video file in the video library when it is determined that the limbs of the digital human in the second video file are not distorted.

[0088] In one embodiment of this application, the second receiving unit includes a fifth processing module, a sixth processing module, and a seventh processing module. The fifth processing module is used to process the initial setting instruction using a first GPT model to obtain the initial background story text. The first GPT model is used to generate the background story text of the digital human based on the initial setting instruction. The sixth processing module is used to process the initial background story text using a second GPT model to obtain the first action description text. The second GPT model is used to generate the first action description text of the digital human based on the initial background story text. The seventh processing module is used to process the initial setting instruction using a third GPT model to obtain the first image. The third GPT model is used to generate the first image of the digital human based on the initial setting instruction.

[0089] In one embodiment of this application, the sixth processing module includes a first processing submodule and a second processing submodule. The first processing submodule is used to modify the initial background story text based on the background story modification instruction when a background story modification instruction is received, to obtain a target background story text. The second processing submodule is used to process the target background story text using the second GPT model to obtain the first action description text.

[0090] The first processing unit includes an eighth processing module and a ninth processing module. The eighth processing module is used to modify the first action description text based on the action description modification instruction when it receives the action description modification instruction, so as to obtain the target action description text. The ninth processing module is used to process the first image and the target action description text using the first text to generate a video model, so as to obtain the first video file.

[0091] In one embodiment of this application, the second receiving unit includes a tenth processing module and an eleventh processing module. The tenth processing module is used to process the real-time information instruction and the initial background story text using a fourth GPT model to obtain the second action description text. The fourth GPT model is used to correct the initial background story text based on the real-time information instruction to obtain the second action description text. The eleventh processing module is used to process the real-time information instruction and the first image using a fifth GPT model to obtain the second image. The fifth GPT model is used to correct the first image based on the real-time information instruction to obtain the second image.

[0092] In one embodiment of this application, the second processing unit includes a receiving module, a twelfth processing module, a thirteenth processing module, and a fourteenth processing module. The receiving module is used to receive user instructions and use the sixth GPT model to interact with the user instructions to generate the GPT interactive text.

[0093] The twelfth processing module is used to calculate the matching degree of all video files in the video library based on the GPT interactive text using the seventh GPT model, to obtain a first matching degree and a second matching degree. The first matching degree is the matching degree between the first video file and the GPT interactive text, and the second matching degree is the matching degree between the second video file and the GPT interactive text. The thirteenth processing module is used to determine the target video file as the first video file when the first matching degree is greater than the second matching degree. The fourteenth processing module is used to determine the target video file as the second video file when the first matching degree is less than or equal to the second matching degree.

[0094] In one embodiment of this application, the above-mentioned apparatus further includes a third processing unit. After determining the target video file from the video library using GPT technology, the third processing unit is used to store the mapping relationship between the target video file and the GPT interactive text in a video mapping relationship table of the database.

[0095] The aforementioned digital human video processing device includes a processor and a memory. The first receiving unit, second receiving unit, first processing unit, and second processing unit are all stored as program units in the memory. The processor executes these program units stored in the memory to achieve their respective functions. All of the above modules are located in the same processor; alternatively, the modules may be located in different processors in any combination.

[0096] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured; adjusting kernel parameters can address the problem of current solutions generating digital human figures with stiff movements, resulting in a poor user experience.

[0097] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0098] This invention provides a computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device containing the computer-readable storage medium to perform the digital human video processing method.

[0099] This invention provides a processor for running a program, wherein the program executes the digital human video processing method.

[0100] This invention provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs at least the following steps: receiving an initial setting instruction, and based on the initial setting instruction, generating a first image of a digital human, a first action description text, and an initial background story text using GPT technology. The initial setting instruction includes basic information about the digital human, including its age, gender, and personality traits. The initial background story text is the text of the digital human's background story. The invention also provides a real-time information instruction, and based on the real-time information instruction, the first image, and the initial background story text, generating a second image of the digital human and a second action description text using GPT technology. The real-time information instruction includes real-time information... The aforementioned real-time information includes the current location's time, the solar term at the current location, and the festival at the current location. A first text-to-video model is used to process the first image and the first action description text to obtain a first video file. A second text-to-video model is then used to process the second image and the second action description text to obtain a second video file. Both the first and second video files are stored in a video library. GPT interactive text is obtained, and based on this text, GPT technology is used to determine a target video file from the video library and play the target video file. The target video file is either the first or the second video file. The GPT interactive text is the text in which the user interacts with GPT at the current moment. The devices mentioned in this document can be servers, PCs, tablets, mobile phones, etc.

[0101] Optionally, storing the first video file and the second video file in a video library includes: processing the first video file using a first GPT evaluation model to determine whether the limbs of the digital human in the first video file are distorted, and processing the second video file using a second GPT evaluation model to determine whether the limbs of the digital human in the second video file are distorted; the first GPT evaluation model is used to detect whether the limbs of the digital human in the first video file are distorted, and the second GPT evaluation model is used to detect whether the limbs of the digital human in the second video file are distorted; if it is determined that the limbs of the digital human in the first video file are distorted, and / or if it is determined that the limbs of the digital human in the second video file are distorted, generating an alarm message to indicate that the limbs of the digital human in the first video file are distorted, and / or indicating that the limbs of the digital human in the second video file are distorted; if it is determined that the limbs of the digital human in the first video file are not distorted, storing the first video file in the video library; if it is determined that the limbs of the digital human in the second video file are not distorted, storing the second video file in the video library.

[0102] Optionally, based on the aforementioned initial setting instructions, the generation of the digital human's first image, first action description text, and initial background story text using GPT technology includes: processing the aforementioned initial setting instructions using a first GPT model to obtain the aforementioned initial background story text, wherein the first GPT model is used to generate the background story text of the digital human based on the aforementioned initial setting instructions; processing the aforementioned initial background story text using a second GPT model to obtain the aforementioned first action description text, wherein the second GPT model is used to generate the aforementioned first action description text of the digital human based on the aforementioned initial background story text; and processing the aforementioned initial setting instructions using a third GPT model to obtain the aforementioned first image, wherein the third GPT model is used to generate the aforementioned first image of the digital human based on the aforementioned initial setting instructions.

[0103] Optionally, the initial background story text is processed using the second GPT model to obtain the first action description text, including: upon receiving a background story modification instruction, modifying the initial background story text based on the background story modification instruction to obtain a target background story text; and processing the target background story text using the second GPT model to obtain the first action description text.

[0104] The first image and the first action description text are processed using a first text-to-video model to obtain a first video file, including: upon receiving an action description modification instruction, modifying the first action description text based on the action description modification instruction to obtain a target action description text; and processing the first image and the target action description text using the first text-to-video model to obtain the first video file.

[0105] Optionally, based on the aforementioned real-time information command, the aforementioned first image, and the aforementioned initial background story text, the second image and second action description text of the digital human are generated using GPT technology, including: processing the aforementioned real-time information command and the aforementioned initial background story text using a fourth GPT model to obtain the aforementioned second action description text, wherein the fourth GPT model is used to correct the aforementioned initial background story text based on the aforementioned real-time information command to obtain the aforementioned second action description text; and processing the aforementioned real-time information command and the aforementioned first image using a fifth GPT model to obtain the aforementioned second image, wherein the fifth GPT model is used to correct the aforementioned first image based on the aforementioned real-time information command to obtain the aforementioned second image.

[0106] Optionally, obtaining GPT interactive text includes: receiving user instructions and using the sixth GPT model to interact with the user instructions to generate the GPT interactive text.

[0107] Based on the aforementioned GPT interactive text, the target video file is determined from the aforementioned video library using GPT technology, including: using the seventh GPT model based on the aforementioned GPT interactive text, calculating the matching degree of all video files in the aforementioned video library to obtain a first matching degree and a second matching degree, wherein the first matching degree is the matching degree between the aforementioned first video file and the aforementioned GPT interactive text, and the aforementioned second matching degree is the matching degree between the aforementioned second video file and the aforementioned GPT interactive text; if the aforementioned first matching degree is greater than the aforementioned second matching degree, the target video file is determined to be the aforementioned first video file; if the aforementioned first matching degree is less than or equal to the aforementioned second matching degree, the target video file is determined to be the aforementioned second video file.

[0108] Optionally, after determining the target video file from the video library using GPT technology, the method further includes storing the mapping relationship between the target video file and the GPT interactive text in a video mapping table of the database.

[0109] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having at least the following method steps: receiving an initial setting instruction, and based on the initial setting instruction, using GPT technology to generate a first image of a digital human, a first action description text, and an initial background story text, wherein the initial setting instruction includes basic information about the digital human, including the digital human's age, gender, and personality traits, and the initial background story text is the text of the digital human's background story; receiving a real-time information instruction, and based on the real-time information instruction, the first image, and the initial background story text, using GPT technology to generate a second image of the digital human and a second action description text, wherein the real-time information instruction includes real-time information, and the real-time information... This includes the current location's time information, the solar term at the current location, and the festival at the current location; a first text-to-video model is used to process the first image and the first action description text to obtain a first video file, and a second text-to-video model is used to process the second image and the second action description text to obtain a second video file, and the first and second video files are stored in a video library; GPT interactive text is obtained, and based on the GPT interactive text, a target video file is determined from the video library using GPT technology, and the target video file is played, wherein the target video file is either the first video file or the second video file, and the GPT interactive text is the text in which the user interacts with GPT at the current moment.

[0110] Optionally, storing the first video file and the second video file in a video library includes: processing the first video file using a first GPT evaluation model to determine whether the limbs of the digital human in the first video file are distorted, and processing the second video file using a second GPT evaluation model to determine whether the limbs of the digital human in the second video file are distorted; the first GPT evaluation model is used to detect whether the limbs of the digital human in the first video file are distorted, and the second GPT evaluation model is used to detect whether the limbs of the digital human in the second video file are distorted; if it is determined that the limbs of the digital human in the first video file are distorted, and / or if it is determined that the limbs of the digital human in the second video file are distorted, generating an alarm message to indicate that the limbs of the digital human in the first video file are distorted, and / or indicating that the limbs of the digital human in the second video file are distorted; if it is determined that the limbs of the digital human in the first video file are not distorted, storing the first video file in the video library; if it is determined that the limbs of the digital human in the second video file are not distorted, storing the second video file in the video library.

[0111] Optionally, based on the aforementioned initial setting instructions, the generation of the digital human's first image, first action description text, and initial background story text using GPT technology includes: processing the aforementioned initial setting instructions using a first GPT model to obtain the aforementioned initial background story text, wherein the first GPT model is used to generate the background story text of the digital human based on the aforementioned initial setting instructions; processing the aforementioned initial background story text using a second GPT model to obtain the aforementioned first action description text, wherein the second GPT model is used to generate the aforementioned first action description text of the digital human based on the aforementioned initial background story text; and processing the aforementioned initial setting instructions using a third GPT model to obtain the aforementioned first image, wherein the third GPT model is used to generate the aforementioned first image of the digital human based on the aforementioned initial setting instructions.

[0112] Optionally, the initial background story text is processed using the second GPT model to obtain the first action description text, including: upon receiving a background story modification instruction, modifying the initial background story text based on the background story modification instruction to obtain a target background story text; and processing the target background story text using the second GPT model to obtain the first action description text.

[0113] The first image and the first action description text are processed using a first text-to-video model to obtain a first video file, including: upon receiving an action description modification instruction, modifying the first action description text based on the action description modification instruction to obtain a target action description text; and processing the first image and the target action description text using the first text-to-video model to obtain the first video file.

[0114] Optionally, based on the aforementioned real-time information command, the aforementioned first image, and the aforementioned initial background story text, the second image and second action description text of the digital human are generated using GPT technology, including: processing the aforementioned real-time information command and the aforementioned initial background story text using a fourth GPT model to obtain the aforementioned second action description text, wherein the fourth GPT model is used to correct the aforementioned initial background story text based on the aforementioned real-time information command to obtain the aforementioned second action description text; and processing the aforementioned real-time information command and the aforementioned first image using a fifth GPT model to obtain the aforementioned second image, wherein the fifth GPT model is used to correct the aforementioned first image based on the aforementioned real-time information command to obtain the aforementioned second image.

[0115] Optionally, obtaining GPT interactive text includes: receiving user instructions and using the sixth GPT model to interact with the user instructions to generate the GPT interactive text.

[0116] Based on the aforementioned GPT interactive text, the target video file is determined from the aforementioned video library using GPT technology, including: using the seventh GPT model based on the aforementioned GPT interactive text, calculating the matching degree of all video files in the aforementioned video library to obtain a first matching degree and a second matching degree, wherein the first matching degree is the matching degree between the aforementioned first video file and the aforementioned GPT interactive text, and the aforementioned second matching degree is the matching degree between the aforementioned second video file and the aforementioned GPT interactive text; if the aforementioned first matching degree is greater than the aforementioned second matching degree, the target video file is determined to be the aforementioned first video file; if the aforementioned first matching degree is less than or equal to the aforementioned second matching degree, the target video file is determined to be the aforementioned second video file.

[0117] Optionally, after determining the target video file from the video library using GPT technology, the method further includes storing the mapping relationship between the target video file and the GPT interactive text in a video mapping table of the database.

[0118] This application also provides a digital human video processing system, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include methods for executing any of the aforementioned digital human video processing methods. By considering real-time information and processing the generation of digital human video files based on GPT technology, the video files are made more consistent with the current location's festivals and solar terms, making the digital human's movements in the video files more flexible compared to existing solutions, improving the user experience, and thus solving the problem that the movements of digital human images generated by current solutions are relatively stiff, resulting in a poor user experience.

[0119] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0120] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0121] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0122] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0123] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0124] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0125] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0126] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0127] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0128] As can be seen from the above description, the embodiments of this application achieve the following technical effects:

[0129] 1) The digital human video processing method of this application takes into account real-time information and uses GPT technology to process the generation of digital human video files, so that the video files are more in line with the festival and solar term of the current location, and the movements of the digital human in the video files are more flexible than the existing solutions, improving the user experience and solving the problem that the movements of the digital human images generated by the current solutions are relatively stiff, resulting in a poor user experience.

[0130] 2) The digital human video processing device of this application, by taking into account real-time information and processing the generation of digital human video files based on GPT technology, makes the video files more consistent with the festival and solar term of the current location, and makes the movements of the digital human in the video files more flexible than the existing solutions, thereby improving the user experience and solving the problem that the movements of the digital human images generated by the current solutions are relatively stiff, resulting in a poor user experience.

[0131] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for processing digital human videos, characterized in that, include: Receive initial setting instructions, and based on the initial setting instructions, use GPT technology to generate a first image of the digital human, a first action description text, and an initial background story text. The initial setting instructions include the basic information of the digital human, including the age, gender, and personality traits of the digital human. The initial background story text is the text of the background story of the digital human. Receive real-time information instructions, and based on the real-time information instructions, the first image, and the initial background story text, use GPT technology to generate a second image and a second action description text for the digital human. The real-time information instructions include real-time information, which includes the time information of the current location, the solar term of the current location, and the festival of the current location. The first image and the first action description text are processed using a first text-to-video model to obtain a first video file, and the second image and the second action description text are processed using a second text-to-video model to obtain a second video file. The first video file and the second video file are then stored in a video library. Obtain GPT interactive text, and based on the GPT interactive text, use GPT technology to determine the target video file from the video library and play the target video file. The target video file is either the first video file or the second video file. The GPT interactive text is the text in which the user interacts with GPT at the current moment.

2. The method according to claim 1, characterized in that, Storing the first video file and the second video file into the video library includes: The first video file is processed using a first GPT evaluation model to determine whether the limbs of the digital human in the first video file are distorted, and the second video file is processed using a second GPT evaluation model to determine whether the limbs of the digital human in the second video file are distorted. The first GPT evaluation model is used to detect whether the limbs of the digital human in the first video file are distorted, and the second GPT evaluation model is used to detect whether the limbs of the digital human in the second video file are distorted. If it is determined that the limbs of the digital human in the first video file are distorted, and / or if it is determined that the limbs of the digital human in the second video file are distorted, an alarm message is generated to indicate that the limbs of the digital human in the first video file are distorted, and / or to indicate that the limbs of the digital human in the second video file are distorted; If it is determined that the limbs of the digital human described in the first video file are not distorted, the first video file is stored in the video library; If it is determined that the limbs of the digital human described in the second video file have not been distorted, the second video file is stored in the video library.

3. The method according to claim 1, characterized in that, Based on the initial setting instructions, the first image of the digital human, the first action description text, and the initial background story text are generated using GPT technology, including: The initial setting instructions are processed using the first GPT model to obtain the initial background story text. The first GPT model is used to generate the background story text of the digital human based on the initial setting instructions. The initial background story text is processed using a second GPT model to obtain the first action description text. The second GPT model is used to generate the first action description text of the digital human based on the initial background story text. The initial setting instructions are processed using the third GPT model to obtain the first image image. The third GPT model is used to generate the first image image of the digital human based on the initial setting instructions.

4. The method according to claim 3, characterized in that, The process of processing the initial background story text using the second GPT model to obtain the first action description text includes: upon receiving a background story modification instruction, modifying the initial background story text based on the background story modification instruction to obtain the target background story text; and processing the target background story text using the second GPT model to obtain the first action description text. The method involves processing the first image and the first action description text using a first text-to-video model to obtain a first video file, including: upon receiving an action description modification instruction, modifying the first action description text based on the action description modification instruction to obtain a target action description text; and processing the first image and the target action description text using the first text-to-video model to obtain the first video file.

5. The method according to claim 1, characterized in that, Based on the real-time information command, the first image, and the initial background story text, a second image and a second action description text of the digital human are generated using GPT technology, including: The fourth GPT model is used to process the real-time information command and the initial background story text to obtain the second action description text. The fourth GPT model is used to correct the initial background story text based on the real-time information command to obtain the second action description text. The real-time information command and the first image are processed using the fifth GPT model to obtain the second image. The fifth GPT model is used to correct the first image based on the real-time information command to obtain the second image.

6. The method according to any one of claims 1 to 5, characterized in that, Obtaining GPT interactive text includes: receiving user instructions and using the sixth GPT model to interact with the user instructions to generate the GPT interactive text; Based on the GPT interactive text, the target video file is determined from the video library using GPT technology, including: using the seventh GPT model based on the GPT interactive text to calculate the matching degree of all video files in the video library, obtaining a first matching degree and a second matching degree, wherein the first matching degree is the matching degree between the first video file and the GPT interactive text, and the second matching degree is the matching degree between the second video file and the GPT interactive text; if the first matching degree is greater than the second matching degree, the target video file is determined to be the first video file; if the first matching degree is less than or equal to the second matching degree, the target video file is determined to be the second video file.

7. The method according to any one of claims 1 to 5, characterized in that, After determining the target video file from the video library using GPT technology, the method further includes: The mapping relationship between the target video file and the GPT interactive text is stored in the video mapping relationship table of the database.

8. A digital human video processing apparatus, characterized in that, include: The first receiving unit is used to receive an initial setting instruction and, based on the initial setting instruction, use GPT technology to generate a first image of the digital human, a first action description text, and an initial background story text. The initial setting instruction includes the basic information of the digital human, which includes the age, gender, and personality traits of the digital human. The initial background story text is the text of the background story of the digital human. The second receiving unit is used to receive real-time information instructions, and based on the real-time information instructions, the first image, and the initial background story text, to generate a second image and a second action description text of the digital human using GPT technology. The real-time information instructions include real-time information, which includes the time information of the current location, the solar term of the current location, and the festival of the current location. The first processing unit is configured to process the first image and the first action description text using a first text-to-video model to obtain a first video file, and to process the second image and the second action description text using a second text-to-video model to obtain a second video file, and to store the first video file and the second video file in a video library. The second processing unit is used to acquire GPT interactive text, and based on the GPT interactive text, use GPT technology to determine a target video file from the video library and play the target video file. The target video file is either the first video file or the second video file. The GPT interactive text is the text in which the user interacts with GPT at the current moment.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the digital human video processing method according to any one of claims 1 to 7.

10. A digital human video processing system, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including a method for processing digital human video according to any one of claims 1 to 7.

Images