Methods, apparatus, electronic devices and readable storage media for generating digital human videos

By decoupling the digital human video generation process into multiple stages and utilizing a multimodal large model for processing, the problems of high cost and low efficiency in existing technologies are solved, and flexible and efficient digital human video generation is achieved.

CN123123665APending Publication Date: 2026-11-17BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611199944.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-07
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies are costly and inefficient in generating digital human videos. If any step in the end-to-end generation solution fails, the entire process must be re-executed, leading to generation failure.

Method used

The digital human video generation process is broken down into four stages: target background image, video, parameters, and template video. A multimodal large model is used for decoupling processing, supporting independent storage and retries. Token bucket and user priority management are combined to manage the parallel processing of the multimodal large model.

Benefits of technology

It reduces the cost of digital human video generation, improves generation efficiency and flexibility, supports independent processing at any stage, and avoids waste of computing resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123665A_ABST
    Figure CN123123665A_ABST
Patent Text Reader

Abstract

This disclosure provides a method, apparatus, electronic device, and readable storage medium for generating digital human videos, relating to artificial intelligence technologies such as computer vision, large-scale models, and digital humans. The method for generating digital human videos includes: obtaining a target background image based on a target user's avatar image and a target scene; obtaining a target background video including the target scene and the target user based on the target background image; obtaining target digital human parameters corresponding to the target scene and the target user based on the target background video; obtaining a target template video including the target scene and the target digital human based on target audio parameters and the target digital human parameters; and, in response to the target template video passing review, obtaining a target digital human video based on the video parameters and target text of the target template video. This disclosure decouples the digital human video generation process, improving the flexibility of digital human video generation, reducing the generation cost, and increasing the generation efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of internet technology, and more particularly to the fields of artificial intelligence technology such as computer vision, large-scale models, and digital humans. It provides a method, apparatus, electronic device, and readable storage medium for generating digital human videos. Background Technology

[0002] Existing technologies typically require users to shoot videos to generate digital human videos, resulting in high generation costs. Furthermore, existing technologies usually employ end-to-end generation schemes when generating digital human videos from shot videos, which suffers from tight coupling between various stages in the conversion process from "shot video" to "digital human video." Failure in any stage requires the entire process to be re-executed, leading to low generation efficiency for digital human videos. Summary of the Invention

[0003] According to a first aspect of this disclosure, a method for generating a digital human video is provided, comprising: obtaining a target background image based on a target user's avatar image and a target scene; obtaining a target background video including the target scene and the target user based on the target background image; obtaining target digital human parameters corresponding to the target scene and the target user based on the target background video; obtaining a target template video including the target scene and the target digital human based on target audio parameters and the target digital human parameters; and, in response to the target template video passing review, obtaining a target digital human video based on the video parameters and target text of the target template video.

[0004] According to a second aspect of this disclosure, an apparatus for generating digital human videos is provided, comprising: an image acquisition unit, configured to obtain a target background image based on a target user's avatar image and a target scene; a video acquisition unit, configured to obtain a target background video including the target scene and the target user based on the target background image; a parameter acquisition unit, configured to obtain target digital human parameters corresponding to the target scene and the target user based on the target background video; a conversion unit, configured to obtain a target template video including the target scene and the target digital human based on target audio parameters and the target digital human parameters; and a generation unit, configured to, in response to the target template video passing review, obtain a target digital human video based on the video parameters and target text of the target template video.

[0005] According to a third aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0006] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method described above.

[0007] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described above.

[0008] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0010] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure;

[0011] Figure 2 This is a schematic diagram according to the second embodiment of the present disclosure;

[0012] Figure 3 This is a schematic diagram according to the third embodiment of the present disclosure;

[0013] Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure;

[0014] Figure 5 This is a schematic diagram according to the fifth embodiment of the present disclosure;

[0015] Figure 6 This is a block diagram of an electronic device used to implement the digital human video generation method of the embodiments of this disclosure. Detailed Implementation

[0016] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and mechanisms are omitted in the following description.

[0017] Figure 1 This is a schematic diagram based on the first embodiment of this disclosure. (See diagram below.) Figure 1 As shown, the method for generating digital human videos in this embodiment specifically includes the following steps:

[0018] S101. Obtain the target background image based on the target user's profile picture and the target scene;

[0019] S102. Based on the target base image, obtain a target base video including the target scene and the target user;

[0020] S103. Based on the target base video, obtain the target digital human parameters corresponding to the target scene and the target user;

[0021] S104. Based on the target sound parameters and the target digital human parameters, obtain a target template video including the target scene and the target digital human;

[0022] S105. In response to the target template video passing the review, obtain the target digital human video based on the video parameters and target text of the target template video.

[0023] The digital human video generation method of this embodiment breaks down the digital human video generation process into five stages: "acquiring the target background image", "acquiring the target background video", "acquiring the target digital human parameters", "acquiring the target template video", and "generating the target digital human video". On the one hand, this decouples the digital human video generation process, thereby supporting the ability to independently process any stage of the digital human video generation process (e.g., independent storage, independent retries, etc.), which can improve the flexibility of digital human video generation. On the other hand, by only performing multiple transformations on the acquired target user's avatar image, a target digital human video corresponding to the target scene, target user, and target text can be generated, which can reduce the generation cost of digital human videos and improve the generation efficiency of digital human videos.

[0024] In this embodiment, during the execution of S101, the avatar image of the target user can be acquired through real-time shooting, or an existing image input or selected by the target user can be acquired as the avatar image. This embodiment acquires the avatar image only with the user's consent or authorization to ensure the legality of the acquisition. Furthermore, after acquiring the avatar image, the user can be informed that the acquired avatar image will be used to generate the user's digital human. With the user's consent or authorization, subsequent digital human generation, digital human video generation, etc., can then be performed based on the acquired avatar image.

[0025] In this embodiment, after executing S101 to obtain the target user's avatar image, the obtained avatar image can also be quality checked. Then, if the avatar image passes the quality check, the subsequent background image can be generated.

[0026] In this embodiment, S101 performs quality verification on the avatar image, including format verification (verifying whether the image format of the avatar image is a preset image format), resolution verification (verifying whether the resolution of the avatar image meets the preset resolution requirements), and face verification (verifying whether there is a face or multiple faces in the avatar image).

[0027] In this embodiment, when executing S101, in response to the fact that the image format of the avatar image is a preset image format, the resolution of the avatar image meets the preset resolution requirements, and the avatar image contains only one face, it is determined that the avatar image passes the quality check.

[0028] In this embodiment, when selecting a target scene in step S101, the following implementation method can be adopted: obtain the target quantity and target domain. In this embodiment, the target quantity is used to represent the number of background images to be generated, and the target domain is one of the following: legal domain, e-commerce domain, education domain, and medical domain; select the scene with the corresponding target quantity from the scene library of the corresponding target domain as the target scene.

[0029] In other words, this embodiment combines the two types of information, the number of targets and the target domain, to determine the target scene, which can improve the accuracy of the target scene determination. Furthermore, when there are multiple targets, this embodiment can select multiple target scenes, enabling the generation of base images corresponding to different scenes. Then, different digital human videos can be generated based on the base images corresponding to different scenes, thereby improving the diversity of digital human video generation.

[0030] In this embodiment, a scenario library corresponding to different fields is pre-set. Each scenario library includes multiple scenarios corresponding to different fields. For example, in the scenario library corresponding to the legal field, it may include scenarios such as lawyer's office, conference room, and legal document room.

[0031] In this embodiment, after completing the acquisition of the avatar image and the selection of the target scene in S101, the target background image can be obtained based on the avatar image and the target scene. The target background image obtained in this embodiment includes the target scene and the face of the target user, and may also include the torso of the target user.

[0032] In this embodiment, when executing S101, the first multimodal large model can be called, and the avatar image and the description of the corresponding target scene can be input into the first multimodal large model. Then, the target base image can be obtained according to the output of the first multimodal large model. In this embodiment, a target base image will be generated for different target scenes.

[0033] After executing S101 to obtain the target background image generated by the first multimodal large model, this embodiment may further include the following: performing multi-dimensional detection on the target background image. In this embodiment, the multi-dimensional detection includes at least two of the following: facial feature integrity detection, image clarity detection, human proportion detection, and composition rationality detection; obtaining the quality score of the target background image based on the detection scores of different dimensions; and determining that the target background image passes the detection and is retained if the obtained quality score is greater than or equal to a preset score threshold.

[0034] In other words, this embodiment only retains the target base image after multi-dimensional detection to perform the subsequent generation process, thereby avoiding the problem of low generation quality of digital human videos caused by low generation quality of target base images at the source, and improving the generation quality of digital human videos.

[0035] In this embodiment, when performing S101 to detect the integrity of facial features in the target background image, it can detect whether the facial features of the face in the target background image are complete. If the facial features are complete, the detection score is 1; otherwise, it is 0. When performing image sharpness detection on the target background image, the Laplacian variance method can be used to calculate the sharpness of the target background image, and the calculated sharpness is used as the detection score. When performing person proportion detection on the target background image, the ratio of the area of ​​the person in the target background image to the area of ​​the image can be calculated as the detection score. When performing composition rationality detection on the target background image, the offset between the center of the face in the target background image and the center of the image can be calculated as the detection score.

[0036] In this embodiment, when executing S101, the weight values ​​corresponding to different detection dimensions can also be obtained, and then the quality score of the target substrate image can be obtained based on the weight values ​​of different detection dimensions and the detection score.

[0037] It is understood that when executing S101, this embodiment may also include the following: in response to the obtained quality score being less than a preset score threshold, proceed to the step of obtaining the target background image based on the avatar image and the target scene, and repeat this process until it is determined that the quality score of the target background image is greater than or equal to the preset score threshold or the preset number of retries is reached.

[0038] In other words, this embodiment can regenerate low-quality base plate images through a retry mechanism, avoiding the problem that subsequent generation processes cannot be executed because the base plate images fail multi-dimensional detection. Furthermore, this embodiment will use a preset number of retries to constrain the retry process, avoiding the problem of wasting computing resources caused by infinitely generating base plate images.

[0039] If the target base plate image obtained by executing S101 in this embodiment still fails the multi-dimensional detection after reaching the preset number of retries, manual intervention can be carried out. After manually modifying the prompt text used when generating the base plate image, the base plate image generation process is triggered again, and then the generated base plate image is reviewed by a human.

[0040] It is understandable that if the target user has previously generated a digital human video, this embodiment can also use the corresponding background image obtained from the background image library as the target background image when executing S101, based on the target user's avatar image and the selected target scene; wherein, the background image library records the correspondence between user information (including avatar image and scene) and background images.

[0041] In other words, this embodiment stores the base image generated by the first multimodal large model in the base image library so that when the same user generates a digital human video again, the target base image can be directly obtained from the base image library. The direct acquisition of the target base image is also achieved by decoupling the digital human video generation process in this embodiment.

[0042] In this embodiment, after obtaining the target base image by executing S101, S102 is executed to obtain the target base video including the target scene and the target user based on the target base image.

[0043] The target base video obtained by executing S102 in this embodiment is specifically a video of the target user making natural micro-movements (such as head turning, blinking, etc.) in the target scene. The target base video can be a video with sound or a video without sound.

[0044] In this embodiment, when executing S102, the second multimodal large model can be called, the obtained target base plate image can be input into the second multimodal large model, and then the target base plate video converted from the target base plate image can be obtained according to the output result of the second multimodal large model.

[0045] It is understandable that if the target user has previously generated a digital human video, this embodiment can also use the corresponding base video obtained from the base video library as the target base video when executing S102; wherein, the base video library records the correspondence between base images and base videos.

[0046] In other words, this embodiment stores the base video generated by the second multimodal large model in the base video library so that when the same user generates a digital human video again, the target base video can be directly obtained from the base video library; and the direct acquisition of the target base video is also achieved by decoupling the digital human video generation process in this embodiment.

[0047] It is understood that in this embodiment, if there are multiple target base plate images when executing S102, this embodiment will obtain different target base plate videos according to different target base plate images.

[0048] In this embodiment, after executing S102 to obtain the target base video including the target scene and the target user, S103 is executed to obtain the target digital human parameters corresponding to the target scene and the target user based on the target base video.

[0049] The target digital human parameters obtained by executing S103 in this embodiment include the image parameters of the target digital human itself and the scene parameters in which the target digital human is located; wherein, the image parameters may include the face parameters, clothing parameters, posture parameters, etc. of the target digital human, and the scene parameters may include the background parameters, decoration parameters, etc. of the scene in which the target digital human is located.

[0050] In this embodiment, the target digital person is the digital person corresponding to the target user; that is, in this embodiment, through S103, the purpose of image cloning is achieved based on the target base video, thereby obtaining the relevant parameters of the target digital person corresponding to the target user.

[0051] It is understood that, in generating the target digital human, this embodiment can further determine whether the user's consent or authorization has been obtained. If it is determined that the generation of the digital human has been consented to or authorized by the user, the image is cloned based on the obtained target base video to generate the user's digital human.

[0052] In this embodiment, when executing S103, the third multimodal large model can be called, the target base plate video can be input into the third multimodal large model, and then the target digital human parameters corresponding to the target scene and target user can be obtained according to the output results of the multimodal large model.

[0053] It is understandable that if the target user has previously generated a digital human video, this embodiment can also use the corresponding digital human parameters obtained from the digital human parameter library as the target digital human parameters when executing S103, based on the target base video; wherein, the digital human parameter library records the correspondence between the base video and the digital human parameters.

[0054] In other words, this embodiment stores the digital human parameters generated by the third multimodal large model in the digital human parameter library so that when the same user generates a digital human video again, the target digital human parameters can be directly obtained from the digital human parameter library; and the direct acquisition of the target digital human parameters is also achieved by decoupling the digital human video generation process in this embodiment.

[0055] In this embodiment, after executing S103 to obtain the target digital human parameters corresponding to the target scene and target user, S104 is executed to obtain the target template video including the target scene and target digital human based on the target sound parameters and target digital human parameters.

[0056] The target template video obtained by executing S104 in this embodiment includes "the speech behavior of the target digital person corresponding to the target user in the target scene with the corresponding voice style".

[0057] In this embodiment, when executing S104, an audio file sent by the target user can be received, and the sound parameters obtained by parsing the audio file (in this embodiment, a third multimodal large model can be used for parsing) can be used as the target sound parameters.

[0058] In this embodiment, if no audio file sent by the target user is received during S104, the default audio parameters corresponding to the target user's attribute information are obtained from the preset audio parameter library and used as the target audio parameters. The target audio parameters can be the default audio parameters corresponding to the user's gender, the default audio parameters corresponding to the user's age, or the default audio parameters corresponding to the user's occupation.

[0059] In the target template video obtained by executing S104 in this embodiment, the speech content of the target digital human can be preset content, which can be "Your digital human template video has been generated and you can proceed with digital human video production".

[0060] In this embodiment, when executing S104, the third multimodal large model can be called, and the obtained target sound parameters and target digital human parameters can be input into the third multimodal large model. Then, based on the output of the third multimodal large model, a target template video including the target scene and the target digital human can be obtained.

[0061] It is understandable that if the target user has previously generated a digital human video, this embodiment can also use the corresponding template video obtained from the template video library as the target template video when executing S104, based on the target digital human parameters and the target voice parameters; wherein, the template video library records the correspondence between the target parameters (including digital human parameters and voice parameters) and the template video.

[0062] In other words, this embodiment stores the template video generated by the third multimodal large model in the template video library so that when the same user generates a digital human video again, the target template video can be directly obtained from the template video library; and the direct acquisition of the target template video is also achieved by decoupling the digital human video generation process in this embodiment.

[0063] In this embodiment, after obtaining the target template video by executing S104, the obtained target template video can be sent to the corresponding reviewers, who will then conduct a manual review and return the manual review results.

[0064] In this embodiment, after executing S104 to obtain a target template video including the target scene and the target digital human, S105 is executed in response to the target template video passing the review, and the target digital human video is obtained according to the video parameters and target text of the target template video.

[0065] In this embodiment, when executing S105, if the manual review result of the target template video is determined to be passed, the target template video can be determined to have passed the review.

[0066] If the target template video fails the review, this embodiment can also proceed to S104 to regenerate the target template video based on the target digital human parameters and the target voice parameters, until it is determined that the generated target template video has passed the review or the preset number of retries has been reached.

[0067] In this embodiment, when executing S105, the text input or selected by the target user can be obtained as the target text, which is the speech content of the target digital human in the generated target digital human video.

[0068] In this embodiment, when executing S105, the target text can also be obtained in the following way: determine the target knowledge base according to the target domain, different domains correspond to different knowledge bases, and the knowledge base includes professional knowledge of the corresponding domain; use the knowledge text obtained from the target knowledge base as the target text.

[0069] In addition, when executing S105, this embodiment can also obtain user information of the target user, such as the target user's name and workplace, and then display the obtained user information in the generated target digital human video in the form of text.

[0070] In this embodiment, when executing S105 to obtain the target digital human video based on the video parameters and target text of the target template video, the implementation method can be as follows: generate candidate digital human videos based on the video parameters and target text of the target template video. In this embodiment, a pre-trained video generation model can be used to generate candidate digital human videos; calculate the similarity between the face of the digital human in the candidate digital human video and the face in the avatar image; in response to the similarity calculation result being greater than or equal to a preset similarity threshold, use the candidate digital human video as the target digital human video.

[0071] In other words, this embodiment can also calculate the similarity between the digital human face in the candidate digital human video and the target user's face, and then determine whether the digital human in the obtained candidate digital human video is close to the target user based on the similarity calculation result. Only when similarity is determined will the generated candidate digital human video be used as the target digital human video, which can improve the accuracy of the obtained target digital human video.

[0072] In this embodiment, when executing S105, multiple target texts can be obtained. For example, if the target field is the legal field, relevant legal cases and legal knowledge can be obtained as target texts. Then, different target texts are combined with different video parameters to obtain multiple combinations. Based on the multiple combination results, multiple candidate digital human videos are generated. This embodiment can achieve the purpose of mass production of digital human videos.

[0073] It is understood that in this embodiment, when executing S105, the calculation of face similarity may not be performed, and the generated candidate digital human video may be directly used as the target digital human video.

[0074] In addition, after obtaining the target digital human video by executing S105, this embodiment can also perform a digital human video claiming operation.

[0075] The claiming process includes automatic claiming and manual claiming. If the target user has authorized automatic claiming, this embodiment can establish a correspondence between the target digital human and the target user after executing S105 to obtain the target digital human video. If the target user has not authorized automatic claiming, this embodiment can send the target digital human video to the target user after executing S105 to obtain the target digital human video, and then establish a correspondence between the target digital human and the target user after the target user confirms it.

[0076] In this embodiment, after the target digital human video is obtained by executing S105 and claimed by the target user, the obtained target digital human video can be published to the corresponding content platform, thereby automating the generation and publication of digital human videos.

[0077] Figure 2This is a schematic diagram according to the second embodiment of this disclosure. (See diagram below.) Figure 2 As shown, when executing S101 "obtain the target background image based on the target user's avatar image and the target scene" in this embodiment, the following may be included:

[0078] S201. Perform attribute recognition on the portrait image to obtain facial attribute information;

[0079] S202. Based on the facial attribute information and the target scene, obtain the target prompt text;

[0080] S203. Generate the target background image based on the avatar image and the target prompt text.

[0081] In other words, this embodiment can also obtain target prompt text for generating target background images based on the facial attribute information obtained by attribute recognition of avatar images, and then use the obtained target prompt text to generate target background images, which can improve the accuracy of target background image generation.

[0082] The facial attribute information obtained by executing S201 in this embodiment may include gender information, age information, etc.

[0083] In this embodiment, when executing S202 to obtain the target prompt text based on the face attribute information and the target scene, the following implementation method can be adopted: obtain the first prompt text corresponding to the face attribute information and the second prompt text corresponding to the target scene; concatenate the obtained first prompt text and the second prompt text, and use the concatenated result as the target prompt text.

[0084] The first prompt text obtained by executing S202 in this embodiment may include information such as clothing requirements and person's posture corresponding to the facial attribute information; the second prompt text obtained by executing S202 in this embodiment may include descriptive information of the target scene, such as background and decoration.

[0085] In this embodiment, when executing S202, a third prompt text can also be obtained. The third prompt text includes information such as the proportion of the person, image quality, and skin color. Then, the concatenation result of the first prompt text, the second prompt text, and the third prompt text is used as the target prompt text.

[0086] In this embodiment, when executing S203, the avatar image and target prompt text can be input into the multimodal large model, and then the target base image can be obtained based on the output of the multimodal large model.

[0087] In this embodiment, when executing S203, if there are multiple target scenes, different target prompt texts can be obtained based on the first prompt text corresponding to the face attribute information and the second prompt text corresponding to different target scenes. Then, different target prompt texts can be used to generate target background images corresponding to different scenes.

[0088] Figure 3 This is a schematic diagram according to the third embodiment of this disclosure. (See diagram below.) Figure 3 As shown, this embodiment illustrates a six-stage decoupled flowchart for the digital human video generation process; in this embodiment, the acquisition of target sound parameters is also considered as a processing stage:

[0089] After receiving the avatar image uploaded by the target user, the task scheduling center triggers the first stage, which is the generation of the target background image.

[0090] After receiving the target base image generated in the first stage, the task scheduling center triggers the second stage, namely the generation of the target base video.

[0091] After receiving the target baseboard video generated in the second stage, the task scheduling center triggers the third stage, namely the generation of target sound parameters (i.e., sound cloning).

[0092] After receiving the target voice parameters generated in the third stage, the task scheduling center triggers the fourth stage, namely the generation of the target digital human parameters (i.e., image cloning).

[0093] After receiving the target digital human parameters generated in the fourth stage, the task scheduling center triggers the fifth stage, namely the generation of the target template video.

[0094] After receiving the target template video generated in the fifth stage, the task scheduling center triggers the review of the target template video. If the target template video passes the review, the sixth stage, namely the generation of the target digital human video, is triggered.

[0095] In addition, after the task scheduling center completes the generation of the target digital human video, it can send the generated target digital human video to the target user, and then publish the target digital human video after confirming that the target user has claimed the rights.

[0096] It is understood that after completing the processing of each stage, this embodiment can also generate a stage identifier corresponding to each stage (different target users have different stage identifiers in the same stage), and then record the output results of the current stage according to the generated stage identifier; for example, the stage identifier corresponding to the first stage is stage 1 (if there are user 1 and user 2, the stage identifier can be user 1 stage 1, user 2 stage 2, etc.), and the data corresponding to stage 1 is "target base plate image set", the stage identifier corresponding to the second stage is stage 2, and the data corresponding to stage 2 is "target base plate video set", etc.

[0097] Therefore, this embodiment can re-execute the task or trace the output results of the upstream stage according to the stage identifier, so as to achieve the purpose of re-execution only from the stage where the execution failed after any stage fails, without having to execute the previous stage of the current stage.

[0098] Figure 4 This is a schematic diagram according to the fourth embodiment of this disclosure. (See diagram below.) Figure 4 As shown in the figure, this embodiment illustrates the structural diagram of the digital human video generation system; the digital human video generation system includes a base image generation module, a base video generation module, a sound cloning module, an image cloning module, a template video generation module, an approval module, and a digital human video generation module.

[0099] Among them, the base plate image generation module calls the first multimodal large model; the base plate video generation module calls the second multimodal large model; and the sound cloning module, image cloning module, and template video generation module call the third multimodal large model.

[0100] Since there are multiple target users generating digital human videos simultaneously, the number of parallel processing operations for different multimodal large models is limited. For example, the parallel processing volume of the second multimodal large model is 1, and the parallel processing volume of the third multimodal large model is 5. Therefore, this embodiment can also combine the "token bucket" and "user priority" methods to achieve concurrent control of different multimodal large models.

[0101] Specifically, this embodiment can distribute the remaining tokens in the token bucket to users with higher priority when the number of parallel processing operations of the current multimodal large model is less than the number of users. Then, after determining that the user has completed the execution of a certain stage of the task, the user's token is revoked and redistributed to the next user with the highest priority, so as to continue the execution and effectively manage the parallel processing of the multimodal large model.

[0102] In addition, this embodiment can also store data generated at different stages, such as storing the base plate image generated by the base plate image generation module in the base plate image library, and storing the base plate video generated by the base plate video generation module in the base plate video library, etc.

[0103] Figure 5 This is a schematic diagram according to the fifth embodiment of this disclosure. (See diagram below.) Figure 5 As shown, the digital human video generation apparatus 500 of this embodiment includes:

[0104] Image acquisition unit 501 is used to obtain a target background image based on the target user's avatar image and the target scene;

[0105] Video acquisition unit 502 is used to obtain a target base video including the target scene and the target user based on the target base image;

[0106] The parameter acquisition unit 503 is used to obtain target digital human parameters corresponding to the target scene and the target user based on the target base video;

[0107] The conversion unit 504 is used to obtain a target template video including the target scene and the target digital human based on the target sound parameters and the target digital human parameters;

[0108] The generation unit 505 is used to generate a target digital human video in response to the target template video passing the review, based on the video parameters and target text of the target template video.

[0109] Image acquisition unit 501 can acquire the avatar image of the target user through real-time shooting, or it can acquire an existing image input or selected by the target user as the avatar image.

[0110] After acquiring the avatar image of the target user, the image acquisition unit 501 can also perform quality verification on the acquired avatar image. Then, if the avatar image passes the quality verification, the subsequent background image will be generated.

[0111] The image acquisition unit 501 performs quality checks on the portrait images, including format checks, resolution checks, and face checks.

[0112] In response to the fact that the image format of the avatar image is a preset image format, the resolution of the avatar image meets the preset resolution requirements, and the avatar image contains only one face, the image acquisition unit 501 determines that the avatar image has passed the quality check.

[0113] When selecting a target scene, the image acquisition unit 501 can adopt the following implementation method: acquire the target quantity and target domain; select the scene corresponding to the target quantity from the scene library of the corresponding target domain as the target scene.

[0114] In other words, the image acquisition unit 501 combines the two types of information, namely the number of targets and the target domain, to determine the target scene, which can improve the accuracy of the target scene determination. Furthermore, when there are multiple targets, this embodiment can select multiple target scenes to generate base images corresponding to different scenes, and then generate different digital human videos based on the base images corresponding to different scenes, thereby improving the diversity of digital human video generation.

[0115] In this embodiment, a scenario library corresponding to different fields is pre-set. Each scenario library includes multiple scenarios corresponding to different fields. For example, in the scenario library corresponding to the legal field, it may include scenarios such as lawyer's office, conference room, and legal document room.

[0116] After acquiring the avatar image and selecting the target scene, the image acquisition unit 501 can obtain the target background image based on the avatar image and the target scene. The target background image obtained in this embodiment includes the target scene and the target user's face, and may also include the target user's torso.

[0117] The image acquisition unit 501 can call the first multimodal large model, and then input the portrait image and the description of the corresponding target scene into the first multimodal large model, and then obtain the target base image based on the output of the first multimodal large model; wherein, in this embodiment, a target base image will be generated for different target scenes.

[0118] After obtaining the target substrate image generated by the first multimodal large model, the image acquisition unit 501 may also include the following: performing multi-dimensional detection on the target substrate image; obtaining a quality score of the target substrate image based on the detection scores of different dimensions; and determining that the target substrate image passes the detection and retaining it in response to the obtained quality score being greater than or equal to a preset score threshold.

[0119] In other words, the image acquisition unit 501 only retains the target background image after multi-dimensional detection to perform the subsequent generation process, thereby avoiding the problem of low generation quality of digital human video caused by low generation quality of target background image at the source, and improving the generation quality of digital human video.

[0120] The image acquisition unit 501 can also acquire the weight values ​​of different detection dimensions, and then obtain the quality score of the target substrate image based on the weight values ​​of different detection dimensions and the detection score.

[0121] It is understandable that the image acquisition unit 501 may also perform the following: in response to the obtained quality score being less than a preset score threshold, it shall proceed to the step of obtaining the target background image based on the avatar image and the target scene, and repeat this process until the quality score of the target background image is determined to be greater than or equal to the preset score threshold or the preset number of retries is reached.

[0122] In other words, the image acquisition unit 501 can regenerate the low-quality base plate image through the retry mechanism, avoiding the problem that the subsequent generation process cannot be executed because the base plate image fails the multi-dimensional detection. In addition, this embodiment will also use a preset number of retries to constrain the retry process, avoiding the problem of wasting computing resources caused by the infinite generation of the base plate image.

[0123] If the target base image obtained by the image acquisition unit 501 still fails the multi-dimensional detection after reaching the preset number of retries, manual intervention can be carried out. After manually modifying the prompt text used when generating the base image, the base image generation process can be triggered again, and then the generated base image can be reviewed by a human.

[0124] Understandably, if the target user has previously generated a digital human video, the image acquisition unit 501 can also use the corresponding background image obtained from the background image library as the target background image based on the target user's avatar image and the selected target scene; wherein, the background image library records the correspondence between user information (including avatar image and scene) and background images.

[0125] In other words, the image acquisition unit 501 stores the base image generated by the first multimodal large model in the base image library so that when the same user generates a digital human video again, the target base image can be directly obtained from the base image library. The direct acquisition of the target base image is also achieved by decoupling the digital human video generation process in this embodiment.

[0126] When obtaining the target background image based on the target user's avatar image and the target scene, the image acquisition unit 501 can also use the following methods: perform attribute recognition on the avatar image to obtain facial attribute information; obtain target prompt text based on the facial attribute information and the target scene; and generate the target background image based on the avatar image and the target prompt text.

[0127] In other words, the image acquisition unit 501 can also obtain target prompt text for generating target background images based on the facial attribute information obtained by attribute recognition of the portrait image, and then use the obtained target prompt text to generate target background images, which can improve the accuracy of target background image generation.

[0128] The facial attribute information obtained by the image acquisition unit 501 may include gender information, age information, etc.

[0129] When the image acquisition unit 501 obtains the target prompt text based on the facial attribute information and the target scene, the following implementation method can be adopted: obtain the first prompt text corresponding to the facial attribute information and the second prompt text corresponding to the target scene; concatenate the obtained first prompt text and the second prompt text, and use the concatenated result as the target prompt text.

[0130] The image acquisition unit 501 can also acquire a third prompt text, which includes information such as the proportion of the person, image quality, and skin color. The concatenation result of the first prompt text, the second prompt text, and the third prompt text is then used as the target prompt text.

[0131] The image acquisition unit 501 can input the avatar image and target prompt text into the multimodal large model, and then obtain the target base image based on the output of the multimodal large model.

[0132] If there are multiple target scenes, the image acquisition unit 501 can obtain different target prompt texts based on the first prompt text corresponding to the facial attribute information and the second prompt text corresponding to different target scenes, and then use the different target prompt texts to generate target background images corresponding to different scenes.

[0133] In this embodiment, after the image acquisition unit 501 obtains the target base image, the video acquisition unit 502 obtains the target base video, which includes the target scene and the target user, based on the target base image.

[0134] The target background video obtained by the video acquisition unit 502 is specifically a video of the target user making natural micro-movements (such as head turning, blinking, etc.) in the target scene. The target background video can be a video with sound or a video without sound.

[0135] The video acquisition unit 502 can call the second multimodal large model, input the obtained target base plate image into the second multimodal large model, and then obtain the target base plate video converted from the target base plate image based on the output result of the second multimodal large model.

[0136] Understandably, if the target user has previously generated digital human videos, the video acquisition unit 502 can also use the corresponding background video obtained from the background video library as the target background video based on the target background image; wherein, the background video library records the correspondence between background images and background videos.

[0137] In other words, the video acquisition unit 502 stores the base plate video generated by the second multimodal large model in the base plate video library so that when the same user generates a digital human video again, the target base plate video can be directly obtained from the base plate video library; and the direct acquisition of the target base plate video is also achieved by decoupling the digital human video generation process in this embodiment.

[0138] Understandably, if there are multiple target base plate images, the video acquisition unit 502 will obtain different target base plate videos based on the different target base plate images.

[0139] In this embodiment, after the video acquisition unit 502 obtains the target base video including the target scene and the target user, the parameter acquisition unit 503 obtains the target digital human parameters corresponding to the target scene and the target user based on the target base video.

[0140] The target digital human parameters obtained by the parameter acquisition unit 503 include the image parameters of the target digital human itself and the scene parameters of the target digital human; among them, the image parameters may include the face parameters, clothing parameters, posture parameters, etc. of the target digital human, and the scene parameters may include the background parameters, decoration parameters, etc. of the scene in which the target digital human is located.

[0141] In this embodiment, the target digital person is the digital person corresponding to the target user; that is, the parameter acquisition unit 503 achieves the purpose of image cloning based on the target base video, thereby obtaining the relevant parameters of the target digital person corresponding to the target user.

[0142] The parameter acquisition unit 503 can call the third multimodal large model, input the target base plate video into the third multimodal large model, and then obtain the target digital human parameters corresponding to the target scene and target user based on the output results of the multimodal large model.

[0143] Understandably, if the target user has previously generated a digital human video, the parameter acquisition unit 503 can also use the corresponding digital human parameters obtained from the digital human parameter library as the target digital human parameters based on the target base video; wherein, the digital human parameter library records the correspondence between the base video and the digital human parameters.

[0144] In other words, the parameter acquisition unit 503 stores the digital human parameters generated by the third multimodal large model into the digital human parameter library so that when the same user generates a digital human video again, the target digital human parameters can be directly obtained from the digital human parameter library. The direct acquisition of the target digital human parameters is also achieved by decoupling the digital human video generation process in this embodiment.

[0145] In this embodiment, after the parameter acquisition unit 503 obtains the target digital human parameters corresponding to the target scene and the target user, the conversion unit 504 obtains the target template video including the target scene and the target digital human parameters based on the target sound parameters and the target digital human parameters.

[0146] The target template video obtained by the conversion unit 504 includes "the speech behavior of the target digital human corresponding to the target user in the target scene with the corresponding voice style".

[0147] The conversion unit 504 can receive an audio file sent by the target user, and then use the sound parameters obtained by parsing the audio file (in this embodiment, a third multimodal large model can be used for parsing) as the target sound parameters.

[0148] If no audio file is received from the target user, the conversion unit 504 retrieves the default audio parameters corresponding to the target user's attribute information from the preset audio parameter library and uses them as target audio parameters. The target audio parameters can be the default audio parameters corresponding to the user's gender, the default audio parameters corresponding to the user's age, or the default audio parameters corresponding to the user's occupation.

[0149] In the target template video obtained by the conversion unit 504, the speech content of the target digital human can be preset content, which can be "Your digital human template video has been generated and you can proceed with digital human video production".

[0150] The conversion unit 504 can call the third multimodal large model, input the obtained target sound parameters and target digital human parameters into the third multimodal large model, and then obtain the target template video including the target scene and the target digital human based on the output of the third multimodal large model.

[0151] It is understandable that if the target user has previously generated a digital human video, the conversion unit 504 can also use the corresponding template video obtained from the template video library as the target template video based on the target digital human parameters and the target voice parameters; wherein, the template video library records the correspondence between the target parameters (including digital human parameters and voice parameters) and the template videos.

[0152] In other words, the conversion unit 504 stores the template video generated by the third multimodal large model in the template video library so that when the same user generates a digital human video again, the target template video can be directly obtained from the template video library; and the direct acquisition of the target template video is also achieved by decoupling the digital human video generation process in this embodiment.

[0153] After obtaining the target template video, the conversion unit 504 can also send the obtained target template video to the corresponding reviewers, who will then conduct a manual review and return the manual review results.

[0154] In this embodiment, after the conversion unit 504 obtains the target template video including the target scene and the target digital human, the generation unit 505, in response to the target template video passing the review, obtains the target digital human video based on the video parameters and target text of the target template video.

[0155] The generation unit 505 can determine that the target template video has passed the review if the manual review result of the target template video is deemed to have passed the review.

[0156] If the target template video fails the review, the generation unit 505 can also transfer to the conversion unit 504 to regenerate the target template video based on the target digital human parameters and the target audio parameters, until the generated target template video passes the review or the preset number of retries is reached.

[0157] The generation unit 505 can acquire the text input or selected by the target user as the target text, which is the speech content of the target digital human in the generated target digital human video.

[0158] The generation unit 505 can also obtain the target text in the following ways: determine the target knowledge base based on the target domain; and use the knowledge text obtained from the target knowledge base as the target text.

[0159] In addition, the generation unit 505 can also obtain user information of the target user, such as the target user's name and workplace, and then display the obtained user information in text form in the generated target digital human video.

[0160] When generating the target digital human video based on the video parameters and target text of the target template video, the generation unit 505 can adopt the following implementation method: generate candidate digital human videos based on the video parameters and target text of the target template video; calculate the similarity between the face of the digital human in the candidate digital human video and the face in the avatar image; and, in response to the similarity calculation result being greater than or equal to a preset similarity threshold, use the candidate digital human video as the target digital human video.

[0161] In other words, the generation unit 505 can also calculate the similarity between the digital human face in the candidate digital human video and the target user's face, and then determine whether the digital human in the obtained candidate digital human video is close to the target user based on the similarity calculation result. Only when similarity is determined will the generated candidate digital human video be used as the target digital human video, which can improve the accuracy of the obtained target digital human video.

[0162] The generation unit 505 can acquire multiple target texts, combine different target texts with different video parameters to obtain multiple combinations, and then generate multiple candidate digital human videos based on the multiple combination results, thereby achieving the purpose of mass production of digital human videos.

[0163] Understandably, the generation unit 505 may also skip the facial similarity calculation and directly use the generated candidate digital human video as the target digital human video.

[0164] In addition, after obtaining the target digital human video, the generation unit 505 can also perform a digital human video claiming operation.

[0165] The claiming process includes automatic claiming and manual claiming. If the target user authorizes automatic claiming, the generation unit 505 can establish a correspondence between the target digital human and the target user after obtaining the target digital human video. If the target user does not authorize automatic claiming, the generation unit 505 can send the target digital human video to the target user after obtaining it, and then establish a correspondence between the target digital human and the target user after the target user confirms it.

[0166] After obtaining the target digital human video and having it claimed by the target user, the generation unit 505 can publish the obtained target digital human video to the corresponding content platform, thereby automating the generation and publication of the digital human video.

[0167] The acquisition, storage, and application of any type of information, such as user personal information, involved in the technical solutions disclosed herein comply with relevant laws and regulations and do not violate public order and good morals.

[0168] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0169] like Figure 6 The diagram shown is a block diagram of an electronic device for generating digital human video according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0170] like Figure 6As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded from storage unit 608 into random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0171] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of displays, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0172] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the method for generating digital human videos. For example, in some embodiments, the method for generating digital human videos may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608.

[0173] In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by computing unit 601, one or more steps of the digital human video generation method described above may be performed. Alternatively, in other embodiments, computing unit 601 may be configured to perform the digital human video generation method by any other suitable means (e.g., by means of firmware).

[0174] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0175] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable digital human video generation device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0176] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0177] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for showing information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0178] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0179] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0180] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0181] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for generating digital human videos, comprising: Based on the target user's profile picture and the target scene, obtain the target background image; Based on the target base image, a target base video including the target scene and the target user is obtained; Based on the target base video, the target digital human parameters corresponding to the target scene and the target user are obtained; Based on the target sound parameters and the target digital human parameters, a target template video including the target scene and the target digital human is obtained; In response to the target template video passing the review, the target digital human video is obtained based on the video parameters and target text of the target template video.

2. The method according to claim 1, wherein, The process of obtaining the target background image based on the target user's profile picture and the target scene includes: The facial image is subjected to attribute recognition to obtain facial attribute information; Based on the facial attribute information and the target scene, the target prompt text is obtained; The target background image is generated based on the avatar image and the target prompt text.

3. The method according to claim 2, wherein, The step of obtaining the target prompt text based on the facial attribute information and the target scene includes: Obtain a first prompt text corresponding to the facial attribute information and a second prompt text corresponding to the target scene; The first prompt text and the second prompt text are concatenated, and the concatenated result is used as the target prompt text.

4. The method according to claim 1, wherein, Determining the target scenario includes: Obtain the target quantity and target area; From the scene library corresponding to the target domain, select scenes corresponding to the target number as the target scenes.

5. The method according to claim 1, further comprising: After obtaining the target base image, multi-dimensional detection is performed on the target base image; The quality score of the target substrate image is obtained based on the detection scores of different dimensions. In response to the quality score being greater than or equal to a preset score threshold, the target base image is determined to have passed the detection and is retained.

6. The method according to claim 1, wherein, Obtaining the target text includes: Determine the target knowledge base based on the target domain; The knowledge text obtained from the target knowledge base will be used as the target text.

7. The method according to claim 1, wherein, The process of obtaining the target digital human video based on the video parameters and target text of the target template video includes: Based on the video parameters and target text of the target template video, candidate digital human videos are generated; Calculate the similarity between the face of the digital human in the candidate digital human video and the face in the avatar image; If the similarity calculation result is greater than or equal to a preset similarity threshold, the candidate digital human video is selected as the target digital human video.

8. The method according to claim 1, wherein, The step of generating candidate digital human videos based on the video parameters and target text of the target template video includes: By combining the video parameters of different target template videos with different target texts, multiple combined results can be obtained; Based on the multiple combination results, multiple candidate digital human videos are generated.

9. An apparatus for generating digital human videos, comprising: The image acquisition unit is used to obtain the target background image based on the target user's avatar image and the target scene; The video acquisition unit is used to obtain a target background video including the target scene and the target user based on the target background image; The parameter acquisition unit is used to obtain target digital human parameters corresponding to the target scene and the target user based on the target base video; The conversion unit is used to obtain a target template video including the target scene and the target digital human based on the target sound parameters and the target digital human parameters; The generation unit is configured to, in response to the target template video passing the review, obtain the target digital human video based on the video parameters and target text of the target template video.

10. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

11. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

12. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.