Video processing method and device, electronic equipment and storage medium

Abstract

The invention provides a video processing method and device, electronic equipment and a storage medium, and relates to the technical field of data processing. The implementation scheme is as follows: acquiring an input video and a preview audio for digital human model training; generating a digital human model based on the input video; applying the preview audio to a digital human model to generate a plurality of preview videos, the digital human images in the plurality of preview videos corresponding to the user images in the input video, and the audio of the plurality of preview videos being the same as the preview audio; obtaining a material used for generating a target video and selection of at least one preview video in the plurality of preview videos; and generating a target video based on the material and the selected preview video, the target video having an audio corresponding to the material.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a video processing method, a video processing device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Traditional voice-over videos require real people to appear on screen, and have certain requirements on the real person's appearance, body movements, facial expressions, wording, and intonation of the reading of the text. They are also directly related to the shooting and recording tools, external environment, etc.

[0003] The methods described in this section are not necessarily methods that have been previously conceived or used. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0004] The present disclosure provides a data processing method, a video processing device, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to one aspect of the present disclosure, a video processing method is provided, comprising: obtaining an input video and preview audio for training a digital human model; generating a digital human model based on the input video; applying the preview audio to the digital human model to generate a plurality of preview videos, wherein the digital human image in the plurality of preview videos corresponds to the user image in the input video, and the audio of the plurality of preview videos is the same as the preview audio; obtaining material for generating a target video and selecting at least one preview video from the plurality of preview videos; and generating a target video based on the material and the selected preview video, wherein the target video has audio corresponding to the material.

[0006] According to another aspect of the present disclosure, a video processing device is also provided, including: a first input unit, configured to obtain an input video and preview audio for digital human model training; a model generation unit, configured to generate a digital human model based on the input video; a preview unit, configured to apply the preview audio to the digital human model to generate a plurality of preview videos, wherein the digital human image in the plurality of preview videos corresponds to the user image in the input video, and the audio of the plurality of preview videos is the same as the preview audio; a second input unit, configured to obtain material for generating a target video and a selection of at least one preview video from the plurality of preview videos; and an output unit, configured to generate a target video based on the material and the selected preview video, wherein the target video has audio corresponding to the material.

[0007] According to another aspect of the present disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, and the computer program implements the above method when executed by the at least one processor.

[0008] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing a computer program is also provided, wherein the computer program implements the above method when executed by a processor.

[0009] According to another aspect of the present disclosure, a computer program product is further provided, comprising a computer program, wherein the computer program implements the above method when executed by a processor.

[0010] By using the embodiments provided by the present disclosure, multiple preview videos generated after the digital human model training is completed can be previewed, and a target video can be generated based on at least one preview video selected from the multiple preview videos. In this way, the user can select at least one preview video from the multiple preview videos as the basis for reasoning according to the actual situation or personal preference, thereby generating the required digital human video.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0013] Figure 1 An exemplary flow chart of a video processing method according to an embodiment of the present disclosure is shown;

[0014] Figure 2 A schematic diagram showing a method for selecting multiple video clips according to an embodiment of the present disclosure;

[0015] Figure 3 An exemplary specific flow chart of a video processing method according to an embodiment of the present disclosure is shown;

[0016] Figure 4 An exemplary block diagram showing a video processing device according to an embodiment of the present disclosure; and

[0017] Figure 5A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0018] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0019] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0020] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0021] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0022] Digital Human / Meta Human refers to a virtual character or image generated by computer technology that can simulate human appearance, behavior and language. Digital Humans usually use technologies such as 3D modeling, animation, artificial intelligence (AI) and natural language processing (NLP) to interact with users. The emergence of digital humans has broken the strong dependence of video works on users and the surrounding environment. Users can shoot video materials of a certain length to train digital human models. After the digital human model is trained, video content close to real people can be inferred based on the text material content that the user wants to infer.

[0023] In related technologies, videos uploaded by users for digital human model training can often only train a digital human model that generates a single digital human image.

[0024] In order to allow users to select a suitable digital human image from multiple digital human images for digital human video synthesis, the present disclosure provides a new video processing method.

[0025] Figure 1 An exemplary flow chart of a video processing method according to an embodiment of the present disclosure is shown.

[0026] In step S102, an input video and a preview audio for digital human model training are obtained.

[0027] In step S104, a digital human model is generated based on the input video.

[0028] In step S106, the preview audio is applied to the digital human model to generate a plurality of preview videos, wherein the digital human images in the plurality of preview videos correspond to the user images in the input video, and the audios of the plurality of preview videos are the same as the preview audio.

[0029] In step S108, materials for generating a target video and selection of at least one preview video from among a plurality of preview videos are obtained.

[0030] In step S110 , a target video is generated based on the material and the selected preview video, wherein the target video has audio corresponding to the material.

[0031] By using the video processing method provided by the embodiment of the present disclosure, multiple preview videos generated after the digital human model training is completed can be previewed, and a target video can be generated based on at least one preview video selected from the multiple preview videos. In this way, the user can select at least one preview video from the multiple preview videos as the basis for reasoning according to the actual situation or personal preference, thereby generating the required digital human video.

[0032] The principles of the present disclosure will be described in detail below.

[0033] In step S102, the input video and preview audio for digital human model training are obtained. The user can upload the personal oral video and preview audio for digital human model training to the server via the network.

[0034] In step S104, a digital human model is generated based on the input video.

[0035] In some embodiments, step S104 may include: determining a plurality of video clips from the input video, wherein each video clip has a corresponding user image; and generating a digital human model based on the plurality of video clips.

[0036] In some examples, the video uploaded by the user may include multiple user images at different time periods. For example, from 0 to 10 minutes of the input video, the user image is a doctor; from 10 to 20 minutes of the input video, the user image is a teacher, then the input video uploaded by the user is determined as two video clips, wherein the user image in the first video clip is a doctor, and the user image in the second video clip is a teacher, and a digital human model is generated based on the above two video clips.

[0037] In some embodiments, step S104 may include: determining multiple video segments from the input video based on a video review standard, wherein the video review standard includes a video quality standard and a video content standard, and the determined multiple video segments simultaneously meet the video quality standard and the video content standard.

[0038] In some examples, the video quality criteria may include one or more of the following:

[0039] - Check the proportion of the face in the picture, do not be too close to the camera or too far away from the camera

[0040] - Pay attention to the light changes on the face in the clip, and avoid obvious swallowing of saliva

[0041] -Avoid long-distance movement when adjusting standing or sitting posture, as well as large body and head shaking

[0042] - The character's eyes need to look directly at the camera. They can blink, but they cannot glance up, down, left, or right.

[0043] - Do not close your eyes, shake your body, or use hand gestures as the start and end moments of the template

[0044] - The character does not have frequent gestures, ensuring that the user's gesture frequency remains consistent

[0045] - The video in this action sequence has a high frequency of editing or pauses to avoid cuts

[0046] -Avoid having too much noise in the video that is louder than the human voice, or problems with the sound reception throughout the video.

[0047] In some examples, video content standards may include confirming that the video content does not involve illegal, unlawful, or other content that is inappropriate for dissemination in a video.

[0048] In some examples, the input video can be screened, for example, by machine review of text and images, to determine multiple video clips that meet both the video quality standard and the video content standard for use in generating a digital human model. Machine review of text and images (also known as automatic text and image review) is a technology that uses artificial intelligence and machine learning technology to automatically review image and text content. It is widely used in content review, social media, advertising review and other fields to ensure that the published content complies with relevant laws, regulations and platform regulations. Compared with manual review, it can reduce review costs and improve review efficiency.

[0049] Screening videos uploaded by users through video review standards can not only improve the quality of the videos used for digital human training, thereby improving the quality of the generated digital human model, but also eliminate malicious videos uploaded by users.

[0050] In some embodiments, step S104 may include: generating a digital human model based on the input video using small sample learning. Small sample learning is a machine learning method that allows the model to learn new concepts or tasks through very few training samples. Unlike traditional machine learning, which requires a large amount of labeled data, small sample learning can achieve effective learning and prediction when data is scarce. In some examples, users only need to upload a personal oral video of more than ten minutes to half an hour to train an effective digital human model.

[0051] In step S106, the preview audio is applied to the digital human model to generate a plurality of preview videos, wherein the digital human images in the plurality of preview videos correspond to the user images in the input video, and the audios of the plurality of preview videos are the same as the preview audio.

[0052] In some examples, the preview audio is applied to a trained digital human model, and the digital human model can perform inference based on the preview audio to generate multiple preview videos with the same audio as the preview audio.

[0053] In some embodiments, the digital human image in each of the multiple preview videos corresponds to the user image in the multiple video clips.

[0054] In some examples, the user images in multiple (e.g., two) video clips are respectively a doctor and a teacher, and the digital human image in each of the multiple (e.g., two) preview videos is respectively a doctor and a teacher corresponding to the user images in the multiple video clips.

[0055] In some embodiments, the user image and the digital human image include lip movements, facial expressions, and body movements.

[0056] In step S108, materials for generating a target video and selection of at least one preview video from among a plurality of preview videos are obtained.

[0057] In some examples, a user may upload material for inference. A user may preview at least one of a plurality of preview videos, and then may (e.g., based on actual needs or personal preferences of the user) select at least one of the plurality of preview videos as a template video for inference.

[0058] In some embodiments, the material includes at least one of audio and text. In some examples, the user can upload audio, text, or a combination of audio and text as the material for reasoning.

[0059] In step S110 , a target video is generated based on the material and the selected preview video, wherein the target video has audio corresponding to the material.

[0060] In some examples, if the material uploaded by the user includes audio, the uploaded audio is used to perform digital human audio-to-video inference on the selected preview video (this process is also called image-driven, which is a method of combining a digital human model trained with video samples and timbre information to simulate lip shape, facial expressions, human voice, body movements, etc., to infer the complete video content) to generate a digital human video, the audio of which is the uploaded audio.

[0061] In some examples, if the material uploaded by the user includes text, the text is converted into audio (for example, through TTS (Text to Speech) technology), and the generated audio is used to perform digital human audio-to-video inference on the selected preview video to generate a digital human video, the audio of which is the audio generated by the text conversion. TTS (Text-to-Speech) technology is a technology that converts written text into audible speech, generating natural and fluent speech output by analyzing the text content.

[0062] In some embodiments, step S110 includes: determining whether the material meets the material review standard; and in response to the material meeting the material review standard, generating a target video based on the material and the selected preview video.

[0063] In some examples, image and text machine review can be used to determine whether the material uploaded by the user meets the material review standards.

[0064] In some embodiments, in response to the material not satisfying the material review criteria, generating the target video based on the material and the selected preview video is abandoned.

[0065] In some examples, in response to determining (for example, using image and text machine review) that the material does not meet the material review standards, the generation of the target video based on the material and the selected preview video is abandoned, and the user may also be notified that the uploaded material does not meet the material review standards.

[0066] In some embodiments, the material review criteria include voiceprint review criteria for audio and text content review criteria for text.

[0067] In some examples, text content review criteria for text include determining (e.g., through text and image machine review) that text uploaded by a user involves content that is illegal, unlawful, or otherwise unsuitable for public dissemination.

[0068] Voiceprint verification is a biometric technology that verifies identity by analyzing and identifying individual voice characteristics. Voiceprints are similar to fingerprints. Each person's voice has a unique frequency, pitch, and pronunciation method, which can be used to confirm identity. In some examples, the voiceprint review criteria for audio include determining (for example, through voiceprint verification) that the voiceprint of the audio material uploaded by the user is the same as the voiceprint of the input video previously uploaded by the user for digital human model training, thereby preventing users from stealing other people's voices and causing infringement or other risks.

[0069] As mentioned above, using the video processing method described above, multiple preview videos generated after the digital human model training is completed can be previewed, and a target video can be generated based on at least one preview video selected from the multiple preview videos. In this way, the user can select at least one preview video from the multiple preview videos as the basis for reasoning according to actual conditions or personal preferences, thereby generating the required digital human video.

[0070] Using the above method, users only need to upload a short personal video for digital human model training, and can preview multiple preview videos generated by the digital human model. After the digital human model training is completed, users can upload materials and select preview videos as the basis for material reasoning based on actual conditions or personal preferences, thereby generating the digital human video the user wants. It can also prevent users from stealing other people's voices and uploading malicious videos for digital human models or malicious materials for reasoning.

[0071] Figure 2 A schematic diagram of selecting multiple video clips in a video processing method according to an embodiment of the present disclosure is shown.

[0072] like Figure 2 FIG. 2 is a schematic diagram showing a method of selecting multiple video clips from a video sequence 200 according to an embodiment of the present disclosure. Figure 2 In the example shown, the entire video sequence includes N frames, and the relevant features of the person in each frame (lip shape, body movements, facial expressions, etc.) are extracted, and the 2nd and 3rd frames are selected from the N frames as segment 1, and the 5th and 6th frames are selected as segment 2.

[0073] Figure 3An exemplary specific flow chart of a video processing method 300 according to an embodiment of the present disclosure is shown.

[0074] The exemplary specific process of the video processing method 300 may include the following steps:

[0075] Step 301: The user submits video material and generates audio material of a preview video to a creative gateway (creative-interface);

[0076] Step 302, the creation gateway stores the material in the library and sends it to the creation service (bcut-interface);

[0077] Step 303: The creative service transcodes the material video into different video resolutions, and the auditor or machine can subsequently select different resolutions for university review;

[0078] Step 304, the creative business is reviewed and trained;

[0079] Step 305: The Studio background reviews the material, reviews the filmed material frame by frame, and checks whether there are any unqualified films. The clips that can be used for training and the excellent clips are selected and marked;

[0080] Step 306: The Studio backend sends the approved materials and annotation information to the algorithm engineering for digital human model training;

[0081] Step 307: Algorithm Engineering transmits the trained digital human model assets back to the Studio backend;

[0082] Step 308: The Studio backend performs manual or machine review on the trained digital human model;

[0083] Step 309: After the model is reviewed and approved, the Studio backend sends the trained assets of the digital human model (which may include a preview video inferred from the audio material) to the user, for example, via SMS or a web page pop-up notification;

[0084] Step 310: The user uploads the inference text to the creation gateway. In some examples, the user may also upload the inference audio or a combination of text and audio.

[0085] Step 311, the creation gateway can save the user's drafts and works (copywriting, recordings, background images and background music) to the creation service at any time, so that the user can see the previous draft content the next time he enters the website or application due to abnormal closure;

[0086] Step 312: The creation service stores the user's draft and work in a database, such as MySQL;

[0087] Step 313: The creation service creates an inference task and sends it to the AI ​​gateway (aikit);

[0088] Step 314: The AI ​​gateway sends the inference task to the algorithm engineering for inference execution. The algorithm engineering relies on text and image machine review, TTS, voiceprint verification, and image drive. When the inference material uploaded by the user is text, the TTS model needs to be called to store and convert the text into an audio file. Before the conversion, the text and image machine review is used to verify whether there is sensitive content. If so, the synthesis request is rejected and the user is informed. When the inference material uploaded by the user is audio, voiceprint verification is required to prevent the user from stealing other people's voiceprints. Then, the audio file uploaded by the user or the audio file converted by TTS is combined with the digital human inference model to adapt the lip shape, facial expression, body movement, etc. The user can also select different clips for synthesis based on the preview video, and finally infer the digital human video;

[0089] Step 315: Algorithm Engineering will train and infer the digital human video and transmit it back to the AI ​​gateway. The cloud will store user data, and users can submit their work directly through the website.

[0090] In some examples, the Studio backend can also report the approval content of each node to the HIVE data warehouse.

[0091] Figure 4 An exemplary block diagram of a video processing device according to an embodiment of the present disclosure is shown.

[0092] like Figure 4 As shown, the video processing device 400 may include a first input unit 410 , a model generation unit 420 , a preview unit 430 , a second input unit 440 , and an output unit 450 .

[0093] The first input unit 410 may be configured to obtain an input video and a preview audio for digital human model training. The model generation unit 420 may be configured to generate a digital human model based on the input video. The preview unit 430 may be configured to apply the preview audio to the digital human model to generate a plurality of preview videos, wherein the digital human image in the preview video corresponds to the user image in the input video, and the audio of the plurality of preview videos is the same as the preview audio. The second input unit 440 may be configured to obtain material for generating a target video and a selection of at least one preview video in the plurality of preview videos. The output unit 450 may be configured to generate a target video based on the material and the selected preview video, wherein the target video has an audio corresponding to the material.

[0094] In some embodiments, generating a digital human model based on an input video includes: determining a plurality of video clips from the input video, wherein each video clip has a corresponding user image; and generating a digital human model based on the plurality of video clips.

[0095] In some embodiments, the digital human image in each of the multiple preview videos corresponds to the user image in the multiple video clips.

[0096] In some embodiments, the material includes at least one of audio and text.

[0097] In some embodiments, the output unit is further configured to: determine whether the material meets the material review criteria; and in response to the material meeting the material review criteria, generate a target video based on the material and the selected preview video.

[0098] In some embodiments, the output unit is further configured to: in response to the material not satisfying the material review standard, abandon generating the target video based on the material and the selected preview video.

[0099] In some embodiments, the material review criteria include voiceprint review criteria for audio and text content review criteria for text.

[0100] In some embodiments, determining multiple video segments from an input video includes: determining multiple video segments from the input video based on a video review standard, wherein the video review standard includes a video quality standard and a video content standard, and the determined multiple video segments simultaneously meet the video quality standard and the video content standard.

[0101] In some embodiments, the user image and the digital human image include lip movements, facial expressions, and body movements.

[0102] In some embodiments, generating a digital human model based on an input video includes: generating a digital human model based on the input video using small sample learning.

[0103] It should be understood that Figure 4 The various units of the device 400 shown in FIG. 4 can be used in conjunction with the reference Figure 1 The steps in the method 100 described above correspond to each other. Therefore, the operations, features and advantages described above for the method 100 are also applicable to the device 400 and the units included therein. For the sake of brevity, some operations, features and advantages are not described in detail here.

[0104] It should also be understood that various techniques may be described herein in the general context of software hardware elements or program modules. Figure 4The various units described can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these units can be implemented as computer program codes / instructions, which are configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these units can be implemented as hardware logic / circuits. For example, in some embodiments, one or more of the first input unit 410, the model generation unit 420, the preview unit 430, the second input unit 440, and the output unit 450 can be implemented together in a system on chip (System on Chip, SoC). SoC may include an integrated circuit chip (which includes a processor (e.g., a central processing unit (CPU), a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), a memory, one or more communication interfaces, and / or one or more components in other circuits), and may optionally execute the received program code and / or include embedded firmware to perform functions.

[0105] According to another aspect of the present disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, and the computer program implements the above method when executed by the at least one processor.

[0106] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing a computer program is also provided, wherein the computer program implements the above method when executed by a processor.

[0107] According to another aspect of the present disclosure, a computer program product is further provided, comprising a computer program, wherein the computer program implements the above method when executed by a processor.

[0108] See also Figure 5 , a block diagram of an electronic device 500 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device can be a computer device of different types, such as a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0109] like Figure 5 As shown, the electronic device 500 may include at least one processor 501 , a working memory 502 , an input unit 504 , a display unit 505 , a speaker 506 , a storage unit 507 , a communication unit 508 , and other output units 509 that can communicate with each other via a system bus 503 .

[0110] The processor 501 may be a single processing unit or multiple processing units, all of which may include a single or multiple computing units or multiple cores. The processor 501 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. The processor 501 may be configured to obtain and execute computer-readable instructions stored in a working memory 502, a storage unit 507, or other computer-readable media, such as program codes of an operating system 502a, program codes of an application 502b, and the like.

[0111] The working memory 502 and the storage unit 507 are examples of computer-readable storage media for storing instructions, which are executed by the processor 501 to implement the various functions described above. The working memory 502 may include both volatile memory and non-volatile memory (e.g., RAM, ROM, etc.). In addition, the storage unit 507 may include a hard disk drive, a solid-state drive, a removable medium, including external and removable drives, a memory card, a flash memory, a floppy disk, an optical disk (e.g., a CD, a DVD), a storage array, a network attached storage, a storage area network, etc. The working memory 502 and the storage unit 507 may all be collectively referred to herein as memory or computer-readable storage media, and may be a non-transitory medium capable of storing computer-readable, processor-executable program instructions as computer program code, which may be executed by the processor 501 as a specific machine configured to implement the operations and functions described in the examples herein.

[0112] The input unit 506 may be any type of device capable of inputting information to the electronic device 500. The input unit 506 may receive input digital or character information and generate key signal input related to user settings and / or function control of the electronic device, and may include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone and / or a remote controller. The output unit may be any type of device capable of presenting information, and may include but is not limited to a display unit 505, a speaker 506, and other output units 509, which may include but are not limited to video / audio output terminals, vibrators and / or printers. The communication unit 508 allows the electronic device 500 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as a Bluetooth device, an 802.8 device, a WiFi device, a WiMax device, a cellular communication device and / or the like.

[0113] The application 502b in the working register 502 can be loaded to execute the various methods and processes described above, such as Figure 1 . For example, in some embodiments, the method 100 described above may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 507. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via the storage unit 507 and / or the communication unit 508. When the computer program is loaded and executed by the processor 501, one or more steps of the method 100 described above may be performed. Alternatively, in other embodiments, the processor 501 may be configured to execute the method 100 in any other appropriate manner (e.g., by means of firmware).

[0114] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0115] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0116] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0117] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0118] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0119] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0120] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0121] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A video processing method, comprising: Obtain input video and preview audio for digital human model training; Generate a digital human model based on the input video; Applying the preview audio to the digital human model to generate a plurality of preview videos, wherein the digital human images in the plurality of preview videos correspond to the user images in the input video, and the audios of the plurality of preview videos are the same as the preview audio; Acquire materials for generating a target video and select at least one preview video from among the plurality of preview videos; and A target video is generated based on the material and the selected preview video, wherein the target video has audio corresponding to the material.

2. The video processing method according to claim 1, wherein: Generating a digital human model based on the input video includes: Determining a plurality of video clips from the input video, wherein each video clip has a corresponding user image; and The digital human model is generated based on the multiple video clips.

3. The video processing method according to claim 2, wherein: The digital human image in each of the multiple preview videos corresponds to the user image in the multiple video clips.

4. The video processing method according to claim 2, wherein: Determining a plurality of video segments from the input video comprises: The multiple video segments are determined from the input video based on a video review standard, wherein the video review standard includes a video quality standard and a video content standard, and the multiple video segments determined meet both the video quality standard and the video content standard.

5. The video processing method according to claim 1, wherein: The material includes at least one of audio and text.

6. The video processing method according to claim 5, wherein: Generating a target video based on the material and the selected preview video includes: Determining whether the material meets the material review criteria; and In response to the material satisfying the material review standard, the target video is generated based on the material and the selected preview video.

7. The video processing method according to claim 6, further comprising: In response to the material not satisfying the material review standard, abandoning the generation of the target video based on the material and the selected preview video.

8. The video processing method according to any one of claims 6 to 7, wherein: The material review standards include voiceprint review standards for audio and text content review standards for text.

9. The video processing method according to claim 1, wherein: The user image and the digital human image include lip shape, facial expression and body movement.

10. The video processing method according to claim 1, wherein: Generating a digital human model based on the input video includes: The digital human model is generated based on the input video using small sample learning.

11. A video processing device, comprising: A first input unit is configured to obtain an input video and a preview audio for digital human model training; A model generation unit, configured to generate a digital human model based on the input video; A preview unit, configured to apply the preview audio to the digital human model to generate a plurality of preview videos, wherein the digital human image in the plurality of preview videos corresponds to the user image in the input video, and the audio of the plurality of preview videos is the same as the preview audio; A second input unit configured to obtain a material for generating a target video and a selection of at least one preview video among the plurality of preview videos; and An output unit is configured to generate a target video based on the material and the selected preview video, wherein the target video has audio corresponding to the material.

12. The video processing device according to claim 11, wherein: Generating a digital human model based on the input video includes: Determining a plurality of video clips from the input video, wherein each video clip has a corresponding user image; and The digital human model is generated based on the multiple video clips.

13. The video processing device according to claim 12, wherein: The digital human image in each of the multiple preview videos corresponds to the user image in the multiple video clips.

14. The video processing device according to claim 12, wherein: Determining a plurality of video segments from the input video comprises: The multiple video segments are determined from the input video based on a video review standard, wherein the video review standard includes a video quality standard and a video content standard, and the multiple video segments determined meet both the video quality standard and the video content standard.

15. The video processing device according to claim 11, wherein: The material includes at least one of audio and text.

16. The video processing device according to claim 15, wherein: The output unit is further configured to: Determining whether the material meets the material review criteria; and In response to the material satisfying the material review standard, the target video is generated based on the material and the selected preview video.

17. The video processing device according to claim 16, wherein the output unit is further configured to: In response to the material not satisfying the material review standard, abandoning the generation of the target video based on the material and the selected preview video.

18. The video processing device according to any one of claims 16 to 17, wherein: The material review standards include voiceprint review standards for audio and text content review standards for text.

19. The video processing device according to claim 11, wherein: The user image and the digital human image include lip shape, facial expression and body movement.

20. The video processing device according to claim 11, wherein: Generating a digital human model based on the input video includes: The digital human model is generated based on the input video using small sample learning.

21. An electronic device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores a computer program, which, when executed by the at least one processor, implements the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing a computer program, wherein: The computer program implements the method according to any one of claims 1 to 10 when executed by a processor.

23. A computer program product comprising a computer program, wherein: The computer program implements the method according to any one of claims 1 to 10 when executed by a processor.

Images