Method for realizing 1:1 digital human based on front-end portrait reconstruction

Abstract

The application discloses a method for realizing 1:1 digital people based on front-end portrait reconstruction, which splits voice in a user video and a character image model into phoneme sequences and portrait sequences, processes phonemes and sequence frames, thereby constructing a digital person micro model, loading a corresponding digital person micro model through the front end, then decomposing user question voice into phonemes and giving a time range for each phoneme execution, taking each phoneme and the time range for phoneme execution as a face reconstruction modification parameter, replacing the face position of the portrait in the action sequence with correction data of the face position of the portrait in the canvas, thereby completing matching of the face and voice under a single sequence frame, realizing real-time face reconstruction of the portrait by phonemes, greatly accelerating the efficiency from user question raising to digital person question answering, reconstructing actions through sequence recombination, thereby constructing a digital person video, and greatly reducing the occupation of a picture generation server in the reconstruction process.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of digital people, and particularly relates to a method for realizing one-to-one digital people based on front-end portrait reconstruction. BACKGROUND

[0002] With the rise of large models, artificial intelligence gradually has the ability to communicate with people, making the digital people industry begin to develop rapidly. However, most of the existing digital people are limited by computer computing power, and the face reconstruction based on the backend technology can only complete the picture of the digital people speaking based on voice driving. The video is generated from the server side, and it is difficult to realize the low-cost production of dialogue pictures, so that the current digital people can almost only produce videos, which greatly limits the application scenarios of digital people. At present, the terminal carrier in the hands of ordinary users is a mobile phone. However, due to the limited performance of the mobile phone, it is difficult to present the 3D digital people in detail on the mobile phone. At the same time, if a 3D image is used to make a digital person, an independent APP or application is usually needed as a carrier of the digital person, which increases the threshold for users to use the digital person.

[0003] The prior art generates a new video through lip movement based on voice driving of an existing video or picture. In the existing technology, the video is generated in segments or in a streaming mode. The streaming mode can basically realize real-time interaction. However, whether the video is generated in segments or in a streaming mode, a host with a certain computing capability is needed to support the generation of video segments. This is a great pressure on the video generation service. If multiple people use it at the same time, each user needs a server to support the generation of the video. This will greatly limit the use scenarios of real-time digital people. SUMMARY

[0004] The purpose of the application is to solve the problem of great pressure on the video generation service and the problem of not being suitable for multiple users to use at the same time in the prior art.

[0005] To achieve the above purpose, the application provides a method for realizing one-to-one digital people based on front-end portrait reconstruction, characterized in that the method comprises the following steps:

[0006] S1, obtaining a user video

[0007] Obtain a video shot by a user or an AI-generated video, and remove the background technology in the video through PaddleSeg.

[0008] S2, obtaining a phoneme sequence

[0009] The language of the user video in step S1 is divided into basic phonemes through the corresponding pinyin of Chinese characters or the corresponding phonetic symbols of English, and all phonemes are collected, that is, a phoneme sequence is generated.

[0010] S3, making a digital person micro model

[0011] S301, split the person image model in the user video in step S1 into person sequence frames, and unify the size of the sequence frames;

[0012] S302, perform person reconstruction on each person sequence frame through the phoneme sequence in step S2; each phoneme is used to drive the person sequence frame, so that the person produces lip changes under the driving of the phoneme;

[0013] S303, retain the key features generated in the person reconstruction process in step S302, combine the key features and the phonemes corresponding to the key features into a face reconstruction model corresponding to the person sequence frame, and combine the face reconstruction models generated by each person sequence frame, that is, the face reconstruction micro model of the digital person is generated;

[0014] S4, load the corresponding digital person micro model in the front end

[0015] S401, load the person image data through the person sequence frame in step S301, convert the person sequence frame into an imageData image color object through canvas, save the image imageData data in the order of the person sequence frame, that is, the person sequence frame data;

[0016] S402, load the face micro model in step S303, extract the face feature changes generated by each phoneme corresponding to each person sequence frame from the face micro model, and establish a mapping relationship between the person sequence frame, the phoneme and the face feature change in the front end;

[0017] S5, speech is decomposed into phonemes and the time range executed by each phoneme is given

[0018] After the user's question is processed through the database and the large model tool, the answer text corresponding to the question is generated, the text is converted into speech, because human speech is generated by the pronunciation corresponding to the phoneme, so the phoneme sequence corresponding to the speech and the time period corresponding to each phoneme in the sequence in the speech can be generated at the same time. The text, the speech generated by the text and the phoneme information generated by the speech are simultaneously submitted as an answer;

[0019] S6, the front end reconstructs the face in real time based on the phoneme

[0020] The person sequence generated in step S401 is used for action reconstruction through sequence recombination; the phoneme sequence in step S5 and the time range executed by each phoneme in the sequence are used to realize the real-time reconstruction of the face by the phoneme through the mapping relationship in step S402;

[0021] S7, construct a digital person video

[0022] Step S6 is performed on each sequence frame, and a digital human video is constructed.

[0023] Further, the step S301 of unifying the size of the sequence frame is specifically: setting the width of the sequence frame as 512 pixels; calculating the height of the scaled sequence frame based on the width of the original video, and then using the resize function in the opencv tool to scale the size of the portrait sequence frame to obtain a portrait sequence frame with a unified width.

[0024] Further, the step S302 is specifically: a text containing all phoneme sequences is generated through the step S2, the phoneme sequence text is converted into speech, continuous pictures are generated using speech driving, the continuous pictures are decomposed into video frames, the sequence number of each phoneme corresponding to the video frame is found, the mapping relationship between the phoneme and the portrait sequence frame is established through the sequence number, and the mapping relationship between the phoneme and the portrait sequence frame is used to make the portrait change in mouth shape under the driving of the phoneme.

[0025] Further, the step S6 is specifically:

[0026] The principle of generating a video effect by the portrait sequence is: the principle of video playing is to play picture data in sequence.

[0027] The principle of generating action changes by the video is: a developed action can be differentiated into N static action segments; if the development is regarded as a forward sequence, then reversing each action segment is to retract the developed action; reversing at different time points will generate actions with different amplitudes.

[0028] Based on the above principle, a random function is designed in the front end, and the probability of randomly triggering the sequence reversal is designed; this will make the entire action sequence generate actions with different amplitudes, and form various new actions in vision.

[0029] While playing the audio, the phoneme sequence generated in step S5 and each phoneme in the sequence corresponding to the time period in the speech are used as parameters, the mapping relationship generated in step S402 is used to generate correction data of the position of the face in the face reconstruction process, the correction data of the position of the face is used to replace the position of the face of the portrait in the action sequence in the canvas, so as to complete the matching of the face and the speech under a single sequence frame, and realize real-time reconstruction of the face.

[0030] Beneficial effects:

[0031] 1. The present application inputs user questions into a database and a large model tool, thereby generating answer text corresponding to the questions, converting the text into speech, and generating phoneme sequences corresponding to the speech and the time period in the speech corresponding to each phoneme in the sequence, thereby correcting the face position of the portrait using the phonemes and the time period in the speech as parameters, completing the matching of the face and the speech in a single sequence frame, and realizing real-time answering of user questions and speeding up the process of answering questions. The front end uses asynchronous same frequency audio conversion image, which is imperceptible to the user. This greatly speeds up the process from user questioning to digital human answering questions. In the back end question answering, sufficient optimization is done, so that the most complex questions can be answered within 3 seconds.

[0032] 2. In the process of reconstructing the digital human, the back-end server does not need to generate pictures, greatly reducing the occupation of picture generation servers. In the process of real-time dialogue of the digital human, the back-end server does not need to transmit pictures in real time, greatly reducing the occupation of server network resources.

[0033] 3. The present application reduces the cost of providing personalized digital human services to individual users. It makes it possible to provide one-to-one digital human services at a low cost. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is a person reconstruction method flowchart provided by an embodiment of the present application;

[0035] Figure 2 is a digital human micro model generation entry code diagram provided by the present application;

[0036] Figure 3 is a digital human micro model making code diagram provided by the present application;

[0037] Figure 4 is a digital human reconstruction video model diagram provided by the present application;

[0038] Figure 5 is a digital human reconstruction video effect diagram provided by the present application. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0040] The application principle of the present application will be further described below in combination with the drawings and specific embodiments.

[0041] Embodiment 1:

[0042] As Figures 1-4As shown, a method for realizing a 1-to-1 digital person based on front-end portrait reconstruction, the specific steps are as follows:

[0043] S1, obtaining a user video

[0044] Obtain the video shot by the user or the AI-generated video, and remove the background in the video through PaddleSeg technology.

[0045] S2, obtaining a phoneme sequence

[0046] The language of the video in step S1 is divided into basic phonemes through the corresponding pinyin of Chinese characters or the corresponding phonetic symbols of English, and all phonemes are collected, i.e. a phoneme sequence is generated. For example, the pinyin of "hello" in Chinese is "ni, hao", which is divided into four phonemes "[n, i], [h, ao]". Repeat the process to exhaust all Chinese pinyin to obtain the phoneme sequence of all Chinese characters.

[0047] S3, making a digital person micro model

[0048] S301, split the image model of the person in the video for modeling in step S1 into portrait sequence frames, according to the frame number of the video, generally 25-30 frames / second; unify the size of the sequence frames, set the width of the sequence frames to a fixed value, i.e. 512 pixels; calculate the height of the scaled sequence frames based on the width of the original video, and then use the resize function in the opencv tool to scale the size of the portrait sequence frames to obtain portrait sequence frames with uniform width.

[0049] S302, reconstruct the portrait of each portrait sequence frame through the phoneme sequence in step S2; use each phoneme to drive the portrait sequence frame, so that the portrait changes the mouth shape under the driving of the phoneme. Specifically as follows:

[0050] A text containing all phoneme sequences is generated through step S2, the phoneme sequence text is converted into speech, the continuous picture is generated by using the speech drive, the continuous picture is divided into video frames, the sequence number of each phoneme corresponding to the video frame is found, the mapping relationship between the phoneme and the portrait sequence frame is established through the sequence number, and the mapping relationship between the phoneme and the portrait sequence frame is established. The portrait changes the mouth shape under the driving of the phoneme.

[0051] S303, keep the key features generated in the portrait reconstruction process in step S302, such as the face and the features around the mouth when speaking, combine these key features and the corresponding phonemes into the face reconstruction model corresponding to the portrait sequence frame. The face reconstruction model of each portrait sequence frame is combined, and the face reconstruction micro model of the digital person is generated.

[0052] S4, load the corresponding digital person micro model in the front end

[0053] S401, load the portrait data by the way of portrait sequence frames in step S301, convert the portrait sequence frames into imageData image color objects by canvas, save the image imageData data in sequence frame order, that is, portrait sequence frame data.

[0054] S402, load the face micro model in step S303, extract the face feature changes generated by each phoneme corresponding to each portrait sequence frame from the face micro model, and establish a mapping relationship between the portrait sequence frame, the phoneme, and the face feature change in the front end.

[0055] S5, speech is decomposed into phonemes and the time range of each phoneme is given

[0056] The user question generates an answer text corresponding to the question through a database and a large model tool, converts the text into speech, because human speech is generated by the pronunciation corresponding to the phoneme, so the phoneme sequence corresponding to the speech and the time period in the speech corresponding to each phoneme in the sequence can be generated at the same time as the speech is generated. The text, the speech generated by the text, and the phoneme information generated by the speech are submitted as answers at the same time.

[0057] S6, the front end reconstructs the face in real time based on the phoneme

[0058] Use the portrait sequence generated in step S401 to reconstruct the action by sequence reorganization.

[0059] Action reconstruction principle 1: the principle of video playback is to play pictures in sequence.

[0060] Action reconstruction principle 2: unfolding an action can be differentiated into N static action segments. If unfolding is considered as forward order, then reversing each action segment is to recover the unfolded action. Reversing at different time points will produce actions of different amplitudes.

[0061] Based on the above principles, a random function is designed in the front end, and the probability of triggering sequence reversal is designed. This will make the entire action sequence produce actions of different amplitudes, forming various new actions in vision.

[0062] At the same time of playing the audio, the phoneme sequence generated in step S5 and the time period in the speech corresponding to each phoneme in the sequence are used as parameters to generate correction data of the face position in the face reconstruction process through the mapping relationship generated in step S402. Replace the face position of the portrait in the action sequence in the canvas with the correction data of the face position, so as to complete the matching of the face and the speech under a single sequence frame, realize real-time face reconstruction.

[0063] S7, construct a digital human video

[0064] Step S6 is performed on each sequence frame, i.e. a digital human video is constructed, and the digital human video effect can be referred to Figure 5 .

[0065] The sound type combination front-end code program is as follows:

[0066]

[0067]

[0068] The above merely describes preferred embodiments of the present application but is not intended to limit the present application, and any modification, equivalent replacement, and improvement within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for realizing a 1-to-1 digital human based on front-end portrait reconstruction, characterized in that, Specifically as follows: S1. Obtain user video It acquires user-shot or AI-generated videos and uses PaddleSeg to remove backgrounds from the videos. S2, Obtain the phoneme sequence The language of the user video in step S1 is decomposed into basic phonemes using the pinyin corresponding to Chinese characters or the phonetic symbols corresponding to English characters, and all phonemes are collected to generate a phoneme sequence. S3. Creating a digital human micro-model S301. The human image model in the user video in step S1 is split into human image sequence frames, and the size of the sequence frames is unified. S302. Reconstruct the human image for each human image sequence frame using the phoneme sequence in step S2; use each phoneme to drive the human image sequence frame so that the human image produces lip movements under the drive of the phonemes. S303. Retain the key features generated during the portrait reconstruction process in step S302, and merge the key features and the corresponding phonemes into a face reconstruction model corresponding to the portrait sequence frame; merge the face reconstruction models generated by each portrait sequence frame to generate the digital human's face reconstruction micro-model. S4. Front-end loads the corresponding digital human micro-model. S401. Load portrait data using the portrait sequence frame method in step S301, convert the portrait sequence frame into an imageData image color object using canvas, and save the imageData data, i.e., portrait sequence frame data, in the order of the portrait sequence frames. S402. Load the face micro-model from step S303, extract the facial feature changes generated by each phoneme corresponding to each portrait sequence frame from the face micro-model, and establish a mapping relationship between portrait sequence frames, phonemes and facial feature changes at the front end. S5. Decompose speech into phonemes and provide the execution time range for each phoneme. After processing the user's question through the database and large model tools, the corresponding answer text is generated. The text is then converted into speech. Since human speech is produced by the pronunciation of phonemes, the phoneme sequence corresponding to the speech and the time segment corresponding to each phoneme in the sequence are generated at the same time as the speech. The text, the speech generated from the text, and the phoneme information generated from the speech are submitted as the answer at the same time. S6. Front-end performs real-time face reconstruction based on phonemes. Using the human image sequence generated in step S401, action reconstruction is performed through sequence recombination; using the phoneme sequence in step S5 and the time range of each phoneme in the sequence, the face position is modified through the mapping relationship in step S402, thereby realizing real-time reconstruction of the face by phonemes. S7, Building Digital Human Videos By performing step S6 on each sequence frame, the digital human video can be constructed.

2. The method for realizing a 1-to-1 digital human based on front-end portrait reconstruction according to claim 1, characterized in that, In step S301, the size of the unified sequence frame is specifically as follows: the width of the sequence frame is set to 512 pixels; the height of the scaled sequence frame is calculated based on the width of the original video; and the size of the portrait sequence frame is scaled using the resize function in OpenCV to obtain a portrait sequence frame with a uniform width.

3. The method for realizing a 1-to-1 digital human based on front-end portrait reconstruction according to claim 1, characterized in that, Step S302 specifically involves: generating a text containing all phoneme sequences through step S2, converting the phoneme sequence text into speech, using speech to drive the generation of continuous images, decomposing the continuous images into video frames, finding the sequence number of the video frame corresponding to each phoneme, establishing a mapping relationship between phonemes and human image sequence frames through the sequence number, and using the mapping relationship between phonemes and human image sequence frames to make the human image produce lip movements under the drive of phonemes.

4. The method for realizing a 1-to-1 digital human based on front-end portrait reconstruction according to claim 1, characterized in that, Step S6 is as follows: The principle behind creating video effects from a sequence of human images: The principle of video playback is to play image data in sequence; The principle behind video motion changes: unfolding an action can be divided into N static action segments; if unfolding is considered as a forward sequence, then reversing each action segment is the retraction of the unfolded action; reversing it again at different points in time will produce motion of different amplitudes. Based on the above principles, by designing a random function on the front end and setting the probability of randomly triggering the reverse of the sequence, the entire action sequence will produce movements of different amplitudes, creating various new visual movements. While playing the audio, the phoneme sequence generated in step S5 and the time segment in the speech corresponding to each phoneme in the sequence are used as parameters. The face position correction data in the face reconstruction process is generated through the mapping relationship generated in step S402. The face position correction data is used to replace the face position of the portrait in the action sequence in the canvas, thereby completing the matching of face and speech in a single sequence frame and realizing real-time face reconstruction.

Images