Method for realizing one-to-one digital person based on front-end portrait reconstruction

Abstract

The invention discloses a method for realizing a one-to-one digital person based on front-end portrait reconstruction, which comprises the following steps of: splitting voice and a person image model in a user video into a phoneme sequence and a portrait sequence, processing phonemes and sequence frames so as to construct a digital person micro-model, loading a corresponding digital person micro-model through a front end, and constructing a one-to-one digital person through the front-end portrait reconstruction. Secondly, decomposing question voice of a user into phonemes, giving out an execution time range of each phoneme, taking each phoneme and the execution time range of each phoneme as human face reconstruction modification parameters, and replacing the human face position of the portrait in the action sequence in canvas with the correction data of the human face position, so as to complete matching of the human face and the voice under a single sequence frame; according to the method, face real-time reconstruction of a portrait by phonemes is realized, the efficiency from question asking by a user to question answering by a digital person is greatly accelerated, action reconstruction is performed through sequence recombination, so that a digital person video is constructed, a back-end server does not need to generate a picture in the reconstruction process, and the occupation of a picture generation server is greatly reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of digital humans, and in particular relates to a method for realizing a one-to-one digital human based on front-end portrait reconstruction. Background Art

[0002] With the rise of large models, artificial intelligence has gradually acquired the ability to communicate with people, which has led to the booming development of the digital human industry. However, most of the existing digital humans are limited by computer computing power, and the face reconstruction based on back-end technology can only complete the production of digital human speaking images based on voice drive. It is difficult to produce dialogue images at low cost from the server-side video, which makes the current digital humans almost only able to produce videos, greatly limiting the application scenarios of digital humans. At present, the most common terminal carrier in the hands of ordinary users is the mobile phone, but due to the limited performance of the mobile phone, it is difficult to present 3D digital humans in a delicate manner on the mobile phone. At the same time, if a 3D image is used to make a digital human, it usually requires an independent APP or application as a carrier of the digital human, which raises the threshold for users to use digital humans.

[0003] The existing technology generates new videos by lip-syncing with existing videos or pictures through voice drive. The existing technologies include segment-based video generation and streaming video generation. Streaming video generation can basically achieve real-time interaction. However, whether it is segment-based or streaming generation, a host with certain computing power is required to support the generation of video clips. This puts a lot of pressure on the video generation service. If multiple people use it at the same time, each user needs a server to support the generation of the video. This will greatly limit the use scenarios of real-time digital people. Summary of the invention

[0004] The purpose of the present invention is to solve the problem in the prior art that the video generation service has high pressure and is not suitable for simultaneous use by multiple users.

[0005] To achieve the above object, the present invention provides a method for realizing 1 to 1 digital human based on front-end portrait reconstruction, which is characterized by the following:

[0006] S1. Get user video

[0007] Get videos shot by users or generated by AI, and use PaddleSeg to remove background from the videos.

[0008] S2. Get phoneme sequence

[0009] Decomposing the language of the user video in step S1 into basic phonemes through the pinyin corresponding to Chinese characters or the phonetic symbols corresponding to English characters, and collecting all the phonemes, that is, generating a phoneme sequence;

[0010] S3. Make a digital human micro model

[0011] S301, splitting the character image model in the user video in step S1 into portrait sequence frames, and unifying the sizes of the sequence frames;

[0012] S302, reconstructing the portrait for each portrait sequence frame by using the phoneme sequence in step S2; using each phoneme to drive the portrait sequence frame, so that the portrait produces a mouth shape change under the drive of the phoneme;

[0013] S303, retaining the key features generated in the portrait reconstruction process in step S302, merging the key features and the phonemes corresponding to the features into a face reconstruction model corresponding to the portrait sequence frame; merging the face reconstruction models generated by each portrait sequence frame to generate a face reconstruction micro-model of the digital human;

[0014] S4. Load the corresponding digital human micro model at the front end

[0015] S401, loading portrait data in the form of portrait sequence frames in step S301, converting the portrait sequence frames into imageData image color objects through canvas, and saving image imageData data in the order of portrait sequence frames, i.e., portrait sequence frame data;

[0016] S402, loading the face micro-model in step S303, extracting the face feature changes caused by each phoneme corresponding to each portrait sequence frame from the face micro-model, and establishing a mapping relationship between the portrait sequence frame, the phoneme and the face feature changes at the front end;

[0017] S5. Decompose speech into phonemes and give the time range for each phoneme to execute

[0018] After processing the user's question through the database and large model tools, the answer text corresponding to the question is generated, and the text is converted into speech. Because human speech is produced by the pronunciation corresponding to the phonemes, when the speech is generated, the phoneme sequence corresponding to the speech and each phoneme in the sequence corresponds to the time segment in the speech can be generated. The text, the speech generated by the text, and the phoneme information generated by the speech are submitted as answers at the same time;

[0019] S6, front-end real-time reconstruction of human face based on phonemes

[0020] Using the portrait sequence generated in step S401, the action is reconstructed by sequence reorganization; using the phoneme sequence in step S5 and the time range of execution of each phoneme in the sequence, the face position is modified through the mapping relationship in step S402, so as to realize real-time reconstruction of the face by phonemes;

[0021] S7. Constructing Digital Human Video

[0022] Step S6 is performed on each sequence frame to construct a digital human video.

[0023] Furthermore, in step S301, the size of the unified sequence frame is specifically as follows: the width of the sequence frame is set to 512 pixels; the height of the scaled sequence frame is calculated based on the width of the original video, and then the size of the portrait sequence frame is scaled using the resize function in the opencv tool to obtain a portrait sequence frame of uniform width.

[0024] Furthermore, step S302 is specifically as follows: a text containing all phoneme sequences is generated through step S2, the phoneme sequence text is converted into speech, a continuous picture is generated using speech drive, the continuous picture is decomposed into video frames, the serial number of the video frame corresponding to each phoneme is found, a mapping relationship between the phoneme and the portrait sequence frame is established through the serial number, and through the mapping relationship between the phoneme and the portrait sequence frame, the portrait produces lip shape changes driven by the phoneme.

[0025] Furthermore, step S6 is specifically as follows:

[0026] The principle of video effects produced by portrait sequences: The principle of video playback is to play the image data in sequence;

[0027] The principle of motion change in video: unfolding an action can be differentiated into N static action segments; considering the unfolding as a positive sequence, reversing each action segment is the retraction of the unfolded action; reversing again at different time points will produce actions of different amplitudes;

[0028] Based on the above principles, a random function is designed at the front end, and the probability of randomly triggering the reverse sequence is designed; this will cause the entire action sequence to produce actions of different amplitudes, forming various new actions visually;

[0029] While playing the audio, the phoneme sequence generated in step S5 and the time segment in the speech corresponding to each phoneme in the sequence are used as parameters, and the correction data of the face position in the face reconstruction process is generated through the mapping relationship generated in step S402. The face position correction data replaces the face position of the portrait in the action sequence in the canvas, thereby completing the matching of the face and the speech in a single sequence frame and realizing real-time reconstruction of the face.

[0030] Beneficial effects:

[0031] 1. In the present invention, the user's question is input into the database and the large model tool to generate the answer text corresponding to the question, and the text is converted into speech, so that the phoneme sequence corresponding to the speech and each phoneme in the sequence corresponds to the time segment in the speech can be generated, so that the phoneme and the corresponding time segment in the speech are used as parameter pairs to correct the face position of the portrait, thereby completing the matching of the face and the speech in a single sequence frame, realizing real-time answering of the user's questions and speeding up the process of answering questions. The front end adopts asynchronous same-frequency audio to image conversion, which is imperceptible to the user. This greatly speeds up the process from user asking questions to digital people answering questions. In terms of answering questions on the back end, sufficient optimization can be done to achieve the fastest answering of more complex questions within 3 seconds.

[0032] 2. In the process of digital human reconstruction, there is no need for the back-end server to generate images, which greatly reduces the occupation of the image generation server. In the process of real-time dialogue between digital humans, there is no need for the back-end server to transmit images in real time, which greatly reduces the occupation of server network resources.

[0033] 3. The present invention reduces the cost of providing personalized digital human services to a single user, making it possible to provide universal one-to-one digital human services at a low cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flow chart of a character reconstruction method provided by an embodiment of the present invention;

[0035] Figure 2 It is a digital human micro-model generation entry code diagram provided by the present invention;

[0036] Figure 3 It is a code diagram for making a digital human micro-model provided by the present invention;

[0037] Figure 4 It is a digital human reconstruction video model diagram provided by the present invention;

[0038] Figure 5 This is a video effect diagram of digital human reconstruction provided by the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0040] The application principle of the present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0041] Embodiment 1:

[0042] like Figure 1-Figure 4As shown in the figure, a method for realizing a one-to-one digital human based on front-end human portrait reconstruction is as follows:

[0043] S1. Obtain the user's video

[0044] Obtain the video shot by the user or the video generated by AI, and remove the background technology in the video through PaddleSeg. PaddleSeg can refer to the website https: / / github.com / PaddlePaddle / PaddleSeg.

[0045] S2. Obtain the phoneme sequence

[0046] Decompose the language of the video in step S1 into basic phonemes through the pinyin corresponding to Chinese characters or the phonetic symbols corresponding to English, and collect all phonemes, that is, generate a phoneme sequence. For example, the pinyin corresponding to the Chinese "你好" is "ni,hao", and the decomposed phonemes are "[n,i], [h,ao]" four phonemes. Repeat this process, and exhaust all Chinese pinyins to obtain the phoneme sequences of all Chinese characters.

[0047] S3. Make a digital human micro-model

[0048] S301. Split the human image model in the video for modeling in step S1 into a sequence of portrait frames. According to the number of frames of the video, generally 25 - 30 frames per second; unify the size of the sequence frames, set the width of the sequence frames to a fixed value, that is, 512 pixels; calculate the height of the scaled sequence frames based on the width of the original video, and then use the resize function in the opencv tool to scale the size of the portrait sequence frames to obtain portrait sequence frames with a unified width.

[0049] S302. Perform human portrait reconstruction on each portrait sequence frame through the phoneme sequence in step S2; use each phoneme to drive the portrait sequence frame, so that the portrait generates lip movement changes under the drive of the phoneme. Specifically as follows:

[0050] A text containing all phoneme sequences is generated through step S2. Convert the phoneme sequence text into speech, use the speech to drive the generation of continuous pictures, decompose the continuous pictures into video frames, find the serial numbers of the video frames corresponding to each phoneme, establish the mapping relationship between the phoneme and the portrait sequence frame through the serial numbers, and make the portrait generate lip movement changes under the drive of the phoneme through the mapping relationship between the phoneme and the portrait sequence frame.

[0051] S303, retaining the key features generated in the portrait reconstruction process in step S302, such as the features of the face and the area around the mouth when speaking, and merging these key features and the phonemes corresponding to the features into a face reconstruction model corresponding to the portrait sequence frame. The face reconstruction models generated by each portrait sequence frame are merged to generate a face reconstruction micro model of the digital human.

[0052] S4. Load the corresponding digital human micro model at the front end

[0053] S401, loading portrait data in the form of portrait sequence frames in step S301, converting the portrait sequence frames into imageData image color objects through canvas, and saving the imageData data in sequence frame order, namely, portrait sequence frame data.

[0054] S402, loading the face micro-model in step S303, extracting the face feature changes caused by each phoneme corresponding to each portrait sequence frame from the face micro-model, and establishing a mapping relationship between the portrait sequence frames, phonemes and face feature changes at the front end.

[0055] S5. Decompose speech into phonemes and give the time range for each phoneme to execute

[0056] The user's question is generated through the database and large model tools to generate the answer text corresponding to the question, and the text is converted into speech. Because human speech is produced by the pronunciation corresponding to the phonemes, when the speech is generated, the phoneme sequence corresponding to the speech can be generated, and each phoneme in the sequence corresponds to the time segment in the speech. The text, the speech generated by the text, and the phoneme information generated by the speech are submitted as answers at the same time.

[0057] S6, front-end real-time reconstruction of human face based on phonemes

[0058] Using the portrait sequence generated in step S401, action reconstruction is performed by sequence recombination.

[0059] Motion reconstruction principle 1: The principle of video playback is to play pictures in sequence.

[0060] Action reconstruction principle 2: An action can be differentiated into N static action segments by unfolding it. If the unfolding is regarded as a positive sequence, then reversing each action segment is the retraction of the unfolded action. Reversing again at different time points will produce actions of different amplitudes.

[0061] Based on the above principles, a random function is designed at the front end, and the probability of randomly triggering the reverse sequence is designed. This will cause the entire action sequence to produce actions of different amplitudes, forming various new actions visually.

[0062] While playing the audio, the phoneme sequence generated in step S5 and the time segment in the speech corresponding to each phoneme in the sequence are used as parameters, and the correction data of the face position in the face reconstruction process is generated through the mapping relationship generated in step S402. The face position correction data replaces the face position of the portrait in the action sequence in the canvas, thereby completing the matching of the face and the speech in a single sequence frame and realizing real-time reconstruction of the face.

[0063] S7. Constructing Digital Human Video

[0064] Execute step S6 for each sequence frame to construct a digital human video. The digital human video effect can be referred to Figure 5 .

[0065] The sound pattern combined with the front-end code program is as follows:

[0066]

[0067]

[0068] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for realizing 1 to 1 digital human based on front-end portrait reconstruction, characterized in that: The details are as follows: S1. Get user video Get videos shot by users or generated by AI, and use PaddleSeg to remove background from the videos. S2. Get phoneme sequence Decomposing the language of the user video in step S1 into basic phonemes through the pinyin corresponding to Chinese characters or the phonetic symbols corresponding to English characters, and collecting all the phonemes, that is, generating a phoneme sequence; S3. Make a digital human micro model S301, splitting the character image model in the user video in step S1 into portrait sequence frames, and unifying the sizes of the sequence frames; S302, reconstructing the portrait for each portrait sequence frame by using the phoneme sequence in step S2; using each phoneme to drive the portrait sequence frame, so that the portrait produces a mouth shape change under the drive of the phoneme; S303, retaining the key features generated in the portrait reconstruction process in step S302, merging the key features and the phonemes corresponding to the features into a face reconstruction model corresponding to the portrait sequence frame; merging the face reconstruction models generated by each portrait sequence frame to generate a face reconstruction micro-model of the digital human; S4. Load the corresponding digital human micro model at the front end S401, loading portrait data in the form of portrait sequence frames in step S301, converting the portrait sequence frames into imageData image color objects through canvas, and saving image imageData data in the order of portrait sequence frames, i.e., portrait sequence frame data; S402, loading the face micro-model in step S303, extracting the face feature changes caused by each phoneme corresponding to each portrait sequence frame from the face micro-model, and establishing a mapping relationship between the portrait sequence frame, the phoneme and the face feature changes at the front end; S5. Decompose speech into phonemes and give the time range for each phoneme to execute After processing the user's question through the database and large model tools, the answer text corresponding to the question is generated, and the text is converted into speech. Because human speech is produced by the pronunciation corresponding to the phonemes, when the speech is generated, the phoneme sequence corresponding to the speech can be generated, and each phoneme in the sequence corresponds to the time segment in the speech; the text, the speech generated by the text, and the phoneme information generated by the speech are submitted as answers at the same time; S6, front-end real-time reconstruction of human face based on phonemes Using the portrait sequence generated in step S401, the action is reconstructed by sequence reorganization; using the phoneme sequence in step S5 and the time range of execution of each phoneme in the sequence, the face position is modified through the mapping relationship in step S402, so as to realize real-time reconstruction of the face by phonemes; S7. Constructing Digital Human Video Step S6 is performed on each sequence frame to construct a digital human video.

2. The method for realizing one-to-one digital human based on front-end portrait reconstruction according to claim 1, characterized in that: The specific steps of unifying the size of the sequence frames in step S301 are as follows: setting the width of the sequence frames to 512 pixels; calculating the height of the scaled sequence frames based on the width of the original video, and then using the resize function in the opencv tool to scale the size of the portrait sequence frames to obtain portrait sequence frames of uniform width.

3. The method for realizing one-to-one digital human based on front-end portrait reconstruction according to claim 1, characterized in that: Step S302 is specifically as follows: a text containing all phoneme sequences is generated through step S2, the phoneme sequence text is converted into speech, a continuous picture is generated using speech drive, the continuous picture is decomposed into video frames, the serial number of the video frame corresponding to each phoneme is found, a mapping relationship between the phoneme and the portrait sequence frame is established through the serial number, and through the mapping relationship between the phoneme and the portrait sequence frame, the portrait produces lip shape changes driven by the phoneme.

4. The method for realizing one-to-one digital human based on front-end portrait reconstruction according to claim 1, characterized in that: Step S6 is specifically as follows: The principle of video effects produced by portrait sequences: The principle of video playback is to play the image data in sequence; The principle of motion change in video: unfolding an action can be differentiated into N static action segments; considering the unfolding as a positive sequence, reversing each action segment is the retraction of the unfolded action; reversing again at different time points will produce actions of different amplitudes; Based on the above principles, a random function is designed at the front end, and the probability of reversing the random trigger sequence is designed; This will cause the entire action sequence to produce movements of different amplitudes, forming various new movements visually; While playing the audio, the phoneme sequence generated in step S5 and the time segment in the speech corresponding to each phoneme in the sequence are used as parameters, and the correction data of the face position in the face reconstruction process is generated through the mapping relationship generated in step S402. The face position correction data replaces the face position of the portrait in the action sequence in the canvas, thereby completing the matching of the face and the speech in a single sequence frame and realizing real-time reconstruction of the face.

Images