Energy digital human interaction method, system and equipment

Abstract

The embodiment of the invention discloses an energy digital human interaction method, system and equipment, and solves the problem that a digital human model is relatively low in reality sense when the highly fine digital human model is constructed. Comprising the following steps: performing feature extraction on acquired audio data through an audio and visual synchronization audio encoder to obtain audio features; generating an inference digital human head video based on the audio features; in the reasoning digital human head video, determining a pixel value of a first key point region corresponding to the video image; wherein the first key point area is related to the five sense organs of the digital human face and the neck position; based on the pixel values, the reasoning digital human head video is pasted back to an original digital human video corresponding to the digital human model, and a voice-driven digital human video is obtained; and performing facial beautification processing on the digital human in the voice-driven digital human video so as to perform interaction through the digital human after facial beautification processing.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an energy digital human interaction method, system and device. Background Art

[0002] In the digital age, computer vision technology is revolutionizing our visual experience and interaction at an unprecedented speed. Among them, Neural Radiance Fields (NeRF), as a revolutionary technology, is leading the field of 3D reconstruction to a new level. It uses deep learning technology to extract the geometric shape and texture information of objects from images from multiple perspectives, and then uses this information to generate a continuous 3D radiation field, which can present highly realistic 3D models at any angle and distance.

[0003] However, NeRF also faces several technical challenges in the specific application of digital human model construction. First, the existing ER-NeRF model often has difficulty in achieving the ideal lip synchronization effect when processing audio. Second, when NeRF technology is used to build highly detailed digital human models, the digital human head often appears to be larger or smaller, resulting in a low sense of reality for the digital human model. Summary of the invention

[0004] The embodiments of the present application provide an energy digital human interaction method, system and device for solving the following technical problems: when NeRF technology is used to build a highly detailed digital human model, pure sound is difficult to synchronize and the digital human head tends to be larger or smaller, resulting in a low sense of reality for the digital human model.

[0005] The present application embodiment adopts the following technical solutions:

[0006] The embodiment of the present application provides an energy digital human interaction method. The method includes: extracting features from the acquired audio data through an audio and visual synchronous audio encoder to obtain audio features; generating an inferred digital human head video based on the audio features; determining the pixel value of the first key point area corresponding to the video image in the inferred digital human head video; wherein the first key point area is related to the facial features and neck position of the digital human; based on the pixel value, pasting the inferred digital human head video back to the original digital human video corresponding to the digital human model to obtain a voice-driven digital human video; and performing beauty processing on the digital human in the voice-driven digital human video to interact through the beauty-processed digital human.

[0007] The embodiment of the present application can capture key information in the audio signal through audio feature extraction. After feature extraction, different audio contents can be converted into digital human head video in real time, achieving synchronization of audio and video and improving user experience. By determining the key point areas in the video image and accurately controlling the pixel values ​​of these areas, the inferred digital human head video and the original digital human video are fused at the pixel level to solve the problem of the digital human head being too large or too small, making the generated digital human model more realistic and improving the user experience.

[0008] In one implementation of the present application, before performing feature extraction on the acquired audio data through an audio and visual synchronous audio encoder, the method also includes: obtaining user text-based question information; answering the text-based question information through a natural language processing (NLP) module to obtain reply information; segmenting the reply information based on the symbol positions in the reply information, and transmitting the segmented reply information to the answers queue in sequence according to the sentence order, so as to generate a voice-driven digital human video based on the reply information in the answers queue.

[0009] In one implementation of the present application, feature extraction is performed on the acquired audio data through an audio and visual synchronous audio encoder to obtain audio features, specifically including: training an audio and visual synchronous audio encoder based on lip-synchronized multimodal video data; inputting the acquired audio data into the audio and visual synchronous audio encoder to obtain audio features; wherein the audio features include the same feature distribution of speech features and lip movements; inputting the audio features into the basic implicit function based on hash NeRF in ER-NeRF; wherein the basic implicit function after the audio features are input includes a 3D position vector, a visual direction vector, an audio feature vector, and a multi-resolution hash encoder vector.

[0010] In one implementation of the present application, based on the pixel value, the inferred digital human head video is pasted back to the original digital human video corresponding to the digital human model to obtain a voice-driven digital human video, which specifically includes: frame image matching of the original digital human video and the inferred digital human head video; in the matched original digital human video image, determining the second key point area; based on the matched frame image, matching the first key point area with the second key point area to replace the pixel value corresponding to the second key point area with the pixel value corresponding to the first key point area, to obtain a voice-driven digital human video.

[0011] In one implementation of the present application, a second key point area is determined in the original digital human video image after matching, specifically including: in the original digital human video image after matching, a rectangular frame area of ​​a preset video size is drawn with a preset starting pixel point as the initial point; wherein the rectangular frame area is the digital human head video area; frame image face analysis is performed on the inferred digital human head video to determine the pixel value of the first key point area; wherein the first key point area includes at least the skin, left eyebrow, right eyebrow, left eye, right eye, nose, upper lip, lower lip and neck area; based on the pixel coordinate value of the first key point area, a second key point area is determined in the rectangular frame area.

[0012] In one implementation of the present application, a digital human in a voice-driven digital human video is beautified so that the digital human can interact with the user through the beautified digital human, specifically including: detecting the digital human face area and face key points through a YOLO model; mapping the skin color to a qualified color range through a color lookup table; and / or determining a skin mask through a skin smoothing high-contrast retention algorithm, and performing acceptance and lightening processing on the corresponding area of ​​the digital human frame image according to the detail area in the mask; and / or enlarging the eye area and performing face slimming processing on the digital human frame image based on face key points, a large eye algorithm and a local translation method.

[0013] In one implementation of the present application, based on the face key points, the large eye algorithm and the local translation method, the frame image of the digital human is enlarged in the eye area and thinned in the face, specifically including: based on the face key points, determining the center point of the left eye and the center point of the right eye; based on the face key points, using the distance between the center point of the left eye and the first preset key point as the first radius, and using the distance between the center point of the right eye and the second preset key point as the second radius; enlarging the left eye based on the first radius, and enlarging the right eye based on the second radius; based on the face key points, determining the distance between the third preset key point and the fourth preset key point as the left face thinning distance, and thinning the left face of the digital human based on the left face thinning distance; based on the face key points, determining the distance between the fifth preset key point and the sixth preset key point as the right face thinning distance, and thinning the right face of the digital human based on the right face thinning distance.

[0014] The embodiment of the present application provides an energy digital human interaction system, including: a text-to-speech TTS module, an audio feature extraction module, an ernerf algorithm reasoning module and a post-processing module, and the multiple modules are all set up as independent threads; the text-to-speech TTS module is used to generate corresponding audio data from the acquired reply information; the audio feature extraction module is used to extract features from the audio data to obtain audio features consistent with the lip movements and speech features; the ernerf algorithm reasoning module is used to train the digital human model; the post-processing module is used to paste the inferred digital human head video generated based on the audio data back to the digital human model to obtain a voice-driven digital human video, and perform beauty processing on the generated voice-driven digital human video.

[0015] In one implementation of the present application, the system also includes an answers queue, an audios queue, an audiofeatures feature queue, an ernerfs queue and a results queue; the answers queue is used to store the reply information of the natural language processing (NLP) module; the audios queue is used to store the voice files synthesized by the text-to-speech (TTS) module; the audiofeatures feature queue is used to store the audio features extracted by the audio feature extraction module; the ernerfs queue is used to store the generated inference digital human head video; the results queue is used to store the video pasted back to the original digital human model.

[0016] An embodiment of the present application provides an energy digital human interaction device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can: extract features from acquired audio data through an audio and visual synchronous audio encoder to obtain audio features; generate an inferred digital human head video based on the audio features; in the inferred digital human head video, determine the pixel values ​​of a first key point area corresponding to the video image; wherein the first key point area is related to the digital human's facial features and neck position; based on the pixel values, paste the inferred digital human head video back to the original digital human video corresponding to the digital human model to obtain a voice-driven digital human video; perform beauty processing on the digital human in the voice-driven digital human video so as to interact through the beauty-processed digital human.

[0017] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: the embodiments of the present application can capture key information in the audio signal through audio feature extraction, and different audio contents can be converted into digital human head video in real time after feature extraction, so as to achieve synchronization between audio and video and improve user experience. By determining the key point areas in the video image and accurately controlling the pixel values ​​of these areas, the inferred digital human head video and the original digital human video are fused at the pixel level to solve the problem of the digital human head being larger or smaller, making the generated digital human model more realistic and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor.

[0019] In the figure:

[0020] Figure 1 A schematic diagram of an energy digital human interaction system provided in an embodiment of the present application;

[0021] Figure 2 A flow chart of an energy digital human interaction method provided in an embodiment of the present application;

[0022] Figure 3 A schematic diagram of a digital human interaction process provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of a digital human model provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of face analysis provided in an embodiment of the present application;

[0025] Figure 6 A key point detection diagram provided in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of a digital human after beautification provided in an embodiment of the present application;

[0027] Figure 8 A schematic diagram of the structure of an energy digital human interaction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The embodiments of the present application provide an energy digital human interaction method, device and medium.

[0029] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0030] The technical solution proposed in the embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0031] Figure 1 A schematic diagram of an energy digital human interaction system provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the system includes a text-to-speech TTS module, an audio feature extraction module, an ernerf algorithm reasoning module and a post-processing module, and the multiple modules are all set up as independent threads. The text-to-speech TTS module is used to generate corresponding audio data from the acquired reply information. The audio feature extraction module is used to extract features from the audio data to obtain audio features that are consistent with the lip movements and speech features. The ernerf algorithm reasoning module is used to train the digital human model. The post-processing module is used to paste the inferred digital human head video generated based on the audio data back to the digital human model to obtain a voice-driven digital human video, and to perform beauty processing on the generated voice-driven digital human video.

[0032] Specifically, the digital human reasoning template in the embodiment of the present application defines a digital human DigitalHuman, which is composed of five queues and four modules. The first-in-first-out feature of the queue ensures the order of the digital human video.

[0033] The four modules include: text-to-speech (TTS) module, audio feature extraction module, Ernerf algorithm inference module, and post-processing module. These four modules are four independent threads. As long as their input queues are not empty, the threads will execute; if the input queues are empty, the threads will wait.

[0034] Furthermore, the system also includes an answers queue, an audios queue, an audio features queue, an ernerfs queue, and a results queue. The answers queue is used to store the reply information of the natural language processing (NLP) module. The audios queue is used to store the voice files synthesized by the text-to-speech (TTS) module. The audio features queue is used to store the audio features extracted by the audio feature extraction module. The ernerfs queue is used to store the generated inference digital human head video. The results queue is used to store the video pasted back to the original digital human model.

[0035] Specifically, the answers queue of the embodiment of the present application stores the answers of NLP; the audios queue stores the synthesized voice files; the audio features queue stores the extracted audio features; the ernerfs queue stores the videos generated by reasoning; and the results queue stores the videos pasted back to the original images.

[0036] Furthermore, when multiple users ask questions at the same time, the server will generate a digital human instance for each user to ensure that the server can handle questions from multiple users at the same time; when the user withdraws, the server will destroy the instance and release resources in time.

[0037] Figure 2 A flow chart of an energy digital human interaction method provided in an embodiment of the present application, such as Figure 2 As shown, the energy digital human interaction method includes the following steps:

[0038] S101. Perform feature extraction on the acquired audio data through an audio and visual synchronous audio encoder to obtain audio features.

[0039] In one embodiment of the present application, a user's textual question information is obtained. The textual question information is answered by a natural language processing (NLP) module to obtain answer information. The answer information is segmented based on the symbol positions in the answer information, and the segmented answer information is sequentially transmitted to the answers queue in the order of sentences, so as to generate a voice-driven digital human video based on the answer information in the answers queue.

[0040] Specifically, the user input text questions are received through the user interface, and the intent and context of the questions are understood through semantic analysis and other technologies. According to the type of question, relevant information is obtained from the knowledge base, database or through web search, or reasoning is performed using a deep learning model to convert the obtained information or reasoning results into a natural language response. Identify punctuation marks (such as periods, question marks, exclamation marks, etc.) in the response. Based on the position of these symbols, the response information can be segmented into multiple independent sentences. The segmented response information is placed in the answers queue in the order in which they appear in the original text. This queue will serve as the input data source for the subsequent generation of voice-driven digital human videos.

[0041] Furthermore, the digital human interaction process is as follows:

[0042] (1): The user asks questions in text form, such as: The heating is not hot. The embodiment of the present application also supports the user to ask questions in voice form. After capturing the user's audio, the voice is converted into text using speech recognition ASR technology.

[0043] (2): The NLP module gives an answer. Example answer: Regarding the heating not being hot: You can follow the Thermal Power Group WeChat official account and click on the "Warm Service-Practical Video" to see how to eliminate the problem of gas accumulation in floor heating that causes part of the user's home to not be hot, and how to vent the wall-mounted radiator.

[0044] (3): The above answers are segmented according to different punctuation marks: ".|;", etc., and then put into the answers queue. The digital human reasoning module performs reasoning to generate a series of videos.

[0045] (4): Take out the processed digital human videos from the results video queue one by one and transmit them to the user.

[0046] In one embodiment of the present application, an audio and visual synchronization audio encoder is trained based on lip synchronization multimodal video data. The acquired audio data is input into the audio and visual synchronization audio encoder to obtain audio features; wherein the audio features include the same feature distribution of speech features and lip movements. The audio features are input into the basic implicit function based on hash NeRF in ER-NeRF; wherein the basic implicit function after the audio features are input includes a 3D position vector, a visual direction vector, an audio feature vector, and a multi-resolution hash encoder vector.

[0047] Specifically, when training the nerf model, the size of the training video is 512x512, and only contains the head and neck area of ​​the digital human.

[0048] Before training the ernerf model, the training video needs to be preprocessed, including extracting the audio of the training video and generating audio features, extracting training video frame images, face and torso analysis, and face tracking.

[0049] In ER-NeRF, faces are tracked based on the 3DMM (3D Morphable Model) algorithm. 3DMM is a general three-dimensional face model that can be constructed by learning a set of well-controlled 2D face images. However, 3DMM is prone to large errors when processing Asian faces. 3DDFA (3D Dense Face Alignment) has made a series of improvements and optimizations based on 3DMM and is an advanced three-dimensional face alignment technology. 3DDFA uses Cascaded CNN to fit the 3D morphological model, which enables the model to adapt to the shape and expression changes of the face more accurately. 3DDFA pays special attention to the stability in the video face alignment task, and proposes a three-dimensional assisted short video synthesis technology, which enhances the stability and continuity of the model on video data by converting static images into short videos. 3DDFA shows some advantages over 3DMM when modeling Asian faces. Therefore, in the preprocessing stage, the embodiment of the present application uses the latest version of 3DDFA to replace the 3DMM algorithm.

[0050] The embodiment of the present application requires a feature distribution from audio to lip movements. Therefore, an audio and visual synchronization audio encoder is trained based on 155h lip synchronization multimodal video data so that the extracted speech features and lip movements have the same feature distribution. After the speech passes through the audio and visual synchronization audio encoder, the audio features are obtained, which are recorded as l.

[0051] The basic implicit function based on hash NeRF in ER-NeRF is:

[0052] F(x,d,a,H)-->(c,\sigma)

[0053] becomes

[0054] F(x,d,l,H)-->(c,\sigma)

[0055] Among them, x is the 3D position vector, d is the visual direction vector, a is the speech feature, l is the audio feature vector, the audio feature is the feature of synchronization of sound and video, and H is the multi-resolution hash encoder vector.

[0056] S102: Generate an inferred digital human head video based on audio features.

[0057] In one embodiment of the present application, after acquiring the speech feature, the audio feature corresponding to the speech feature is obtained through the trained audio and visual synchronous audio encoder, and the audio feature can achieve the consistency between the lip movements of the digital human and the speech feature. Therefore, the inference digital human head video can be generated based on the audio feature.

[0058] Usually, when training a digital human model, a 5-minute head video with fps (frames per second) = 25 and a resolution of 512x512 is required. Therefore, based on the audio features, the video inferred by the digital human model is also a head video.

[0059] Figure 3 A schematic diagram of a digital human interaction process provided in an embodiment of the present application, such as Figure 3 As shown in the figure, the user asks a question in text form or in voice form. If it is a voice question, it is converted into a text question through ASR (Automatic Speech Recognition). Secondly, the NLP module gives an answer, and the answer text is segmented by preset symbols. The segmented answer is input into the digital human reasoning model to obtain a digital human video, which is stored in the results video queue and transmitted to the user in sequence.

[0060] S103: In the inferred digital human head video, determine the pixel value of the first key point area corresponding to the video image.

[0061] In one embodiment of the present application, the video inferred by the digital human model based on a given voice is a head video. However, in the application scenario, the digital human is usually in the form of half or full body, so it is also necessary to paste the digital human video generated by voice drive back to the original image.

[0062] The model training material in the embodiment of the present application is usually a full-body photo of a model. Figure 4 A schematic diagram of a digital human model provided in an embodiment of the present application, such as Figure 4 As shown in the figure, a full-body photo of a model is used as training material. Then a 512x512 area of ​​the model's head is cut out, that is, Figure 4 The head area marked by the rectangular box in the middle is used as training material. Given a voice, the size of the voice-driven digital human video is also 512x512. When the existing technology uses the NeRF algorithm to process Asian faces, facetracking will have errors, resulting in the problem of the head being larger or smaller. Moreover, methods such as NeRF cannot handle target portraits with long hair well. If the generated digital human video is pasted back frame by frame, first, the problem of the head being larger or smaller will appear; second, if the model has long hair, the effect will be very poor.

[0063] Based on the above problems, the embodiment of the present application uses face parsing technology to paste the generated video image back to the original image, that is, paste it back to the training material image. The difference between the digital human generated by voice drive and the original image lies in the eyes and mouth shape. When pasting back, the eyes and mouth shape of the synthesized digital human will be very different from the original image. Pasting it back rashly will cause ghosting problems, such as double chins.

[0064] Therefore, the embodiment of the present application considers the area around the eyes and mouth, and determines nine areas of skin, left eyebrow, right eyebrow, left eye, right eye, nose, upper lip, lower mouth, and neck in the area, which are used as the first key point area, and determines the pixel value of the first key point area corresponding to each video image in the inferred digital human head video. Figure 5 A face analysis schematic diagram provided in an embodiment of the present application, such as Figure 5 As shown, the digital human face is divided into nine regions, namely, skin, left eyebrow, right eyebrow, left eye, right eye, nose, upper lip, lower lip, and neck.

[0065] S104: Based on the pixel value, the inferred digital human head video is pasted back to the original digital human video corresponding to the digital human model to obtain the voice-driven digital human video.

[0066] In one embodiment of the present application, the original digital human video and the inferred digital human head video are frame-matched. In the matched original digital human video image, a second key point region is determined. Based on the matched frame image, the first key point region is matched with the second key point region to replace the pixel value corresponding to the second key point region with the pixel value corresponding to the first key point region, and obtain the voice-driven digital human video.

[0067] The generated digital human video uses face parsing technology to detect nine types of areas, including skin, left eyebrow, right eyebrow, left eye, right eye, nose, upper lip, lower lip, neck, etc., and then replaces the pixel values ​​of these parts of the original image with the generated digital human pixel values, so that a generated voice-driven digital human video can be obtained.

[0068] Specifically, corresponding video frames are extracted from the original digital human video and the inferred digital human head video. These frames are static images in the video, and the same frame images are matched. Secondly, nine areas of skin, left eyebrow, right eyebrow, left eye, right eye, nose, upper lip, lower lip, and neck are determined in the matched original digital human video image, which are used as the second key point area, and the pixel value of the first key point area corresponding to the inferred digital human head video is determined. The pixel value of the second key point area in the original digital human video frame is replaced with the corresponding pixel value of the first key point area extracted from the inferred digital human head video frame. To ensure that the replaced pixel value is visually coordinated with other parts of the original video frame. The above-mentioned process of frame image matching, key point area determination and pixel value replacement is repeated for each frame in the original digital human video, and each processed frame image is reassembled into a video file according to the frame rate of the original video. In this way, a voice-driven digital human video is obtained, in which the head movement and expression of the digital human are driven by the inferred digital human head video.

[0069] In one embodiment of the present application, in the matched original digital human video image, a preset starting pixel point is used as the initial point to draw a rectangular frame area of ​​a preset video size; wherein the rectangular frame area is the digital human head video area. The frame image face analysis is performed on the inferred digital human head video to determine the pixel value of the first key point area; wherein the first key point area at least includes the skin, left eyebrow, right eyebrow, left eye, right eye, nose, upper lip, lower lip and neck area. Based on the pixel coordinate values ​​of the first key point area, the second key point area is determined in the rectangular frame area.

[0070] Among them, in the original digital human video image, the process of determining the second key point area is: in the original digital human video image, take the position of the upper left corner of the head as the initial pixel point, combine the preset video size (width and height), and draw a rectangular frame on the original video frame. This rectangular frame should be able to completely cover the head area of ​​the digital human, including the skin, facial features, neck and other parts. The size of the training video selected in the embodiment of the present application is 512x512. Based on the first key point area determined by the inference digital human head video, the corresponding second key point area is determined in the rectangular frame of the original digital human video image.

[0071] For example, assuming that the original video reference frame is tmp, the starting pixel of the rectangular frame is start_pix, and the inference video frame is frame, then the original video frame image corresponding to frame is:

[0072] tmp[start_pix[1]:start_pix[1]+512,start_pix[0]:start_pix[0]+512];

[0073] The face parsing of the inference video frame is performed to obtain parsing, which is a 512x512 matrix, in which each element is the category value of each pixel in img. Then the positions of nine categories in frame, including skin, left eyebrow, right eyebrow, left eye, right eye, nose, upper lip, lower lip, and neck, are found respectively, and the pixel values ​​of these positions in tmp are replaced with the values ​​of the corresponding pixels in frame.

[0074] S105: Perform beauty processing on the digital human in the voice-driven digital human video, so as to interact with the digital human through the beauty-processed digital human.

[0075] In one embodiment of the present application, the digital human face area and facial key points are detected by the YOLO model. The skin color is mapped to a color range that meets the conditions by means of a color lookup table, and / or a skin mask is determined by a skin smoothing high contrast retention algorithm, and the corresponding area of ​​the digital human frame image is subjected to acceptance and lightening processing according to the detail area in the mask; and / or the eye area of ​​the digital human frame image is enlarged and the face is thinned based on the facial key points, the large eye algorithm and the local translation method.

[0076] Specifically, in digital human processing, the YOLO model is trained to identify the face area and face key points in the image. The embodiment of the present application uses the YOLO model to detect the face position and 106 face key points. Figure 6 A key point detection diagram provided in an embodiment of the present application, such as Figure 6 As shown in the figure, the position of the key points is related to the facial features. Skin whitening uses a color lookup table to map the skin color to the ideal color range. Given any RGB color value, the corresponding color value can be found in the color lookup table image and the corresponding conversion result can be interpolated. The high-contrast retention algorithm for skin resurfacing refers to obtaining a mask of skin details through high contrast. According to the detailed area in the mask, such as the location of the spot area in the skin, the corresponding area of ​​the original image is lightened to achieve the purpose of spot weakening and skin beautification. According to the determined facial key points, the large eye algorithm is used to enlarge the area around the center of the eye, and the face is thinned based on the local translation method.

[0077] In one embodiment of the present application, based on the key points of the face, the center point of the left eye and the center point of the right eye are determined. Based on the key points of the face, the distance between the center point of the left eye and the first preset key point is used as the first radius, and the distance between the center point of the right eye and the second preset key point is used as the second radius. The left eye is enlarged based on the first radius, and the right eye is enlarged based on the second radius. Based on the key points of the face, the distance between the third preset key point and the fourth preset key point is determined as the left face thinning distance, and the left face of the digital person is thinned based on the left face thinning distance. Based on the key points of the face, the distance between the fifth preset key point and the sixth preset key point is determined as the right face thinning distance, and the right face of the digital person is thinned based on the right face thinning distance.

[0078] Specifically, when enlarging the eyes, the eye area is enlarged outward with the center of the eye as the center point, so as to achieve the enlarged eye effect. For example, in the embodiment of the present application, the center point of the left eye is the 74th point, and the distance between the 74th point and the 52nd point is used as the radius. The center point of the right eye is the 77th point, and the distance between the 77th point and the 61st point is used as the radius. By determining the radius and the center point, the left and right eyes are enlarged respectively.

[0079] Specifically, the embodiment of the present application performs face thinning processing based on the determined key points of the face. For example, the distance from the 10th point to the 81st point is calculated as the distance for thinning the left face. The distance from the 22nd point to the 80th point is calculated as the distance for thinning the right face, and the left and right faces are thinned based on the local translation method.

[0080] Figure 7 A schematic diagram of a digital human after beautification provided in an embodiment of the present application, such as Figure 7 As shown, the digital human image after beautification is more beautiful, thereby improving the user experience.

[0081] Figure 8 This is a schematic diagram of the structure of an energy digital human interaction device provided in an embodiment of the present application. Figure 8As shown, an energy digital human interaction device includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can: extract features from the acquired audio data through an audio and visual synchronous audio encoder to obtain audio features; generate an inferred digital human head video based on the audio features; in the inferred digital human head video, determine the pixel value of the first key point area corresponding to the video image; wherein the first key point area is related to the digital human's facial features and neck position; based on the pixel value, paste the inferred digital human head video back to the original digital human video corresponding to the digital human model to obtain a voice-driven digital human video; perform beauty processing on the digital human in the voice-driven digital human video so as to interact through the beauty-processed digital human.

[0082] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0083] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application may have various modifications and variations. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application.

Claims

1. An energy digital human interaction method, characterized in that: The method comprises: The acquired audio data is subjected to feature extraction through an audio and visual synchronous audio encoder to obtain audio features; Generate an inferred digital human head video based on the audio features; In the inferred digital human head video, pixel values ​​of a first key point area corresponding to the video image are determined; wherein the first key point area is related to the facial features and neck position of the digital human; Based on the pixel value, pasting the inferred digital human head video back to the original digital human video corresponding to the digital human model to obtain the voice-driven digital human video; The digital human in the voice-driven digital human video is beautified so as to interact with the user through the beautified digital human.

2. The energy digital human interaction method according to claim 1, characterized in that: Before extracting features from the acquired audio data by the audio and visual synchronous audio encoder, the method further comprises: Get user question information in text form; Answer the text-based question information through a natural language processing (NLP) module to obtain answer information; Based on the symbol positions in the reply information, the reply information is segmented, and the segmented reply information is transmitted to the answers queue in sequence according to the sentence order, so as to generate the voice-driven digital human video based on the reply information in the answers queue.

3. The energy digital human interaction method according to claim 1, characterized in that: The method of extracting features from the acquired audio data by using an audio and visual synchronous audio encoder to obtain audio features specifically includes: Based on lip-sync multimodal video data, training the audio and visual synchronization audio encoder; Input the acquired audio data into the audio and visual synchronous audio encoder to obtain audio features; wherein the audio features include the same feature distribution of speech features and lip movements; The audio features are input into a basic implicit function based on hash NeRF in ER-NeRF; wherein the basic implicit function after the audio features are input includes a 3D position vector, a visual direction vector, an audio feature vector and a multi-resolution hash encoder vector.

4. The energy digital human interaction method according to claim 1, characterized in that: Based on the pixel value, pasting the inferred digital human head video back to the original digital human video corresponding to the digital human model to obtain the voice-driven digital human video specifically includes: Performing frame image matching between the original digital human video and the inferred digital human head video; In the matched original digital human video image, determining a second key point area; Based on the matched frame image, the first key point area is matched with the second key point area to replace the pixel values ​​corresponding to the second key point area with the pixel values ​​corresponding to the first key point area to obtain the voice-driven digital human video.

5. The energy digital human interaction method according to claim 4, characterized in that: The step of determining a second key point area in the matched original digital human video image specifically includes: In the matched original digital human video image, a rectangular frame area of ​​a preset video size is drawn with a preset starting pixel point as the initial point; wherein the rectangular frame area is the digital human head video area; Performing frame image face analysis on the inferred digital human head video to determine pixel values ​​of a first key point area; wherein the first key point area at least includes skin, left eyebrow, right eyebrow, left eye, right eye, nose, upper lip, lower lip and neck area; Based on the pixel coordinate values ​​of the first key point area, the second key point area is determined in the rectangular frame area.

6. The energy digital human interaction method according to claim 1, characterized in that: The beautifying the digital human in the voice-driven digital human video so as to interact with the digital human after the beautification processing specifically includes: Detect digital face areas and facial key points through the YOLO model; Mapping skin color to a qualifying color range by means of a color lookup table; and / or Determine a skin mask by using a skin grinding high contrast retention algorithm, and perform acceptance and lightening processing on the corresponding area of ​​the frame image of the digital human according to the detail area in the mask; and / or Based on the facial key points, the large eye algorithm and the local translation method, the frame image of the digital human is enlarged in the eye area and thinned in the face.

7. The energy digital human interaction method according to claim 6, characterized in that: The enlarging the eye area and performing face thinning processing on the frame image of the digital human based on the facial key points, the large eye algorithm and the local translation method specifically includes: Based on the facial key points, determining the left eye center point and the right eye center point; Based on the facial key points, the distance between the left eye center point and the first preset key point is used as the first radius, and the distance between the right eye center point and the second preset key point is used as the second radius; Enlarging the left eye based on the first radius, and enlarging the right eye based on the second radius; Based on the facial key points, determining the distance between the third preset key point and the fourth preset key point as a left face slimming distance, and performing face slimming processing on the left face of the digital human based on the left face slimming distance; Based on the facial key points, the distance between the fifth preset key point and the sixth preset key point is determined as the right face thinning distance, and the right face of the digital human is thinned based on the right face thinning distance.

8. An energy digital human interaction system, characterized in that: The system includes a text-to-speech (TTS) module, an audio feature extraction module, an ernerf algorithm reasoning module, and a post-processing module, and a plurality of the modules are all independent thread settings; The text-to-speech (TTS) module is used to generate corresponding audio data from the acquired reply information; The audio feature extraction module is used to extract features from the audio data to obtain audio features consistent with lip movements and speech features; The ernerf algorithm reasoning module is used to train the digital human model; The post-processing module is used to paste the inferred digital human head video generated based on the audio data back to the digital human model to obtain a voice-driven digital human video, and perform beauty processing on the generated voice-driven digital human video.

9. The energy digital human interaction system according to claim 8, characterized in that: The system also includes an answers queue, an audios queue, an audio features queue, an ernerfs queue and a results queue; The answers queue is used to store the reply information of the natural language processing (NLP) module; The audios queue is used to store the voice files synthesized by the text-to-speech TTS module; The audio features feature queue is used to store the audio features extracted by the audio feature extraction module; The ernerfs queue is used to store the generated inference digital human head video; The results queue is used to store the video pasted back to the original digital human model.

10. An energy digital human interaction device, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: The acquired audio data is subjected to feature extraction through an audio and visual synchronous audio encoder to obtain audio features; Generate an inferred digital human head video based on the audio features; In the inferred digital human head video, pixel values ​​of a first key point area corresponding to the video image are determined; wherein the first key point area is related to the facial features and neck position of the digital human; Based on the pixel value, pasting the inferred digital human head video back to the original digital human video corresponding to the digital human model to obtain the voice-driven digital human video; The digital human in the voice-driven digital human video is beautified so as to interact with the user through the beautified digital human.

Images