2D government affair digital human real-time interaction method and system based on AI drawing technology

Abstract

The invention discloses a 2D government affair digital human real-time interaction method and system based on an AI painting technology, and the method comprises the steps: making a digital human image based on a modeling mode combining AI painting and a Live2D technology; acquiring an audio input by a user in real time, and identifying the audio input by the user to obtain an audio identification text and a reply text; according to the reply text, synthesizing and outputting a result audio based on a half-stream speech synthesis method of dependency syntactic analysis; and performing mouth shape synchronous real-time driving and audio broadcasting of the digital human image on the manufactured digital human image based on the result audio. According to the invention, the low-cost construction and deployment of the digital human are realized based on the combination of AI drawing and Live2D technology; when system deployment is carried out on the digital people, a semi-streaming speech synthesis scheme based on dependency syntactic analysis is adopted, and the interaction real-time performance and fluency of government affair digital people belonging to service digital people are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of government digital humans, and in particular to a real-time interaction method and system for 2D government digital humans based on AI painting technology. Background Art

[0002] With the continuous maturity of digital technology, AI virtual digital people are able to enhance the integrated government service capabilities and expand the breadth and depth of digital government platform applications. Compared with traditional manual window services, government digital people can provide 24-hour uninterrupted online services, combined with the most mainstream RAG framework technology of large models + knowledge bases, to meet core needs including but not limited to online policy consultation, intelligent service guidance, and online application and approval of matters. This not only improves the efficiency of government service halls, but also saves the time cost of people doing things, providing strong help for the digital transformation of contemporary government affairs.

[0003] When constructing digital humans, digital human technology is currently divided into two categories according to different dimensions: 2D digital humans and 3D digital humans. 2D digital humans are generally digital humans that are mainly displayed in the form of 2D images or videos. Their production costs are relatively low and their production cycles are short. 3D digital humans are characters created using three-dimensional modeling and rendering technology. Their realism and details are often higher and richer, but their production costs are high and their production cycles are long. In the field of 2D digital humans, the most commonly used method is the computation-driven method, that is, based on a 2D real-life video and the audio output by the speech synthesis model, the relevant deep learning model such as the generative adversarial network GAN is used to clone the characters in the input video based on the input audio to obtain the final virtual digital human video and output it. In addition, live2D technology is also commonly used to construct 2D digital humans. Unlike the computation-driven method, live2D technology is usually based on different layers of the original painting drawn by the original artist, allowing users to manually define key points and create keyframe animations, and finally control the display of different keyframe animations based on the phonemes of the input audio to complete real-time rendering.

[0004] When deploying digital humans, ensuring real-time voice interaction with users is the core issue that needs to be solved after building digital humans. The currently commonly used method is to use a real-time voice interaction solution using streaming technology, that is, streaming the reply text output by a large model, using length or punctuation as boundaries to segment the reply text, and inputting the segmented sentences into the speech synthesis model for semi-streaming output.

[0005] The above-mentioned digital human construction and real-time voice interaction methods can generate digital humans, but the following problems still exist:

[0006] (1) The rendering effect of 2D digital humans built using a computationally driven approach is unstable, has poor real-time performance, and requires high hardware computing power. The rendering effect of digital humans using a computationally driven approach is deeply bound to the training effect of the deep learning model, and because the model depends on the data features of the training set, it will result in poor freedom and lack of details when rendering digital human animations, and it is difficult to adapt to audio accents that do not appear in the training set. At the same time, real-time rendering under this driving approach usually requires very high GPU computing power, especially when running on mobile devices. Real-time driving of high-quality facial animations may still face challenges, which to a certain extent raises the hardware threshold for the deployment of such digital humans.

[0007] (2) Although the live2D technology using manual drive can further improve the real-time rendering efficiency of digital humans by presetting keyframe animations in advance, its production cost and threshold are relatively higher than those of computationally driven 2D digital humans. It requires the ability to draw clear and delicate original paintings. At the same time, its general construction method is more suitable for building cartoon or illustration style characters, and its character expression modeling requires high precision, and it is more inclined to build identity-type digital humans for entertainment and social interaction. In government affairs scenarios, it is more inclined to service-oriented digital humans, and the image of digital humans is more realistic. Its core function is to replace real-life services and output reliable service content stably and quickly.

[0008] (3) Although the real-time voice interaction solution using streaming technology can achieve semi-streaming output of synthesized audio by segmenting the reply text, it needs to set the segmentation length according to the actual situation or use punctuation marks as boundaries to obtain segmented sentences when segmenting the text. In addition, the segmented sentences may not have a complete syntactic structure or be too long, which will affect the throughput efficiency of speech synthesis and further affect the coherence and real-time performance of the entire voice response of the digital human. At the same time, when the digital human is performing voice interaction, there may be a long pause in the digital human because the speech synthesis model temporarily fails to output the synthesized audio in time, thereby affecting the fluency of the digital human's response.

[0009] Therefore, when building and deploying existing government digital humans, there are problems such as high deployment costs and production thresholds, poor real-time and continuity of voice responses, and easy lags that cause digital humans to pause for long periods of time. Summary of the invention

[0010] The technical problem to be solved by the present invention is that the existing government digital human has high deployment cost and production threshold when it is constructed and deployed, the real-time and continuity of voice response are poor, and it is easy to cause the digital human to pause for a long time. The purpose of the present invention is to provide a real-time interaction method and system for 2D government digital human based on AI painting technology, and to achieve low-cost deployment of digital human based on the combination of AI painting and Live2D technology, and to further achieve rapid production of facial expressions and lip shapes by using the glue function of live2D; when the digital human is systematically deployed, a semi-stream speech synthesis scheme based on dependency syntax analysis is adopted to improve the real-time and smoothness of the interaction of government digital human belonging to the service type.

[0011] The present invention is achieved through the following technical solutions:

[0012] In a first aspect, the present invention provides a 2D government digital human real-time interaction method based on AI painting technology, the method comprising:

[0013] Create digital human images based on a modeling method that combines AI painting with Live2D technology;

[0014] Acquire user input audio in real time, recognize the user input audio, and obtain audio recognition text and reply text;

[0015] According to the reply text, the semi-stream speech synthesis method based on dependency syntactic analysis is used to synthesize the output result audio;

[0016] Based on the result audio, the lip sync of the digital human image is driven in real time and the audio is broadcasted.

[0017] Furthermore, based on the modeling method combining AI painting and Live2D technology, digital human images are produced, including:

[0018] Obtaining input prompt words to define the image of the digital human, the input prompt words including face orientation, hair, and eye details;

[0019] According to the input prompt words, the AI ​​painting model is used to control the generation of the original painting of the digital human;

[0020] Use super-resolution models to further improve the quality of digital human original paintings and generate high-resolution original paintings;

[0021] Use Photoshop to split and layer the original picture to get multiple layers;

[0022] Import multiple layers into live2D software, draw key points for key layers, and use the glue function to bond the layers together, realize blinking animation and five vowel lip shape animation, model the digital human image, and output the digital human image engineering file.

[0023] Furthermore, Photoshop tools are used to split and layer the original picture, including:

[0024] The entire face, eyes, mouth, neck and hair in the original painting were separated and layered using PhotoShop tools. The eyes include eye sockets, eyeballs and whites of the eyes; the mouth includes the upper lip, lower lip and chin; the face is the remaining part without the eyes and mouth.

[0025] Furthermore, the user input audio is obtained in real time, and the user input audio is recognized to obtain the audio recognition text and the reply text, including:

[0026] Get user input audio in real time based on the web front end;

[0027] The streaming speech recognition model FunASR is used to recognize user input audio in real time and output audio recognition text;

[0028] All the contents of the audio recognition text are input into the large model, and all the contents of the audio recognition text, namely the question text, are converted into text vectors, and the matching related texts are retrieved in the policy vector knowledge base, and the answer, namely the reply text, is output based on the streaming output method.

[0029] Furthermore, according to the reply text, a semi-stream speech synthesis method based on dependency syntactic analysis is used to synthesize and output result audio, including:

[0030] The reply text output by the large model is divided into coarse-grained sentences using punctuation marks, including commas, periods, exclamation marks, question marks, and semicolons.

[0031] The dependency syntax analysis method is used to analyze the syntactic structure of the divided sentence structures to determine whether they meet the complete syntactic definition; the complete syntactic definition includes the subject-predicate structure or the subject-predicate-object structure;

[0032] If the complete syntactic definition is met, the sentence is used as the input of the speech synthesis model and stored in the data queue to be synthesized;

[0033] If not satisfied, continue to receive the reply text output by the large model until the next punctuation mark appears and repeat the above process.

[0034] Further, according to the reply text, a semi-stream speech synthesis method based on dependency syntactic analysis is used to synthesize and output the result audio, which also includes:

[0035] The speech synthesis model obtains text from the data queue and synthesizes it to obtain the result audio; the result audio is output to the synthesis audio queue of the digital human rendering module of the front-end interaction layer in a semi-streaming manner, and transition audio is inserted into the audio queue.

[0036] In a second aspect, the present invention further provides a 2D government digital human real-time interaction system based on AI painting technology, which is based on the above-mentioned 2D government digital human real-time interaction method based on AI painting technology; the system comprises:

[0037] Digital human construction module, used to create digital human images based on a modeling method that combines AI painting and Live2D technology;

[0038] Voice access module, used to obtain user input audio in real time;

[0039] The speech recognition module is used to recognize the user input audio and obtain the audio recognition text and reply text;

[0040] A speech synthesis module is used to synthesize the output result audio according to the reply text using a semi-stream speech synthesis method based on dependency syntactic analysis;

[0041] The digital human rendering module is used to drive the lip sync of the digital human image in real time and perform audio broadcasting based on the result audio.

[0042] Furthermore, the system is deployed on the implementation architecture of the front-end interaction layer, business processing layer and basic service layer from top to bottom;

[0043] The front-end interaction layer is deployed with voice access module, digital human construction module and digital human rendering module. The front-end interaction layer is responsible for rendering the digital human image to display the user's front-end interaction.

[0044] The business processing layer is deployed with speech recognition module, large model module and speech synthesis module. The business processing layer is used to process the backend user speech data, reply using the large model and synthesize audio;

[0045] The basic service layer is deployed with speech recognition models, large model modules and speech synthesis models. The basic service layer is responsible for providing basic services for various businesses in the business processing layer.

[0046] In the third aspect, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned 2D government digital human real-time interaction method based on AI painting technology is implemented.

[0047] In a fourth aspect, the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned 2D government digital human real-time interaction method based on AI painting technology is implemented.

[0048] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0049] 1. The present invention provides a real-time interactive method and system for 2D government digital humans based on AI painting technology. The method combines AI painting with Live2D technology to achieve low-cost deployment of digital humans, and uses the glue function of live2D to further achieve rapid production of facial expressions and lip shapes. When deploying digital humans in the system, a semi-stream speech synthesis solution based on dependency syntax analysis is used to improve the real-time and smoothness of interaction of government digital humans belonging to service-type digital humans.

[0050] 2. The present invention provides a real-time interactive method and system for 2D government digital human based on AI painting technology. The 2D government digital human based on AI painting technology has the ability to be deployed and operated on multiple platforms at low cost. Live2D technology is used to drive the digital human by manually setting keyframe animations, and the incoming audio sequence is automatically converted into a phoneme sequence and different preset lip animations are called to complete lip synchronization. This effectively avoids the need for GPU computing power in interactive rendering of digital human built using computing-driven methods, and realizes low-cost deployment of digital human. At the same time, thanks to the powerful compatibility of live2D technology, it can be run in scenarios such as web browsers, mobile devices, and desktop applications.

[0051] 3. The real-time interaction method and system of 2D government digital humans based on AI painting technology in the present invention, and the construction method of 2D government digital humans based on AI painting technology have higher production efficiency, lower production cost and threshold; combined with the core demand of government digital humans to provide fast and stable services, the use of open source AI painting models does not require highly customized operations on the image in combination with user needs, that is, the step of training the model with a large amount of data can be omitted, and only a small number of core key prompts (such as facial orientation, background simplification, etc.) and other personalized prompts are needed to complete the rapid production of the original painting, and then the super-resolution model can be used to further improve the resolution and delicacy of the original painting, which effectively reduces the labor cost of professional hand-drawn original paintings required by live2D when producing digital human images. At the same time, based on the written prompt words, a simple customized construction of the digital human image can be quickly completed. Furthermore, the prompt words can reduce the redundant facial details of the generated image and simplify the image background so that the original painting splitting operation can be completed faster, and the redundant facial texture layers are reduced, which further improves the production efficiency of the digital human and the rendering speed during interaction. Finally, combined with the glue function of live2D, the rapid production of facial expressions and lip shapes can be realized, making the overall facial movement details more realistic.

[0052] 4. The 2D government digital human real-time interaction method and system based on AI painting technology of the present invention, and the improvement scheme of digital human real-time voice interaction based on dependency syntax analysis and insertion of transition audio further standardize the digital human voice interaction mode, and are more intelligent. There is no need to set judgment conditions such as the length of each sentence or division based on punctuation marks. It can directly complete the automatic judgment of the syntactic structure of the input text through the dependency syntax analysis model, further improving the naturalness and coherence of the synthesized speech. At the same time, appropriate transition audio is inserted into the blank pauses that may appear in the output audio to further improve the fluency and authenticity of the digital human interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings:

[0054] Figure 1 A flowchart of constructing a digital human image based on a modeling method combining AI painting and Live2D technology in the present invention;

[0055] Figure 2 This is an implementation architecture diagram of the 2D government affairs digital human real-time interaction method based on AI painting technology of the present invention;

[0056] Figure 3 This is a flowchart of the digital human real-time speech interaction processing based on the semi-stream speech synthesis method of dependency syntax analysis of the present invention;

[0057] Figure 4 It is an overall flow chart of the real-time interaction method of 2D government digital human based on AI painting technology of the present invention;

[0058] Figure 5 This is a structural block diagram of the 2D government digital human real-time interaction system based on AI painting technology of the present invention. DETAILED DESCRIPTION

[0059] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0060] When building and deploying existing government digital humans, there are problems such as high deployment costs and production thresholds, poor real-time and continuity of voice responses, and easy pauses and freezes of digital humans.

[0061] The present invention designs a 2D government digital human real-time interaction method and system based on AI painting technology, the main purposes of which are:

[0062] 1. Reduce the deployment cost and production threshold of 2D government digital humans. By combining AI painting with Live2D technology, the AI ​​painting model can be used to automatically complete the production of original paintings. The super-resolution model can then be combined to further improve the quality and delicacy of the original paintings. This effectively reduces the labor cost of hand-painting original paintings by professionals when live2D is used to produce digital human images. At the same time, live2D technology is used to drive digital humans by manually setting keyframe animations, avoiding the need for GPU computing power for interactive rendering of digital humans built using a computing-driven method, thus achieving low-cost deployment of digital humans.

[0063] 2. To meet the core needs of government digital humans that belong to the service type to provide interactive services in real time and stably, AI painting technology is used to generate customized digital human images that meet government scenarios based on the written prompt words. Furthermore, prompt words are used to reduce redundant facial details of the generated image, such as upper and lower eyelashes, eye highlights, etc., to facilitate rapid layering of the original painting. Finally, combined with the glue function of live2D, the rapid production of facial expressions and lip shapes is further realized, which improves the production efficiency and rendering speed of digital humans, meeting the core needs of government digital humans that prefer to provide services quickly and stably.

[0064] 3. Improve the real-time and smoothness of the interaction of government digital people, which belong to the service category. When deploying digital people in the system, use a semi-streaming speech synthesis method based on dependency syntactic analysis. For the sentences output by the large model, first use punctuation marks to make a coarse-grained division, and then use the dependency syntactic analysis model to perform syntactic analysis on the divided sentence structure. If the complete syntactic definition is met, the sentence is used as the input of the speech synthesis model. This effectively improves the reasoning throughput efficiency of the speech synthesis model, further standardizes the current division rules of speech synthesis input sentences, avoids the output of single words or text audio without syntactic structure, and maintains the coherence of the overall reply speech of the digital person. Furthermore, when the synthetic audio data may not be generated when the digital person is rendered, appropriate transition audio is inserted to improve the fluency and authenticity of the digital person's speech interaction.

[0065] A real-time interactive method for 2D government digital humans based on AI painting technology, such as Figure 1 As shown in the figure, the method first inputs a prompt word to specify the image details of the digital human, including facial orientation, hair, and eye details; based on the input prompt word, an open source AI painting model (such as stable-difussion) and other open source fine-tuning models of various styles based on the model (such as an open source model of realistic style commonly used in the government affairs field) are used to control the generation of the corresponding digital human original painting; a super-resolution model is used to generate a high-resolution original painting image for the generated digital human original painting image; then, Photoshop is used to modify the entire face, eyes (eye sockets, eyeballs and whites of the eyes), mouth (upper part) and other parts of the original painting image; The face (lips, lower lip and chin), neck and hair are split into multiple layers; the split multiple layers are imported into the live2D software, key points are drawn for key layers such as eye sockets, the entire face, upper lip, lower lip and chin, the glue function is used to complete the bonding of eye sockets and facial skin, upper lip and facial skin, lower lip and chin and facial skin, and the blinking animation and five vowel lip shape animation are completed, so as to complete the rapid modeling of the live2D digital human; the audio and lip shape dynamic synchronization settings are completed in the live2D software to bind the five vowel lip shape information; finally, the digital human image engineering file is exported using the software.

[0066] like Figure 2 As shown in the figure, the implementation architecture of the real-time interaction method of 2D government digital people based on AI painting technology is divided into a front-end interaction layer responsible for rendering the digital human for front-end interactive display to users, a business processing layer for back-end user voice data processing, using large models to reply and synthesize audio, and a basic service layer responsible for providing basic services for each business of the business processing layer.

[0067] Among them, the basic service layer includes the base large model that supports the large model module (such as the open source large models of the qwen and chatglm series) and the vector policy knowledge base built using the text embedding model, supporting the speech recognition module (such as the FunASR speech recognition model) and supporting the speech synthesis module (such as the GPT-Sovits and other speech synthesis models).

[0068] The business processing layer includes a speech recognition module that supports real-time speech recognition. It is responsible for obtaining the question text by real-time reasoning of the audio input by the user through the speech recognition model. The question text is then input into the large model module built based on the policy knowledge base and the base large model. This module is responsible for converting the question text into a text vector and recalling it in the policy vector knowledge base to obtain matching related text. Finally, the large model is used to summarize and output the answer text based on the streaming output method. Finally, the answer text is input into the speech synthesis module based on the speech synthesis model.

[0069] like Figure 3 As shown, the speech synthesis module will first divide the sentence according to the punctuation marks output by the large model. When the output sentence of the large model obtained includes punctuation marks such as commas, periods, exclamation marks, question marks, semicolons, etc., the sentence before the symbol is input into the dependency syntax analysis model for syntactic structure analysis. If it meets the definition of a complete sentence, such as a subject-predicate or subject-predicate-object structure, the sentence is used as the input of the speech synthesis module; if not, the streaming text of the large model continues to be received until the next punctuation mark appears and the above process is repeated; if the speech synthesis model is inference at this time, it will be stored in the data queue first. The speech synthesis model obtains the text from the data queue and synthesizes the final audio, which is output to the synthetic audio queue of the digital human rendering module of the front-end interaction layer in the above-mentioned semi-streaming manner.

[0070] The front-end interaction layer is responsible for interacting with the front-end users, including a voice access module for accessing user voice input. This module is responsible for real-time collection of user input and inputting it into the voice recognition module of the business processing layer for processing; the digital human rendering module is responsible for real-time rendering and output of digital human actions based on the audio output of the speech synthesis module to drive the preset lip animation and other facial expression animations. At the same time, if the synthesized audio queue is empty, the transition audio pre-generated by the speech synthesis model is inserted to maintain the continuity of the digital human's real-time voice interaction.

[0071] Example 1

[0072] like Figure 4 As shown, the present invention provides a 2D government digital human real-time interaction method based on AI painting technology, which is applied to the construction and deployment of digital human images in normal scenarios. The method includes:

[0073] Step 1: Create a digital human image based on a modeling method that combines AI painting and Live2D technology;

[0074] Step 2, obtaining user input audio in real time, and recognizing the user input audio to obtain audio recognition text and reply text;

[0075] Step 3, synthesizing the output result audio according to the reply text using a semi-stream speech synthesis method based on dependency syntactic analysis;

[0076] Step 4: Based on the result audio, the lip sync of the digital human image is driven in real time and the audio is broadcast.

[0077] As a further implementation, step 1 specifically includes:

[0078] Step 11, obtaining input prompt words to define the digital human image, the input prompt words including face orientation, hair and eye details;

[0079] Step 12, according to the input prompt word, use the AI ​​painting model to control the generation of the original painting of the digital human;

[0080] Step 13, using a super-resolution model to further improve the quality of the original digital human painting and generate a high-resolution original painting image;

[0081] Step 14, using Photoshop tools to split and layer the original picture to obtain multiple layers;

[0082] Step 15, import multiple layers into live2D software, draw key points for key layers, and use the glue function to complete the bonding of each layer, realize blinking animation and five vowel lip shape animation, model the digital human image, and output the digital human image engineering file.

[0083] Specifically, in step 11, a prompt word is input, and the input prompt word includes facial orientation, hair, and eye details; specifically, including:

[0084] Write the positive prompt words "looking at viewer" and "portrait" for the AI ​​painting model to control the character's face and head orientation to look forward;

[0085] Set the positive prompt word "forehead" to highlight the character's forehead, so as to make the character have no bangs details as much as possible, reduce the difficulty of splitting the hair layer later and the difficulty of completing the face after the hairstyle is split;

[0086] Set the positive prompt words "single eyelid" and "short eyelashes" to make the character's eyelids single and minimize the length of the eyelashes so that they do not extend beyond the eye sockets, which is conducive to the rapid production of subsequent blinking animations;

[0087] The positive prompt word "open mouth" is set to complete the generation of the mouth of the original face, which is conducive to the subsequent production of mouth shape animation; at the same time, the positive prompt word "white background" is set to make the generated original background as white as possible to facilitate the subsequent background cutout production of the original painting.

[0088] Specifically, in step 12, the open source AI painting model stable-difussion or other open source models of various styles (realistic, 2.5D or two-dimensional, etc.) fine-tuned based on the model are used to generate the corresponding original painting image based on the input prompt word, and the number of iterations is set to 20 or 25, the sampling method is "DPM++2Sa", the input prompt word guidance coefficient is 7, and the resolution size of the generated image is controlled. The generated image size is generally controlled to 512x512, and can be expanded to 1024x1024 if the video memory is sufficient. Furthermore, the original painting of the mouth and lips of the character when the mouth is closed in the original painting needs to be generated twice based on a fixed random number seed of the generation model.

[0089] Specifically, in step 13, the super-resolution model R-ESRGAN4x+ is used to further increase the resolution of the generated original image quality to 2048x2048.

[0090] Specifically, in step 14, the pen tool of Photoshop software is used to split the original image to obtain at least the eyes including eye sockets, whites of eyes, and eyeballs, the mouth including upper lip, lower lip and chin, and the entire facial skin part of the eyes and mouth, as well as the hair, neck and other layers are removed.

[0091] Specifically, step 15 includes:

[0092] Import all the layer files (.psd format) in step 14 into the live2D software, automatically generate key points for all layers, use the glue function to glue the key points of the eye socket part of the eye with the key points of the facial skin around the eye socket so that when dragging the eye socket to perform animations such as blinking, the surrounding facial skin can be driven by the glue function. The key points of the upper lip of the mouth and the key points of the surrounding facial skin nearby, as well as the key points of the lower lip and chin and the surrounding facial skin, are glued together, so that when making lip animations later, they can drive the surrounding facial skin to add real details when making actions such as opening the mouth and pouting.

[0093] In the live2D software, a bending deformer is established based on the eye socket, and the Bezier curve parameters are set to 5x5 splits and 2x2 divisions. Two key frames are set, the first key frame is open eyes, and the second key frame is completed by dragging the Bezier curve to complete the eye shape of the closed eyes to complete the blinking animation; two bending deformers are established for the upper lip, lower lip and chin, and the Bezier curve parameters are set to 5x5 splits and 2x2 divisions. Then, the Bezier curve is dragged to complete the production of three key frames for the lip opening and closing animation, which are the default closed mouth, half-open lips and wide open mouth, two key frames for lip deformation, which are the default closed mouth and clenched teeth, and two key frames for the lip pouting animation. , two keyframes for the default closed mouth and pouting, and O-shaped mouth animation are produced, respectively. Then the dynamic synchronization setting function of the live2D software is enabled, the sampling rate is set to 30.0, the smoothing rate is set to 60, and the above four keyframe animations are bound to complete the setting of five vowel lip shape animations, including type A (wide open mouth, set the lip opening and closing animations to the wide open mouth state), type O (lips are in O shape, set the O-shaped mouth animation to the O-shaped mouth state), type I (upper and lower lips are closed, the mouth is in a straight line, and the lip deformation animation is set to the clenched teeth state), type U (set the lip pouting animation to the pouting state) and the last type E (set the lip opening and closing animation to the half-open lips state).

[0094] Enable the Edit Texture Set function in the Modeling Options in the live2D software, add all the original paintings in layers to get the modeling texture set, and finally select Export as .moc3 file in the Export Runtime File option to get the final digital human image engineering file.

[0095] As a further implementation, the user input audio is obtained in real time, and the user input audio is recognized to obtain the audio recognition text and the reply text, including:

[0096] Get user input audio in real time based on the web front end;

[0097] The streaming speech recognition model FunASR is used to recognize user input audio in real time and output audio recognition text;

[0098] All the contents of the audio recognition text are input into the large model, and all the contents of the audio recognition text, namely the question text, are converted into text vectors, and the matching related texts are retrieved in the policy vector knowledge base, and the answer, namely the reply text, is output based on the streaming output method.

[0099] As a further implementation, according to the reply text, a semi-stream speech synthesis method based on dependency syntactic analysis is used to synthesize the output result audio, including:

[0100] The reply text output by the large model is divided into coarse-grained sentences using punctuation marks, including commas, periods, exclamation marks, question marks, and semicolons.

[0101] The dependency syntax analysis method is used to analyze the syntactic structure of the divided sentence structures to determine whether they meet the complete syntactic definition; the complete syntactic definition includes the subject-predicate structure or the subject-predicate-object structure;

[0102] If the complete syntactic definition is met, the sentence is used as the input of the speech synthesis model and stored in the data queue to be synthesized;

[0103] If not satisfied, continue to receive the reply text output by the large model until the next punctuation mark appears and repeat the above process.

[0104] As a further implementation, according to the reply text, a semi-stream speech synthesis method based on dependency syntactic analysis is used to synthesize the output result audio, which also includes:

[0105] The speech synthesis model obtains text from the data queue and synthesizes it to obtain the result audio; the result audio is output to the synthesis audio queue of the digital human rendering module of the front-end interaction layer in a semi-streaming manner.

[0106] In specific implementation, the present invention performs real-time interaction of 2D government digital human based on the digital human image constructed in step 1, as follows:

[0107] 1. Obtain user input audio in real time based on the web front end;

[0108] 2. Use the streaming speech recognition model FunASR to recognize user input audio in real time and output audio recognition text;

[0109] 3. After all the contents of the audio recognition text are converted into vectors, the Chinese vector embedding model bge-large-zh-v1.5 is first used to convert all the contents of the audio recognition text, that is, the question text, into vectors. Then, the cosine similarity matching algorithm is used to match and recall the relevant policy vectors in the constructed policy vector library to obtain the matching text fragments. The number of returned text fragments is set to 3, and the matching threshold is 0.9. Finally, the matched fragment content and the question content are input into the large model for the large model to answer. The output format is streaming output, that is, the reply text.

[0110] 4. Judge the reply text output by the large model. If the current text is a punctuation mark such as a comma, period, semicolon, exclamation mark or question mark, input the text content before the current punctuation mark into the dependency syntax model, that is, the zh_core_web_sm model in the spacy library of Python. If the output of the model includes the subject and predicate, it can be considered as a complete sentence. The subject is determined by the dependency analysis result of the word being a nominal subject nsubj or a nominal subject nsubjpass in the passive voice, and the predicate is determined by the word type being the predicate VERB or the dependency analysis result being the central word ROOT. If the definition of a complete sentence is not met, continue to obtain the output text of the large model until the next punctuation mark appears and repeat the above process judgment. The sentences identified by the complete syntactic structure are stored in the data queue to be synthesized.

[0111] 5. Call the speech synthesis model GPT-Sovits, set the reference audio of the required timbre, the reference audio text and the corresponding model (timbre) file, synthesize the text answer output in semi-streaming mode of the data queue and output the resulting audio to the web front end in wav format.

[0112] 6. The Web front end uses the pixi.js file that drives the live2D SDK to create a live2D model object, uses the speak method encapsulated by cubism4.js (this method encapsulates the digital human image project file produced in step 1), inputs the address of the synthesized audio file output by the speech synthesis model in a semi-streaming manner, and defines the function to be executed after each speech broadcast is completed, establishes a synthesized audio queue, obtains and sequentially stores the audio file stream output by the speech synthesis model in a semi-streaming manner; after each speech broadcast is completed, the subsequent audio in the synthesized audio queue is executed to achieve the real-time lip-sync driving and audio broadcasting of the digital human image after accessing the synthesized audio. Further, if the audio queue is empty at this time after the speech broadcast is completed, insert and play the pre-synthesized transition audio, such as "Let me think about it", "Let me think about it", etc., to ultimately achieve real-time interaction between the user and the digital human, and ensure the smoothness and no pauses during the digital human mouth type driving and audio output.

[0113] The innovation of the present invention is:

[0114] (1) Low-cost and rapid construction of 2D government digital humans based on AI painting technology, and the construction of digital humans based on the combination of AI painting and Live2D technology;

[0115] First, for the construction method of government digital human based on AI painting technology, one is the rapid generation method of general digital human image original painting for government needs, including the construction of key prompt word settings, such as the control of facial orientation, hair, eye details and mouth settings, and the relevant parameters when AI generates original paintings, such as acquisition methods, number of iterations, etc., and the generation steps of two times when generating the mouth and closing the mouth of the original painting with a fixed generation model random number seed. The second is the specific modeling method of digital human image based on live2D technology, including the key parts of the image before modeling, including the eyes (eye sockets, whites of the eyes, eyeballs), mouth (upper lip and lower lip and chin), face (remove the remaining parts of the eyes and mouth), hair, and neck. The steps of using glue to bond the eyes (eye sockets and surrounding facial skin) and mouth (upper lip and surrounding facial skin, lower lip and surrounding facial skin) to increase real details.

[0116] Second, real-time rendering of 2D government digital human based on live2D preset animation;

[0117] For the real-time rendering method of 2D government digital human based on live2D preset animation, first, the blinking animation of the character and the lip shape of the five key vowels are produced when building the live2D digital human model, including key frame animation production methods based on lip deformation, lip opening and closing, lip pouting and O-shaped mouth. Second, in the semi-streaming output of the speech synthesis model and the management of the synthesized audio queue by the front end, based on the function definition of the live2D front-end audio driver function after the playback event ends, the real-time call of the subsequent audio after the current audio playback is completed is realized, which further improves the fluency of the real-time rendering of the digital human and enhances the interactive experience with the user.

[0118] (2) A digital human real-time voice interaction solution based on dependency syntax analysis and insertion of transition audio;

[0119] The real-time voice interaction solution for digital humans based on dependency syntactic analysis and insertion of transitional audio, firstly, automatically performs voice segmentation on the sentences output by the large model through dependency syntactic analysis to obtain input text with a complete syntactic structure. Secondly, the pause in the interaction of digital humans caused by the possibility of an empty synthetic audio queue is further improved by inserting and playing appropriate transitional audio, which further improves the fluency and authenticity of the overall response of the digital humans.

[0120] Example 2

[0121] like Figure 5 As shown, the difference between this embodiment and embodiment 1 is that this embodiment provides a 2D government digital human real-time interaction system based on AI painting technology, which is based on the 2D government digital human real-time interaction method based on AI painting technology in embodiment 1; the system includes:

[0122] Digital human construction module, used to create digital human images based on a modeling method that combines AI painting and Live2D technology;

[0123] Voice access module, used to obtain user input audio in real time;

[0124] The speech recognition module is used to recognize the user input audio and obtain the audio recognition text and reply text;

[0125] A speech synthesis module is used to synthesize the output result audio according to the reply text using a semi-stream speech synthesis method based on dependency syntactic analysis;

[0126] The digital human rendering module is used to drive the lip sync of the digital human image in real time and perform audio broadcasting based on the result audio.

[0127] As a further implementation, the system is deployed on the implementation architecture of the front-end interaction layer, business processing layer and basic service layer from top to bottom;

[0128] The front-end interaction layer is deployed with voice access module, digital human construction module and digital human rendering module. The front-end interaction layer is responsible for rendering the digital human image to display the user's front-end interaction.

[0129] The business processing layer is deployed with speech recognition module, large model module and speech synthesis module. The business processing layer is used to process the backend user speech data, reply using the large model and synthesize audio;

[0130] The basic service layer is deployed with speech recognition models, large model modules and speech synthesis models. The basic service layer is responsible for providing basic services for various businesses in the business processing layer.

[0131] Among them, the execution process of each unit can be executed according to the process steps of the 2D government digital human real-time interaction method based on AI painting technology in Example 1, and will not be repeated one by one in this embodiment.

[0132] At the same time, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned 2D government digital human real-time interaction method based on AI painting technology is implemented.

[0133] At the same time, the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned 2D government digital human real-time interaction method based on AI painting technology.

[0134] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0135] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0136] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0138] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A 2D government digital human real-time interaction method based on AI painting technology, characterized in that: The method includes: Create digital human images based on a modeling method that combines AI painting with Live2D technology; Acquire user input audio in real time, and recognize the user input audio to obtain audio recognition text and reply text; According to the reply text, synthesize and output result audio using a semi-stream speech synthesis method based on dependency syntactic analysis; Based on the result audio, the produced digital human image is driven in real time by lip sync and audio broadcast is performed.

2. The 2D government digital human real-time interaction method based on AI painting technology according to claim 1 is characterized in that: Based on the modeling method combining AI painting and Live2D technology, digital human images are produced, including: Obtaining input prompt words to define the digital human image, wherein the input prompt words include face orientation, hair, eye details, and oral cavity; According to the input prompt words, an AI painting model is used to control the generation of a digital human original painting; Using a super-resolution model to further improve the quality of the original digital human painting to generate a high-resolution original painting image; Using Photoshop to split and layer the original picture to obtain multiple layers; Import multiple layers into live2D software, draw key points for key layers, and use the glue function to bond the layers together, realize blinking animation and five vowel lip shape animations, model the digital human image, and output the digital human image engineering file.

3. The 2D government digital human real-time interaction method based on AI painting technology according to claim 2 is characterized in that: The step of using the PhotoShop tool to split and layer the original picture includes: The entire face, eyes, mouth, neck and hair in the original painting are separated and layered using Photoshop tools, wherein the eyes include eye sockets, eyeballs and whites of the eyes, the mouth includes the upper lip, lower lip and chin, and the face is the remaining part after removing the eyes and mouth.

4. The 2D government digital human real-time interaction method based on AI painting technology according to claim 1 is characterized in that: Acquire user input audio in real time, and recognize the user input audio to obtain audio recognition text and reply text, including: Get user input audio in real time based on the web front end; Using a streaming speech recognition model FunASR, the user input audio is recognized in real time and an audio recognition text is output; All the contents of the audio recognition text are input into the big model, all the contents of the audio recognition text, i.e. the question text, are converted into text vectors, and the matching related texts are retrieved in the policy vector knowledge base, and the answer, i.e. the reply text, is output based on the streaming output mode.

5. The 2D government digital human real-time interaction method based on AI painting technology according to claim 4 is characterized in that: According to the reply text, a semi-stream speech synthesis method based on dependency syntactic analysis is used to synthesize and output result audio, including: The reply text output by the large model is coarsely divided using punctuation marks to obtain divided sentences; the punctuation marks include commas, periods, exclamation marks, question marks, and semicolons; Using dependency syntactic analysis to analyze the divided sentence structures, to determine whether they meet the complete syntactic definition; the complete syntactic definition includes a subject-predicate structure or a subject-predicate-object structure; If the complete syntactic definition is met, the sentence is used as the input of the speech synthesis model and stored in the data queue to be synthesized; If not satisfied, continue to receive the reply text output by the large model until the next punctuation mark appears and repeat the above process.

6. The 2D government digital human real-time interaction method based on AI painting technology according to claim 5 is characterized in that: According to the reply text, a semi-stream speech synthesis method based on dependency syntactic analysis is used to synthesize and output result audio, which also includes: The speech synthesis model obtains text from the data queue and synthesizes it to obtain result audio; outputs the result audio to the synthesis audio queue in a semi-streaming manner, and inserts transition audio into the audio queue.

7. The 2D government digital human real-time interactive system based on AI painting technology is characterized by: The system is based on the 2D government digital human real-time interaction method based on AI painting technology as described in any one of claims 1 to 6; the system includes: Digital human construction module, used to create digital human images based on a modeling method that combines AI painting and Live2D technology; Voice access module, used to obtain user input audio in real time; A speech recognition module, used to recognize the user input audio and obtain audio recognition text and reply text; A speech synthesis module, used for synthesizing and outputting result audio according to the reply text using a semi-stream speech synthesis method based on dependency syntactic analysis; The digital human rendering module is used to drive the lip sync of the digital human image in real time and perform audio broadcasting based on the result audio.

8. The 2D government digital human real-time interactive system based on AI painting technology according to claim 7 is characterized in that: The system is deployed on the implementation architecture of the front-end interaction layer, business processing layer and basic service layer from top to bottom; The front-end interaction layer is deployed with a voice access module, a digital human construction module and a digital human rendering module. The front-end interaction layer is responsible for rendering the digital human image to display the user's front-end interaction; The business processing layer is deployed with a speech recognition module, a large model module and a speech synthesis module. The business processing layer is used to process the background user voice data, use the large model to reply and synthesize audio; The basic service layer is deployed with a speech recognition model, a large model module and a speech synthesis model. The basic service layer is responsible for providing basic services for various businesses of the business processing layer.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the real-time interaction method of 2D government digital human based on AI painting technology is implemented as described in any one of claims 1 to 6.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, the real-time interaction method of 2D government digital human based on AI painting technology as described in any one of claims 1 to 6 is implemented.

Images