Digital human application acceleration method and system based on flash memory controller

Abstract

The invention discloses a digital human application acceleration method and system based on a flash memory controller, and the method comprises the steps: connecting the flash memory controller outside a host side, and presetting each model of a digital human system to a model storage unit of the flash memory controller; presetting the digital human image video to a storage module, and performing human face detection through an NPU module to obtain a human face frame sequence; aSR voice recognition reasoning is carried out on the real-time voice data, and answer text information is obtained through GPU reasoning; tTS reasoning and voice-to-lip reasoning are carried out in sequence through an NPU module, and picture fusion is carried out through an on-chip CPU to obtain a face picture sequence; face repair reasoning is carried out through an NPU module, audio and video fusion coding is carried out through an on-chip CPU, and a digital human image video is obtained. Through the technical scheme of the invention, the load of the GPU and the CPU is greatly reduced, the hardware cost is reduced, and the cost performance of the digital human system is improved.

Description

Technical Field

[0001] The present invention relates to the field of digital human technology, and in particular to a digital human application acceleration method based on a flash memory controller and a digital human application acceleration system based on a flash memory controller. Background Art

[0002] At present, in the existing digital human systems, Figure 1 As shown in the figure, the entire process requires six model inferences, including ASR speech recognition, large model question and answer, TTS text-to-speech, speech-to-lip shape, face detection, and face restoration. Currently, all of these need to be completed based on GPU, which has a high computational cost and cannot meet the needs of real-time conversational digital humans. Summary of the invention

[0003] In view of the above problems, the present invention provides a digital human application acceleration method and system based on a flash memory controller. By implementing the ASR speech recognition, TTS text-to-speech, speech-to-lip shape, face detection and face repair functions that originally need to be implemented on the GPU on the NPU module of the flash memory controller, the GPU load is greatly reduced. The lip shape face fusion and audio and video fusion encoding functions that originally need to be implemented on the CPU are implemented on the on-chip CPU of the flash memory controller, which greatly reduces the CPU load. Compared with the digital human system that originally needed to rely entirely on the GPU to implement the overall process, the hardware cost is greatly reduced, and the cost performance of the digital human system is improved.

[0004] To achieve the above object, the present invention provides a digital human application acceleration method based on a flash memory controller, comprising:

[0005] A flash memory controller is externally connected to the host side, and a deep model inference module NPU module, an on-chip CPU and a storage module are built in the flash memory controller;

[0006] Presetting an ASR speech recognition model, a text-to-speech TTS model, a speech-to-lip shape model, a face detection model, and a face restoration model into a model storage unit of the flash memory controller;

[0007] Presetting the digital human image video into the storage module, performing face detection on the digital human image video through the NPU module, obtaining a face frame sequence and storing it in the storage module;

[0008] Inputting real-time voice data into the flash memory controller, performing ASR voice recognition reasoning according to the ASR voice recognition model through the NPU module, and obtaining reasoning result text information;

[0009] The inference result text information is transmitted to the GPU for large language model inference to obtain answer text information;

[0010] The answer text information is subjected to TTS inference and speech-to-lip shape inference in accordance with the TTS model and the speech-to-lip shape model in sequence by the NPU module to obtain a lip picture sequence;

[0011] The face frame sequence and the lip picture sequence are fused by the on-chip CPU to obtain a face picture sequence with a speech effect;

[0012] Performing face restoration inference on the face picture sequence through the NPU module according to the face restoration model to obtain an optimized face picture sequence;

[0013] The voice information and the optimized face picture sequence are subjected to audio and video fusion encoding by the on-chip CPU to obtain a digital human image video that speaks out answers to the real-time voice data;

[0014] The digital human image video is transmitted back to the host side for playback, completing the digital human question and answer process.

[0015] In the above technical solution, preferably, the flash memory controller adopts an EB1229 flash memory controller.

[0016] In the above technical solution, preferably, the inference result text information is transmitted to the GPU for large language model inference to obtain the answer text information, and the specific process includes:

[0017] The inference result text information is transmitted to the GPU through the data direct link of the flash memory controller, and the GPU infers the inference result text information through its own large language model to obtain corresponding answer text information.

[0018] In the above technical solution, preferably, the answer text information is subjected to TTS inference and speech-to-lip shape inference in accordance with the TTS model and the speech-to-lip shape model in sequence through the NPU module to obtain a lip picture sequence, and the specific process includes:

[0019] The answer text information is subjected to TTS inference by the NPU module according to the TTS model to obtain voice information;

[0020] The speech information is passed through the NPU module to perform speech-to-lip shape inference according to the speech-to-lip shape model to obtain a lip picture sequence.

[0021] The present invention further proposes a digital human application acceleration system based on a flash memory controller, which uses the digital human application acceleration method based on a flash memory controller disclosed in any one of the above technical solutions, including:

[0022] A flash memory connection module, used to connect the flash memory controller externally on the host side, based on the built-in deep model inference module NPU module, on-chip CPU and storage module of the flash memory controller;

[0023] A model built-in module, used for presetting an ASR speech recognition model, a text-to-speech TTS model, a speech-to-lip shape model, a face detection model, and a face restoration model into a model storage unit of the flash memory controller;

[0024] A face detection module, used to pre-set the digital human image video to the storage module, perform face detection on the digital human image video through the NPU module, obtain a face frame sequence and store it in the storage module;

[0025] A speech recognition module, used for inputting real-time speech data into the flash memory controller, performing ASR speech recognition reasoning according to the ASR speech recognition model through the NPU module, and obtaining text information of the reasoning result;

[0026] An inference answer module is used to transmit the inference result text information to the GPU for large language model inference to obtain answer text information;

[0027] A lip image inference module, used to perform TTS inference and speech-to-lip shape inference on the answer text information through the NPU module in accordance with the TTS model and the speech-to-lip shape model in sequence, to obtain a lip image sequence;

[0028] A face fusion module, used for fusing the face frame sequence and the lip picture sequence through the on-chip CPU to obtain a face picture sequence with a speech effect;

[0029] A face restoration module, used for performing face restoration inference on the face image sequence through the NPU module according to the face restoration model to obtain an optimized face image sequence;

[0030] An image fusion module, used to perform audio and video fusion encoding on the voice information and the optimized face picture sequence through the on-chip CPU to obtain a digital human image video that speaks out answers to the real-time voice data;

[0031] The video playback module is used to transmit the digital human image video back to the host side for playback to complete the digital human question and answer process.

[0032] In the above technical solution, preferably, the flash memory controller adopts an EB1229 flash memory controller.

[0033] In the above technical solution, preferably, the reasoning answer module is specifically used for:

[0034] The inference result text information is transmitted to the GPU through the data direct link of the flash memory controller, and the GPU infers the inference result text information through its own large language model to obtain corresponding answer text information.

[0035] In the above technical solution, preferably, the lip image reasoning module is specifically used for:

[0036] The answer text information is subjected to TTS inference by the NPU module according to the TTS model to obtain voice information;

[0037] The speech information is passed through the NPU module to perform speech-to-lip shape inference according to the speech-to-lip shape model to obtain a lip picture sequence.

[0038] Compared with the prior art, the beneficial effects of the present invention are as follows: by implementing the ASR speech recognition, TTS text-to-speech, speech-to-lip shape, face detection and face restoration functions that originally needed to be implemented on the GPU on the NPU module of the flash memory controller, the GPU load is greatly reduced; the lip shape face fusion and audio and video fusion encoding functions that originally needed to be implemented on the CPU are implemented on the on-chip CPU of the flash memory controller, which greatly reduces the CPU load. Compared with the digital human system that originally needed to rely entirely on the GPU to implement the overall process, the hardware cost is greatly reduced, and the cost-effectiveness of the digital human system is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of the overall process of a digital human disclosed in an embodiment of the prior art;

[0040] Figure 2 The present invention is a flowchart of a method for accelerating a digital human application based on a flash memory controller disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0043] like Figure 2 As shown, a digital human application acceleration method based on a flash memory controller provided by the present invention includes:

[0044] The host side is externally connected to a flash memory controller, based on which the flash memory controller has a built-in deep model inference module NPU module, on-chip CPU and storage module;

[0045] Preset the ASR speech recognition model, the text-to-speech TTS model, the speech-to-lip shape model, the face detection model, and the face restoration model to the model storage unit of the flash memory controller;

[0046] Preset the digital human image video to the storage module, perform face detection on the digital human image video through the NPU module, obtain the face frame sequence and store it in the storage module;

[0047] The real-time voice data is input into the flash memory controller, and the ASR voice recognition inference is performed according to the ASR voice recognition model through the NPU module to obtain the text information of the inference result;

[0048] The inference result text information is transmitted to the GPU for large language model inference to obtain the answer text information;

[0049] The answer text information is passed through the NPU module to perform TTS inference and speech-to-lip shape inference according to the TTS model and speech-to-lip shape model in turn to obtain a lip picture sequence;

[0050] The face frame sequence and the lip image sequence are fused through the on-chip CPU to obtain a face image sequence with a speech effect;

[0051] The face image sequence is passed through the NPU module to perform face restoration inference according to the face restoration model to obtain an optimized face image sequence;

[0052] The voice information and optimized face image sequence are encoded by the on-chip CPU for audio and video fusion, and a digital human image video is obtained that speaks out answers to real-time voice data.

[0053] The digital human image video is transmitted back to the host side for playback to complete the digital human question and answer process.

[0054] In this implementation, the ASR speech recognition, TTS text-to-speech, speech-to-lip shape, face detection and face restoration functions that originally needed to be implemented on the GPU are implemented on the NPU module of the flash memory controller, which greatly reduces the GPU load. The lip shape face fusion and audio and video fusion encoding functions that originally needed to be implemented on the CPU are implemented on the on-chip CPU of the flash memory controller, which greatly reduces the CPU load. Compared with the digital human system that originally needed to rely entirely on the GPU to implement the overall process, the hardware cost is greatly reduced and the cost-effectiveness of the digital human system is improved.

[0055] Specifically, based on the flash memory controller, some processes of the digital human system can be implemented on the flash memory controller, which can save GPU resources and improve the performance of some processes at the same time; in addition, combined with the storage function of the flash memory controller, the digital human image can be preset in the flash memory controller to avoid repeated processing of the digital human image and save analysis resources.

[0056] In the above implementation, preferably, the flash memory controller adopts an EB1229 flash memory controller.

[0057] In the above implementation, preferably, the inference result text information is transmitted to the GPU for large language model inference to obtain the answer text information. The specific process includes:

[0058] The inference result text information is transmitted to the GPU through the data direct link of the flash memory controller. The GPU infers the inference result text information through its own large language model to obtain the corresponding answer text information.

[0059] In the above implementation, preferably, the answer text information is subjected to TTS inference and speech-to-lip shape inference in accordance with the TTS model and the speech-to-lip shape model in sequence through the NPU module to obtain a lip picture sequence. The specific process includes:

[0060] The answer text information is passed through the NPU module to perform TTS inference according to the TTS model to obtain voice information;

[0061] The speech information is passed through the NPU module to perform speech-to-lip shape inference according to the speech-to-lip shape model to obtain a lip picture sequence.

[0062] According to the digital human application acceleration method based on flash memory controller disclosed in the above embodiment, during the implementation process, operations are performed according to the following steps.

[0063] Step 1: Preset the ASR speech recognition model, text-to-speech TTS model, speech-to-lip shape model, face detection model, and face restoration model to the model storage unit of the EB1229 flash memory controller;

[0064] Step 2: pre-set the digital human image video to the storage module of the EB1229 flash memory controller, perform face detection through the NPU module on the EB1229 flash memory controller, obtain the face frame sequence corresponding to the digital human image video, and then store it in the storage module of the EB1229 flash memory controller;

[0065] Step 3: The real-time voice data is transferred to the EB1229 flash memory controller, and the ASR speech recognition model is inferred through the NPU module on the EB1229 flash memory controller to obtain the text information of the inference result;

[0066] Step 4: The inference result text information obtained in step 3 is directly transmitted to the GPU through the data direct link of the EB1229 flash memory controller to perform large language model inference to obtain the answer text information;

[0067] Step 5: The answer text information obtained in step 4 is passed through the NPU module on the EB1229 flash memory controller to perform TTS text-to-speech model inference to obtain voice information;

[0068] Step 6: The voice information obtained in step 5 is passed through the NPU module of the EB1229 flash memory controller to perform speech-to-lip shape model inference to obtain a lip picture sequence;

[0069] Step 7: The face frame sequence obtained in step 2 and the lip image sequence obtained in step 6 are fused through the on-chip CPU of the EB1229 flash memory controller to obtain a face image sequence with a speech effect;

[0070] Step 8: The face image sequence obtained in step 7 is passed through the NPU module of the EB1229 flash memory controller to perform face restoration model inference to obtain an optimized face image sequence;

[0071] Step nine: The voice information obtained in step five and the facial image sequence obtained in step eight are encoded by the on-chip CPU of the EB1229 flash memory controller to obtain a digital human image video that can answer the input voice orally;

[0072] Step 10: The digital human image video obtained in step 9 is transmitted back to the host side to complete the digital human question and answer process.

[0073] The present invention further proposes a digital human application acceleration system based on a flash memory controller, which applies the digital human application acceleration method based on a flash memory controller disclosed in any one of the above embodiments, including:

[0074] A flash connection module is used to connect the flash controller externally on the host side, based on the built-in deep model inference module NPU module, on-chip CPU and storage module of the flash controller;

[0075] A model built-in module is used to preset the ASR speech recognition model, the text-to-speech TTS model, the speech-to-lip shape model, the face detection model, and the face restoration model into the model storage unit of the flash memory controller;

[0076] A face detection module is used to pre-set the digital human image video to the storage module, perform face detection on the digital human image video through the NPU module, obtain a face frame sequence and store it in the storage module;

[0077] The speech recognition module is used to input the real-time speech data into the flash memory controller, perform ASR speech recognition inference according to the ASR speech recognition model through the NPU module, and obtain the text information of the inference result;

[0078] The reasoning answer module is used to transmit the text information of the reasoning result to the GPU for large language model reasoning to obtain the answer text information;

[0079] The lip image inference module is used to perform TTS inference and speech-to-lip shape inference on the answer text information through the NPU module according to the TTS model and speech-to-lip shape model in turn to obtain a lip image sequence;

[0080] The face fusion module is used to fuse the face frame sequence and the lip picture sequence through the on-chip CPU to obtain a face picture sequence with a speech effect;

[0081] The face restoration module is used to perform face restoration inference on the face image sequence through the NPU module according to the face restoration model to obtain an optimized face image sequence;

[0082] The image fusion module is used to perform audio and video fusion encoding of voice information and optimized face image sequences through the on-chip CPU to obtain a digital human image video that can answer the real-time voice data orally;

[0083] The video playback module is used to transmit the digital human image video back to the host side for playback to complete the digital human question and answer process.

[0084] In the above implementation, preferably, the flash memory controller adopts an EB1229 flash memory controller.

[0085] In the above implementation, preferably, the reasoning answer module is specifically used for:

[0086] The inference result text information is transmitted to the GPU through the data direct link of the flash memory controller. The GPU infers the inference result text information through its own large language model to obtain the corresponding answer text information.

[0087] In the above implementation, preferably, the lip image reasoning module is specifically used for:

[0088] The answer text information is passed through the NPU module to perform TTS inference according to the TTS model to obtain voice information;

[0089] The speech information is passed through the NPU module to perform speech-to-lip shape inference according to the speech-to-lip shape model to obtain a lip picture sequence.

[0090] According to the digital human application acceleration system based on flash memory controller disclosed in the above-mentioned implementation, the functions to be implemented by each module thereof correspond to the steps of the digital human application acceleration method based on flash memory controller disclosed in the above-mentioned implementation. During the implementation process, the operations are performed with reference to the above-mentioned implementation, which will not be repeated here.

[0091] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A digital human application acceleration method based on a flash memory controller, characterized in that: include: A flash memory controller is externally connected to the host side, and a deep model inference module NPU module, an on-chip CPU and a storage module are built in the flash memory controller; Presetting an ASR speech recognition model, a text-to-speech TTS model, a speech-to-lip shape model, a face detection model, and a face restoration model into a model storage unit of the flash memory controller; Presetting the digital human image video into the storage module, performing face detection on the digital human image video through the NPU module, obtaining a face frame sequence and storing it in the storage module; Inputting real-time voice data into the flash memory controller, performing ASR voice recognition reasoning according to the ASR voice recognition model through the NPU module, and obtaining reasoning result text information; Transmitting the inference result text information to the GPU for large language model inference to obtain answer text information; The answer text information is subjected to TTS inference and speech-to-lip shape inference in accordance with the TTS model and the speech-to-lip shape model in sequence by the NPU module to obtain a lip picture sequence; The face frame sequence and the lip picture sequence are fused by the on-chip CPU to obtain a face picture sequence with a speech effect; Performing face restoration inference on the face picture sequence through the NPU module according to the face restoration model to obtain an optimized face picture sequence; The voice information and the optimized face picture sequence are subjected to audio and video fusion encoding by the on-chip CPU to obtain a digital human image video that speaks out answers to the real-time voice data; The digital human image video is transmitted back to the host side for playback, completing the digital human question and answer process.

2. The method for accelerating digital human applications based on a flash memory controller according to claim 1, characterized in that: The flash memory controller adopts EB1229 flash memory controller.

3. The digital human application acceleration method based on flash memory controller according to claim 1 is characterized in that: The specific process of transmitting the inference result text information to the GPU for large language model inference to obtain answer text information includes: The inference result text information is transmitted to the GPU through the data direct link of the flash memory controller, and the GPU infers the inference result text information through its own large language model to obtain corresponding answer text information.

4. The digital human application acceleration method based on flash memory controller according to claim 1 is characterized in that: The answer text information is subjected to TTS inference and speech-to-lip shape inference in accordance with the TTS model and the speech-to-lip shape model in sequence by the NPU module to obtain a lip picture sequence. The specific process includes: The answer text information is subjected to TTS inference by the NPU module according to the TTS model to obtain voice information; The speech information is passed through the NPU module to perform speech-to-lip shape inference according to the speech-to-lip shape model to obtain a lip picture sequence.

5. A digital human application acceleration system based on a flash memory controller, characterized in that: The digital human application acceleration method based on a flash memory controller as claimed in any one of claims 1 to 4 comprises: A flash memory connection module, used to connect the flash memory controller externally on the host side, based on the built-in deep model inference module NPU module, on-chip CPU and storage module of the flash memory controller; A model built-in module, used for presetting an ASR speech recognition model, a text-to-speech TTS model, a speech-to-lip shape model, a face detection model, and a face restoration model into a model storage unit of the flash memory controller; A face detection module, used to pre-set the digital human image video to the storage module, perform face detection on the digital human image video through the NPU module, obtain a face frame sequence and store it in the storage module; A speech recognition module, used for inputting real-time speech data into the flash memory controller, performing ASR speech recognition reasoning according to the ASR speech recognition model through the NPU module, and obtaining text information of the reasoning result; An inference answer module is used to transmit the inference result text information to the GPU for large language model inference to obtain answer text information; A lip image inference module, used to perform TTS inference and speech-to-lip shape inference on the answer text information through the NPU module in accordance with the TTS model and the speech-to-lip shape model in sequence, to obtain a lip image sequence; A face fusion module, used for fusing the face frame sequence and the lip picture sequence through the on-chip CPU to obtain a face picture sequence with a speech effect; A face restoration module, used for performing face restoration inference on the face image sequence through the NPU module according to the face restoration model to obtain an optimized face image sequence; An image fusion module, used to perform audio and video fusion encoding on the voice information and the optimized face picture sequence through the on-chip CPU to obtain a digital human image video that speaks out answers to the real-time voice data; The video playback module is used to transmit the digital human image video back to the host side for playback to complete the digital human question and answer process.

6. The digital human application acceleration system based on flash memory controller according to claim 5 is characterized in that: The flash memory controller adopts EB1229 flash memory controller.

7. The digital human application acceleration system based on flash memory controller according to claim 5 is characterized in that: The reasoning answer module is specifically used for: The inference result text information is transmitted to the GPU through the data direct link of the flash memory controller, and the GPU infers the inference result text information through its own large language model to obtain corresponding answer text information.

8. The digital human application acceleration system based on flash memory controller according to claim 5, characterized in that: The lip image reasoning module is specifically used for: The answer text information is subjected to TTS inference by the NPU module according to the TTS model to obtain voice information; The speech information is passed through the NPU module to perform speech-to-lip shape inference according to the speech-to-lip shape model to obtain a lip picture sequence.

Images