3D digital human system integration method and system based on credential environment

Abstract

The invention relates to the technical field of software development, and particularly discloses a 3D digital human system integration method and system based on a credential environment, and the method comprises the steps: copying an integration package of a 3D digital human system to a designated server in an offline mode, and carrying out the interactive operation through a terminal; selecting a 3D digital human model through an editing and creating interface of the 3D digital human system, configuring interaction parameters based on the digital human resource management library, and performing a preview test; integrating a multi-channel voice intention analysis model based on the 3D digital human model, performing noise recognition and dialect matching, analyzing user intention, determining answer content from an answer knowledge base, outputting answer voice according to the answer content, rendering the 3D digital human model according to the answer content and the user intention, and driving multi-modal interaction service feedback. The operation state data is monitored in real time to give an abnormal alarm; according to the invention, high performance, high stability and easy landing application of the 3D digital human system in the credential environment can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of software development, more particularly, the present application relates to a 3D digital human system integration method and system based on a signal creation environment. BACKGROUND

[0002] With the development of artificial intelligence and computer graphics, 3D digital humans are gradually applied to government services, online education, financial consulting, virtual customer service and other scenarios. Existing 3D digital human systems usually rely on commercial rendering engines and voice interaction large models optimized based on CUDA architecture, and can achieve relatively natural real-time interaction effects in foreign GPU and general operating system environments. However, in the context of domestic substitution, the 3D digital human system exposes a series of core problems when running in the signal creation environment.

[0003] In terms of GPU rendering, the existing rendering engine is different from the GPU in the rendering pipeline, the driving interface and the instruction set, which leads to a significant decline in rendering performance, frequent rendering errors such as texture tearing and shadow discontinuity, and extremely high compatibility adaptation difficulty. Secondly, in terms of CPU architecture and system integration, the 3D digital human system needs to be deeply coupled with the CPU, the operating system, the database and the middleware. However, the current CPU architecture is diversified, the existing operating system interface is not unified, and the domestic database and message queue have deficiencies in interface compatibility and transaction consistency, which can easily cause problems such as loss of interaction data, delay in multi-modal data synchronization and performance decline, and the overall integration complexity increases significantly. In addition, in terms of deployment and maintenance, different operating systems, hardware devices and network environments can affect the stability and compatibility of the digital human system. Especially in the customer intranet environment, problems such as missing dependent libraries and driver mismatch frequently occur, which leads to a long deployment period, high maintenance cost and the need for manual debugging by professional personnel, which brings great obstacles to actual landing. More prominently, in the voice interaction link, the existing speech synthesis (TTS) and speech recognition (ASR) large models are developed and optimized based on CUDA architecture, and when directly migrated to the GPU, problems such as low efficiency of computing power scheduling, decline in model accuracy, unnatural speech synthesis, low dialect recognition rate and recognition failure in noisy environments often occur, which seriously affects the interaction experience.

[0004] Therefore, it is necessary to provide a 3D digital human system integration method and system based on a signal creation environment to solve the above technical problems. In order to solve the above problems, a technical solution is provided. SUMMARY

[0005] In order to overcome the above-mentioned defects of the prior art, the present application provides a 3D digital human system integration method and system based on a signal creation environment, which is used to solve the problems of poor rendering adaptation, difficult system integration, complex deployment and maintenance, and difficult migration of voice large models faced by the existing 3D digital human system in the application process.

[0006] To achieve the above object, the present application provides the following technical solutions: The 3D digital human system integration method based on the Xinhua environment comprises the following steps: Copy the integration package of the 3D digital human system to the designated server through an offline mode, and run interactively through a terminal; Select a 3D digital human model through the creation interface of the 3D digital human system, configure interaction parameters based on the digital human resource management library, and perform preview testing; Integrate a multi-channel speech intent analysis model based on the 3D digital human model, perform noise recognition and dialect matching, analyze user intent, determine the answer content from the answer knowledge base, and output the answer speech according to the answer content; Render the 3D digital human model according to the answer content and user intent, drive the multi-modal interaction service feedback, and monitor the running state data in real time to perform abnormal alarm.

[0007] As a further scheme of the present application, selecting a 3D digital human model through the creation interface of the 3D digital human system specifically comprises: supporting drag-and-drop configuration of the 3D digital human model image, providing an expression binding tool and a motion editing panel, being able to associate speech rhythm to configure lip motion synchronization parameters, and supporting custom motion duration through a preset daily motion library.

[0008] As a further scheme of the present application, integrating a multi-channel speech intent analysis model based on the 3D digital human model, performing noise recognition and dialect matching, analyzing user intent, determining the answer content from the answer knowledge base, and outputting the answer speech according to the answer content specifically comprises the following steps: Obtain a user speech signal through a speech collection terminal, pre-process the user speech signal to output a first speech signal; Improve the recognition accuracy of the first speech signal in a noisy environment according to the multi-channel speech intent analysis model, and output a second speech signal; Query a matching dialect type in a dialect speech library by combining the second speech signal through a dialect recognition pre-classifier, determine a dedicated dialect recognition sub-model according to the dialect type, and output a third speech signal; Analyze user intent according to the third speech signal, determine the answer content from the answer knowledge base, and output the answer speech according to the answer content through TTS.

[0009] As a further scheme of the present application, improving the recognition accuracy of the first speech signal in a noisy environment according to the multi-channel speech intent analysis model, and outputting a second speech signal specifically comprises the following steps: Collect multi-channel speech audio through a microphone array, segment the multi-channel speech audio based on a fixed time period to obtain a to-be-detected audio segment; Extract the spectral features of the audio segment to be detected, and cut the non-speech audio segment of the audio segment to be detected according to the spectral features to obtain the remaining speech audio segment; wherein the non-speech audio segment includes a mute spectrum segment and a noise spectrum segment; According to the noise spectrum segment, the noise feature is captured, and whether the noise in the remaining speech audio segment affects the speech recognition is detected according to the noise feature; If the speech recognition is not affected, the remaining speech audio segment is output as a second speech signal; if the speech recognition is affected, the remaining speech audio segment is marked as a difficult-to-recognize audio segment; Based on the speech audio segment of the previous period and the speech audio segment of the next period of the difficult-to-recognize audio segment, the difficult-to-recognize audio segment is detected by secondary speech recognition.

[0010] As a further scheme of the present application, according to the noise spectrum segment, the noise feature is captured, and whether the noise in the remaining speech audio segment affects the speech recognition is detected according to the noise feature, and the specific steps are as follows: According to the noise spectrum segment, the noise feature is captured, and the noise feature includes a first noise feature and a second noise feature, and the time sequence transformation value of the noise spectrum is calculated As the first noise feature, wherein, is the power spectrum value corresponding to the n+1 frame of the xth noise spectrum segment, is the power spectrum value corresponding to the n frame of the xth noise spectrum segment, is the power spectrum value corresponding to the n+1 frame of the xth noise spectrum segment, is the power spectrum value corresponding to the n frame of the xth noise spectrum segment, and the mean and variance of the noise spectrum segment are calculated as the second noise feature, respectively; According to the noise feature, a primary speech recognition detection model is constructed to detect whether the noise in the remaining speech audio segment affects the speech recognition.

[0011] As a further scheme of the present application, based on the speech audio segment of the previous period and the speech audio segment of the next period of the difficult-to-recognize audio segment, the difficult-to-recognize audio segment is detected by secondary speech recognition, and the specific steps include: The difficult-to-recognize audio segment is processed by ASR to obtain the confidence of each frame; The speech audio segment of the previous period of the difficult-to-recognize audio segment , the speech audio segment of the next period, is marked as a fuzzy segment; By inputting and as conditions into the Transformer language model, the candidate filling content of the fuzzy segment is predicted, and the candidate filling content is calculated by a text embedding model and The consistency score of the overall semantics is selected, and the candidate filling content corresponding to the highest consistency score is selected as the filling content; The filling content is output again as a second speech signal.

[0012] As a further scheme of the present application, the specific steps of analyzing the user's intention according to the third speech signal are as follows: The first-level speech text is generated by ASR recognition according to the third speech signal, and the first intention feature is obtained by natural language processing on the first-level speech text; The first intention feature is preliminarily classified by the intention classification model, and the first intention category of the user is output, the second user feature is determined in combination with the context of the first-level speech text and the first intention category, and the user's intention is determined according to the second user feature.

[0013] As a further scheme of the present application, the 3D digital human model is rendered according to the answer content and the user's intention, and the multi-modal interactive service feedback is driven, wherein the multi-modal interactive service feedback includes generating corresponding expressions and actions of the 3D digital human model according to the answer content and the user's intention, and synthesizing speech and realizing lip movement synchronization through TTS.

[0014] As a further scheme of the present application, the running state data is monitored in real time to generate an abnormal alarm, specifically: the running state data includes CPU usage, GPU usage, and rendering frame rate, the running state data is monitored in real time, the running state data is compared with a preset threshold, and if the running state data is greater than or equal to the preset threshold, an abnormal alarm is triggered.

[0015] The 3D digital human system integration system based on the signal creation environment, the system includes a 3D digital human system deployment interaction module, a 3D digital human creation preview module, a speech intention analysis and answer generation module, and a 3D digital human rendering and multi-modal interaction module; The 3D digital human system deployment interaction module is used to copy the integration package of the 3D digital human system to the designated server through the offline mode, and the terminal is used for interactive operation; The 3D digital human creation preview module is used to select the 3D digital human model through the creation interface of the 3D digital human system, configure the interaction parameters based on the digital human resource management library, and perform preview testing; The speech intention analysis and answer generation module is used to integrate a multi-channel speech intention analysis model based on the 3D digital human model, perform noise recognition and dialect matching, analyze the user's intention, determine the answer content from the answer knowledge base, and output the answer speech according to the answer content; The 3D digital human rendering and multi-modal interaction module is used for rendering the 3D digital human model according to the answer content and the user intention, driving the multi-modal interaction service feedback, and monitoring the running state data in real time to perform abnormal alarm.

[0016] The technical effects and advantages of the 3D digital human system integration method and system based on the Xinchuang environment of the present application are as follows: the integration package of the 3D digital human system is copied to a designated server in an offline manner, interactive operation is performed through a terminal, a 3D digital human model is selected through the creation interface of the 3D digital human system, interactive parameters are configured based on the digital human resource management library, and preview testing is performed; a multi-channel speech intention analysis model is integrated based on the 3D digital human model, noise recognition and dialect matching are performed, the user intention is analyzed, the answer content is determined from the answer knowledge base, and the answer speech is output according to the answer content; the 3D digital human model is rendered according to the answer content and the user intention, the multi-modal interaction service feedback is driven, and the running state data is monitored in real time to perform abnormal alarm.

[0017] The present application realizes the goal of landing the Xinchuang digital human with simple creation, convenient deployment and stable running through the three-layer architecture design of "creation-operation-adaptation", reduces the threshold of digital human creation through the web creation terminal, solves the compatibility problem of CPU and GPU through the underlying Xinchuang adaptation layer, and finally realizes the goal of landing the Xinchuang digital human with simple creation, convenient deployment and stable running, which is different from the traditional digital human system relying on engine and unable to adapt to Xinchuang environment; the present application integrates a multi-channel speech intention analysis model, noise recognition and dialect matching, improves the robustness and accuracy of speech recognition, enables the 3D digital human to understand the user intention in a complex environment, and enhances the naturalness and intelligence of the interactive experience; the present application outputs speech in combination with the answer knowledge base and drives the 3D digital human model to render multi-modal feedback, realizes the synchronization of speech, expression and action, provides immersive and personalized interactive experience, and improves user satisfaction and service efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 The flowchart of the 3D digital human system integration method based on the Xinchuang environment provided for the embodiments of the present application; Figure 2 The system block diagram of the 3D digital human system integration system based on the Xinchuang environment provided for the embodiments of the present application; Figure 3 The editing interface of the 3D digital human model provided for the embodiments of the present application; Figure 4 The 3D digital human model timbre selection interface provided for the embodiments of the present application. DETAILED DESCRIPTION

[0019] The technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described technical solutions are only a part of the present application, not all. Based on the technical solutions in the present application, all other technical solutions obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0020] Embodiment 1 As Figure 1 shown in the flowchart of the 3D digital human system integration method based on the Xinchuang environment provided by the embodiment of the present application, Figure 1 The execution subject of the method shown in the figure can be a software and / or a hardware device. The execution subject of the present application can include but is not limited to at least one of the following: user equipment, network equipment, etc. Among them, the user equipment can include but is not limited to computers, smart phones, personal digital assistants (Personal Digital Assistant, PDA) and the above-mentioned electronic devices, etc. The network equipment can include but is not limited to a single network server, a server group composed of multiple network servers or a cloud composed of a large number of computers or network servers based on cloud computing, wherein cloud computing is a kind of distributed computing, which is a super virtual computer composed of a group of loosely coupled computers. This embodiment does not make any limitation. Including steps 1 to 4, as follows: Step 1, copy the integration package of the 3D digital human system to the designated server through offline mode, and run through the terminal; The embodiment of the present application effectively solves the distribution and installation problem of the running package in the Internet isolation environment through the offline copying mode, and ensures the controllability and security of the system on the server. At the same time, by directly loading the integration package of the 3D digital human system through the terminal, the traditional complex dependence installation and network adaptation problem is avoided, and the quick delivery and plug-and-play deployment of digital human in the government office are realized.

[0021] The 3D digital human system integration package is a running unit generated in the creation stage and preview test stage, which has the ability to be directly deployed and run in the domestic software and hardware environment, not only contains the rendering resources of the 3D digital human model, including model files, action library, expression library, Shader translation files, but also contains multi-channel voice analysis modules related to voice interaction, dialogue logic configuration files and running environment configuration files, including driver interface parameters adapted to CPU / GPU, operating system compatibility parameters; The "plug and play" deployment is realized, which not only ensures the safety and controllability in the Xinchuang environment, but also significantly reduces the complexity of deployment and maintenance, so that the digital human can be quickly applied to the government office, education classroom, medical diagnosis and other scenes.

[0022] Step 2, select a 3D digital human model through the creation interface of the 3D digital human system, configure interaction parameters based on the digital human resource management library, and perform preview testing; In the construction process of a smart education platform, the developer selects and customizes a 3D digital human model of a classroom assistant type using the visual creation interface provided by the 3D digital human system. The specific operation is as follows: through the drag-and-drop configuration function, the developer selects a young teacher image with affinity in the interface, then sets up basic expressions such as "nodding", "smiling", and "concentration" for the model using the expression binding tool, and sets up "hand gestures during explanation" and "board writing actions" through the action editing panel, while automatically associating the lip movement synchronization parameters with the speech rhythm to ensure the consistency of the mouth shape and the explanation content.

[0023] The developer calls existing resources in the digital human resource management library. This library supports importing 3D models in FBX format that meet the rendering requirements of Three.js; at the same time, it selects a female voice in English and Chinese from the TTS tone library that is suitable for the education scene, making the explanation more realistic in the classroom; in the ASR recognition library, the "education question and answer" recognition scene is selected so that the digital human can understand and answer the knowledge point questions raised by students. In addition, the developer uploads a custom teaching knowledge base so that the digital human can call professional subject knowledge for interaction.

[0024] After completing the configuration, the developer sets up the node flow of "student asking questions-digital human answering" through the dialogue logic configuration panel, and selects the CPU and GPU models in the running parameter preset module, and the system will automatically match the adaptation strategy. Finally, the developer clicks on the preview test function to real-time view the rendering effect and voice interaction effect of the digital human during explanation, and troubleshoot possible rendering errors or voice synchronization problems in the simulated internal network running environment. After testing is completed, the system automatically outputs a 3D digital human system that can run on a server, laying a foundation for subsequent integration and deployment.

[0025] Step 3, based on the 3D digital human model, integrate a multi-channel voice intent analysis model, perform noise recognition and dialect matching, analyze user intent, determine the answer content from the answer knowledge base, and output the answer voice according to the answer content; In the embodiment of the application, a multi-modal interactive terminal based on a 3D digital human model is deployed at the front desk of a hospital. When a patient comes to the front desk, the patient asks the digital human for consultation through voice, for example, "I want to make an appointment for a pediatrician." Due to the noisy environment of the waiting hall, there are background broadcasts and crowd conversations. The system first preprocesses the patient's voice using an integrated multi-channel voice intent analysis model. After the multi-channel signals collected by the microphone array are processed in segments, the system automatically identifies and clips non-voice segments, extracts noise spectrum features, separates and suppresses environmental noise, and outputs a second voice signal that is clearer.

[0026] In the voice recognition link, it is detected that the patient uses Mandarin with a dialect accent, which affects the recognition accuracy. At this time, the system automatically calls a dialect recognition pre-classifier, compares the second voice signal with a dialect voice library, confirms that the voice belongs to a certain dialect accent, switches to the corresponding dialect recognition sub-model, and outputs a third voice signal to ensure the accuracy of voice recognition. The third voice signal is analyzed for intent, the keywords "appointment" and "pediatrics" are extracted, the context of the conversation is combined, the user's final intent is determined to be "make an appointment for a child to see a doctor", the corresponding business process information is retrieved from the medical knowledge base, and the answer content is generated: "You can make an appointment for pediatrics at the registration window on the first floor, or you can directly make an appointment by swiping your card at the self-service machine." Finally, the 3D digital human synthesizes the answer content into natural speech through TTS, and drives the model to generate corresponding lip synchronization and friendly expressions, and real-time broadcasts the answer to the patient, realizing a high-robustness voice interaction experience in a complex acoustic environment.

[0027] Step 4, render the 3D digital human model according to the answer content and user intent, drive the multi-modal interactive service to feedback, and real-time monitor the running state data for abnormal alarm.

[0028] The embodiment of the application first identifies the user's question type through a multi-channel voice intent analysis model based on user voice input, and simultaneously suppresses environmental noise to ensure analysis accuracy. After the intent is recognized, the system drives the 3D digital human model to render, synchronously outputs its facial expressions, gestures, and voice feedback, and provides an intuitive and natural interaction experience to the user. At the same time, the background monitoring module real-time collects running state data, including voice recognition delay, model rendering frame rate, system CPU / GPU usage, etc., and triggers an abnormal alarm based on a pre-set threshold, for example, if the recognition delay is too high or the rendering frame rate is reduced, to ensure stable operation of the service. The entire process not only realizes a closed-loop interaction from voice analysis to digital human feedback, but also ensures the reliability of the system in a high-concurrency, multi-task environment through state monitoring and alarm mechanisms.

[0029] Preferably, the 3D digital human model is selected based on a creation interface, specifically including: supporting drag-and-drop configuration of the 3D digital human model image, providing an expression binding tool and a motion editing panel, being capable of associating voice rhythm to configure lip movement synchronization parameters, supporting custom motion duration through a preset daily motion library, and not needing to manually write rendering codes, thereby corresponding to solve the problem of traditional creation relying on professional engines.

[0030] Preferably, the interactive parameters are configured based on a digital human resource management library, the digital human resource management library including an FBX format import support, a 3D model library adapted to Three.js rendering requirements, a TTS tone library supporting Chinese and English emotional tone selection, an ASR recognition library preset with general and industry-specific recognition scenarios, and supporting user uploading of own compliant resources, including personalized 3D digital human exclusive costumes and industry knowledge base. The interactive parameters are implemented through a dialogue logic configuration panel and a running parameter preset module, the dialogue logic configuration panel being used for providing visual node dragging, setting rules of user questioning and digital human response, and supporting answer knowledge base, and the running parameter preset module being capable of selecting adaptive CPU model and GPU model, target operating system, and automatically matching underlying adaptation strategy.

[0031] Preferably, a running package of the 3D digital human model is output for preview testing, and the preview testing specifically includes: supporting real-time preview of digital human rendering effect and voice interaction effect, simulating an internal network running environment, and troubleshooting rendering errors, voice asynchronization and other problems in advance.

[0032] It should be noted that, in the running package generation stage, a Shader translation layer is added, which can automatically translate the GLSL shader code required by the digital human rendering into a GPU supported instruction set and perform cache optimization to avoid runtime compilation overhead.

[0033] In the virtual assistant system of the service hall, the 3D digital human model configuration and integration method based on a creation interface is adopted. The staff first selects and configures the digital human image through the visual creation interface, the interface supports drag-and-drop operation, and the user does not need to master the development skills of professional rendering engines to complete the basic image building of the digital human. The expression binding tool provided by the system can automatically bind common facial expressions and animation skeletons, the motion editing panel supports adjusting the motion amplitude and duration through the time axis, and can be synchronized with the voice rhythm to automatically generate lip movement effects that meet the lip shape. By calling the preset daily motion library such as nodding, gestures, smiling, etc., the staff can quickly combine natural interaction actions that meet the government affairs scene, and can customize the duration of each action, thereby significantly reducing the threshold of traditional dependence on professional engine script development.

[0034] During the interaction parameter configuration phase, staff customized the system based on the digital human resource management library. This library supports the import of 3D model files in formats such as FBX and automatically adapts to Three.js rendering requirements, ensuring efficient rendering of the model in a browser environment. Simultaneously, the system has a built-in TTS voice library, providing multiple emotional voice options in both Chinese and English to ensure the digital human possesses a friendly and approachable voice. The ASR recognition library has preset recognition optimization strategies for general and government service scenarios, enabling the digital human to accurately understand the user's intent in their inquiries. Furthermore, users can upload their own compliant resources, such as configuring personalized clothing for the digital human or importing knowledge bases from specific fields, achieving customized expansion of the digital human. Through the dialogue logic configuration panel, staff can intuitively set the logical rules for user questions and digital human responses by dragging and dropping nodes, and link external knowledge bases to achieve multi-turn dialogues. The runtime parameter preset module supports selecting different CPU models, GPU models, and target operating systems; the system can automatically match the corresponding underlying adaptation strategies, simplifying the complexity of cross-platform deployment.

[0035] During the runtime package generation phase, the system supports preview testing, allowing users to preview the digital human's rendering and voice interaction effects in real time on a browser. This allows for early detection of potential rendering errors or lip-sync issues in a simulated intranet environment. To facilitate this process, a shader translation layer is introduced to automatically translate the GLSL shader code required for Three.js rendering into the GPU-supported low-level instruction set. Combined with caching optimization strategies, this avoids the performance overhead of runtime dynamic compilation. This mechanism ensures the stability and real-time performance of the digital human rendering even on GPU platforms within a domestic IT environment. Ultimately, a 3D digital human runtime package containing rendering logic, voice interaction logic, and hardware adaptation strategies is output and can be directly copied to the intranet server of the government service hall for practical application.

[0036] Preferably, a multi-channel speech intent parsing model is integrated based on a 3D digital human model to perform noise recognition and dialect matching, parse the user's intent, determine the answer content from the answer knowledge base, and output the answer speech based on the answer content. The specific steps are as follows: The user's voice signal is acquired through a voice acquisition terminal, and the user's voice signal is preprocessed to output the first voice signal. The recognition accuracy of the first speech signal in noisy environments is improved by using a multi-channel speech intent parsing model, and a second speech signal is output. The dialect recognition pre-classifier combines the second speech signal to query and match the dialect type in the dialect speech database. Based on the dialect type, the exclusive dialect recognition sub-model is determined to output the third speech signal. The user intention is analyzed according to the third voice signal, and the answer content is determined from the answer knowledge base, and the answer voice is output according to the answer content through TTS.

[0037] In the consultation scene of the service hall, a multi-channel voice intention analysis system based on a 3D digital human model is deployed. The user interacts with the digital human through the voice collection terminal in the hall. First, the microphone array will collect the user's voice signal in real time, and through the end-side preprocessing module, it will complete the mute trimming, noise segment elimination and endpoint detection, output the first voice signal, and provide clean input for subsequent recognition.

[0038] According to the characteristics of dense flow and noisy environment in the government office, the system calls a multi-channel voice intention analysis model, uses the spatial features and spectral features of multi-channel audio signals to realize beamforming and adaptive noise suppression, thereby improving the recognition accuracy of the first voice signal in a noisy environment, and generating a second voice signal. This step effectively filters the background broadcast sound and crowd conversation sound, ensuring the clarity of the user's voice content.

[0039] After obtaining the second voice signal, the system will automatically call a dialect recognition pre-classifier to compare and analyze the voice signal with the preset dialect voice library to determine the specific dialect category used by the user. For example, when the user's voice is detected to have obvious Wu dialect features, the system automatically selects the corresponding Wu dialect recognition sub-model to transcribe the second voice signal with higher accuracy, outputting a third voice signal, and significantly improving the recognition accuracy in a multi-dialect environment.

[0040] Subsequently, the user intention is analyzed based on the third voice signal. For example, when the user asks, "What materials do I need to prepare for social security transfer?", the intention analysis module will match the voice transcription result with the preset government consultation semantic library to determine that the user's intention belongs to the "social security transfer material consultation" category, and the system will retrieve the corresponding answer from the background answer knowledge base to obtain a complete answer content. Through TTS, the answer content is converted into natural and fluent voice, and is synchronized with the digital human's lip movement, and the 3D digital human model performs voice broadcasting on the large screen in a lively and natural way. For example, the digital human will smile and respond to the user with mouth movements: "Hello, to handle social security transfer, you need to prepare an ID card, a social security card, and a transfer voucher from the original participating place. This effectively solves the problems of noise interference, low dialect recognition rate and unnatural interaction, and ensures the efficient application of 3D digital humans in actual government service scenarios.

[0041] Preferably, the multi-channel voice intention analysis model is used to improve the recognition accuracy of the first voice signal in a noisy environment, and output a second voice signal, the specific steps are as follows: The multi-channel voice audio is collected by a microphone array, and the multi-channel voice audio is segmented based on a fixed time period to obtain a to-be-detected audio segment; Spectrum features of the to-be-detected audio segment are extracted, non-voice audio segments of the to-be-detected audio segment are cropped according to the spectrum features, and remaining voice audio segments are obtained; wherein the non-voice audio segments include mute spectrum segments and noise spectrum segments; Noise features are captured according to the noise spectrum segments, and whether the noise in the remaining voice audio segments affects speech recognition is detected according to the noise features; If the speech recognition is not affected, the remaining voice audio segments are output as second voice signals; if the speech recognition is affected, the remaining voice audio segments are marked as difficult-to-recognize audio segments; Based on voice audio segments of a previous time period and voice audio segments of a subsequent time period of the difficult-to-recognize audio segments, secondary speech recognition detection is performed on the difficult-to-recognize audio segments.

[0042] In the embodiment of the application, spectrum analysis is performed on each to-be-detected audio segment, spectrum features thereof are extracted, and non-voice segments such as long-time mute segments and environmental noise segments are automatically cropped according to the features, so that remaining voice audio segments containing user utterances are obtained. Then, the system further detects the remaining voice audio segments by using noise features extracted from noise spectrum segments, and judges whether the noise in the remaining voice audio segments substantially affects the accuracy of speech recognition.

[0043] If the detection result shows that the noise does not interfere with speech recognition, the remaining voice audio segments are directly output as second voice signals, and enter the subsequent dialect matching and intent analysis process. For example, when the environment is quiet, the user asks “I want to query the medical insurance payment record”, and the system can successfully obtain a clear voice signal and correctly recognize it.

[0044] However, in another scenario, when there are crowd conversations and broadcast playing in the hall at the same time, the system detects that the remaining voice audio segments are significantly affected by noise, which may reduce the recognition accuracy, and automatically marks the segment as a “difficult-to-recognize audio segment”. At this time, the system does not immediately output the result, but combines the voice audio of the previous time period and the subsequent time period of the difficult-to-recognize audio segment to perform secondary speech recognition detection. By correlating and analyzing the acoustic features and semantic information of the context, the missing or ambiguous content in the user's voice is inferred.

[0045] For example, when the user says "I want [fuzzy segment] social security transfer", due to background noise interference, the system fails to recognize the middle segment completely. But combined with the semantics of the previous period "I want" and the next period "social security transfer", the secondary detection module can infer that the fuzzy segment should be "handle", and finally output the correct result "I want to handle social security transfer". As can be seen from this example, the embodiments of the present application not only can efficiently output clear speech recognition results in a quiet environment, but also can improve recognition accuracy through context combination and secondary detection mechanism in a noisy environment, so as to ensure stable, natural and reliable voice interaction between the 3D digital person and the user.

[0046] Preferably, noise features are captured according to the noise spectrum segment, and noise in the remaining speech audio segment is detected according to the noise features to determine whether the noise has an impact on speech recognition. The specific steps are as follows: Noise features are captured according to the noise spectrum segment. The noise features include first noise features and second noise features. The time sequence transformation value of the noise spectrum is calculated As the first noise feature, wherein, is the power spectrum value corresponding to the nth frame of the kth frequency index in the xth noise spectrum segment, is the power spectrum value corresponding to the n+1th frame of the kth frequency index in the xth noise spectrum segment, is the power spectrum value corresponding to the nth frame of the kth frequency index in the xth noise spectrum segment, is the power spectrum value corresponding to the n+1th frame of the kth frequency index in the xth noise spectrum segment, A primary speech recognition detection model is constructed according to the noise features to detect noise in the remaining speech audio segment and determine whether the noise has an impact on speech recognition. The formula of the primary speech recognition detection model is: In the formula: is a noise detection factor, is the power spectrum value corresponding to the nth frame of the kth frequency index in the xth noise spectrum segment, is the power spectrum value corresponding to the n+1th frame of the kth frequency index in the xth noise spectrum segment, is the power spectrum value corresponding to the nth frame of the kth frequency index in the xth noise spectrum segment, is the power spectrum value corresponding to the n+1th frame of the kth frequency index in the xth noise spectrum segment, is the mean of the xth noise spectrum segment, is the variance of the xth noise spectrum segment, is the minimum frame index in the noise frame set, is the total number of frame sequence indexes, is the frequency index, is the frame sequence index.

[0047] It should be noted that, , is the complex spectrum value corresponding to the nth frame of the kth frequency index in the xth noise spectrum segment.

[0048] In actual application, the embodiment of the present application judges whether the noise will affect the speech recognition through the calculated noise detection factor. For example, in a relatively quiet scene, the broadcast sound and the background low-frequency noise are detected as the noise spectrum segment, the noise detection factor value is low, the system determines that the influence on the recognition is small, and therefore the remaining speech signal is directly sent to the speech recognition engine.

[0049] However, in another scenario, when the user asks a question in the hall, someone nearby talks loudly, causing the power spectrum of the noise spectrum segment to change dramatically, the mean is high and the variance is large, and the calculated noise detection factor exceeds the set threshold, which determines that the noise will significantly affect the accuracy of speech recognition. At this time, the system marks the segment as a "difficult-to-recognize audio segment" and triggers the subsequent secondary recognition strategy, combines the context semantics or calls the deep neural network enhancement model for noise compensation.

[0050] Preferably, the difficult-to-recognize audio segment is subjected to secondary speech recognition detection based on the speech audio segment of the previous period and the speech audio segment of the subsequent period, and the specific steps include: The difficult-to-recognize audio segment is subjected to ASR processing to obtain the confidence of each frame; The difficult-to-recognize audio segment is subjected to ASR processing to obtain the confidence of each frame; The speech audio segment of the previous period The speech audio segment of the subsequent period is marked as a fuzzy segment; By inputting and as conditions into the Transformer language model, the candidate filling content of the fuzzy segment is predicted, and the consistency scores of the candidate filling content with and are calculated through a text embedding model, and the candidate filling content corresponding to the highest consistency score is selected as the filling content; The filling content is output again as a second speech signal.

[0051] It should be noted that if the speech audio segment of the previous period and the speech audio segment of the subsequent period of the difficult-to-recognize audio segment also have difficult-to-recognize audio segments, then the system is rolled back to the nearest recognized speech segment or the nearest recognized speech segment for determining the filling content.

[0052] In the embodiment of the present application, the speech audio segment of the previous period of the difficult-to-recognize audio segment is "today we", the speech audio segment of the subsequent period is "meet to discuss the project", the candidate filling content of the fuzzy segment predicted by the Transformer language model is "want" and "just want", and the candidate filling content corresponding to the highest consistency score is "want". At this time, the filling content is "want".

[0053] In the intelligent voice customer service system of some call center, users often have a conversation with digital people in a noisy environment, such as in a subway station or a shopping mall. Due to environmental noise interference, the recognition confidence of some voice segments is too low, and these segments are marked as "difficult-to-recognize audio segments". For example, in a voice interaction, after ASR processing of the difficult-to-recognize audio segment, it is found that the frame-level confidence of the output text is generally lower than the preset threshold, so it is determined that this segment is a fuzzy segment. In order to avoid recognition interruption, the voice segments before and after the difficult-to-recognize audio segment are further extracted and input as context conditions into the language prediction model based on Transformer. The model predicts possible candidate fill-in content according to the context. For example, the user's continuous voice is "I want to check … today's … weather", and the "today's" part is recognized as fuzzy. The system uses "I want to check" and "weather" as conditions to predict multiple candidate segments such as "today's", "tomorrow's", and "this week's".

[0054] Subsequently, a text embedding model such as BERT or Sentence-BERT is called to calculate the overall semantic consistency score of the candidate fill-in content and. Finally, the system selects "today's" with the highest consistency score as the optimal candidate content and outputs it as the fill-in recognition result.

[0055] It should be noted that in another case, if the adjacent time periods of the difficult-to-recognize audio segment and are also determined to be fuzzy segments, the system will further backtrack to find the nearest correctly recognized voice segment forward or backward to provide more stable context conditions. In this way, even in the case of continuous multiple audio segments with recognition difficulties, the system can still rely on context prediction to fill in the missing content, thereby ensuring the coherence of the conversation and the integrity of the recognition.

[0056] Preferably, the specific steps of analyzing the user's intention according to the third voice signal are as follows: generating a first-level voice text through ASR recognition according to the third voice signal, and performing natural language processing on the first-level voice text to obtain first intention features; preliminarily classifying the first intention features through an intention classification model to output a first intention category of the user, determining second user features of the user in combination with the context of the first-level voice text and the first intention category, and determining the user's intention according to the second user features.

[0057] Specifically, the third voice signal is input into an automatic speech recognition system to generate a corresponding first-level speech text, the first-level speech text is subjected to natural language processing, and first intention features are extracted therefrom, the first intention features including semantic vectors, keywords and related entity information; The first intention features are input into an intention classification model for preliminary classification to obtain a first intention category of the user, and based on context information of the first-level speech text and the first intention category, second user features of the user are further inferred, the second user features including a user identity type, interest preferences and current use scenario information; The first intention category and the second user features are combined and input into a final intention reasoning model to generate a final intention of the user, thereby realizing multi-stage analysis, understanding and accurate intention recognition of the user voice.

[0058] In a certain intelligent service hall, a user proposes a business consultation to a 3D digital person terminal through voice. First, a third voice signal of the user is received and input into an automatic speech recognition system to generate a first-level speech text. For example, the user says "I want to apply for a residence permit".

[0059] Next, the system performs natural language processing on the first-level speech text to extract first intention features. At this time, the system extracts keywords "residence permit" and "apply" through word segmentation and entity recognition, and obtains corresponding semantic vectors through a semantic vectorization model, and identifies related entities "certificate type = residence permit" and "operation = apply". These constitute the first intention features.

[0060] Subsequently, the system inputs the first intention features into an intention classification model for preliminary classification to obtain a first intention category "certificate application" of the user. On this basis, the system further infers second user features of the user in combination with context information of the first-level speech text and the first intention category. For example, according to the dialogue history, the system identifies that the user identity type is "migrant worker", the interest preference is "fast certificate application", and the current use scenario is "government affairs hall offline consultation".

[0061] Finally, the first intention category "certificate application" and the second user features, such as the user identity type, the interest preference and the use scenario, are input into a final intention reasoning model to generate a final intention of the user, i.e., "apply for a residence permit for a migrant worker", which is transmitted to a business processing module, and the digital person immediately calls a government affairs knowledge base to return a corresponding answer: "Hello, the residence permit can be applied for at the window on the first floor of this hall, and an ID card, a labor contract and a rental contract need to be prepared." Through the above process, not only can the user's surface needs be accurately identified, but also deeper intent inference can be made by combining the user's identity and context information. This achieves multi-stage voice understanding and accurate intent recognition, ensuring the naturalness and personalization of the interaction process.

[0062] Preferably, the 3D digital human model is rendered based on the answer content and the user's intent, driving multimodal interactive service feedback. The multimodal interactive service feedback includes driving the 3D digital human model to generate corresponding expressions and actions based on the answer content and the user's intent, and synthesizing speech through TTS to achieve lip-movement synchronization.

[0063] The Three.js rendering of 3D digital human models involves adjusting the video memory read / write logic and optimizing the shader compilation rules through a GPU rendering optimization plugin. This supports real-time loading of digital human models and smooth rendering of expressions and movements, avoiding the performance loss and errors associated with traditional engines adapting to GPUs.

[0064] Real-time monitoring of operational status data triggers anomaly alerts. Specifically, the monitoring of operational status data includes CPU utilization, GPU utilization, and rendering frame rate. The operational status data is monitored in real time and compared with preset thresholds. If the operational status data is greater than or equal to the preset threshold, an anomaly alert will be triggered.

[0065] In an embodiment of this invention, in a smart bank lobby, a user asks a question via voice to a digital human terminal: "I would like to check my credit card statement for this month." The system first parses the user's voice and, based on a knowledge base, generates the response: "Your credit card statement for this month is 5680 yuan. You can choose to pay in full or in installments." After receiving the response and the user's intent, the system initiates multimodal interactive service feedback, driving the 3D digital human model to generate corresponding expressions and movements. Specifically, the digital human's face displays a smiling expression, its hands move naturally, and during voice broadcasting, a natural speech is synthesized using a TTS engine, with real-time lip movement parameters driving the synchronization of speech and lip movements, thereby generating the 3D digital human model.

[0066] Meanwhile, the system monitors operational status data in real time, including CPU utilization, GPU utilization, and rendering frame rate. If the monitoring results show that CPU utilization exceeds 85%, GPU utilization is close to full load, or the rendering frame rate is below 30fps, the system will immediately trigger an alarm and prompt maintenance personnel to check the hardware or optimize the model's performance parameters. For example, during a peak period, as multiple users interacted with the digital human simultaneously, GPU utilization spiked to 95%, and the system automatically triggered an alarm and reduced the rendering resolution of some non-critical actions in the background to maintain the overall smoothness of the interaction.

[0067] Through the implementation process, the digital person can stably run in the GPU environment, not only ensuring the natural interaction effect of voice and action expression, but also realizing dynamic monitoring and abnormal alarm processing of the running state, and ensuring the high availability and robustness of the system in complex environments.

[0068] Compared with the prior art, the method steps of the embodiment of the present application realize significant breakthroughs in user experience, precision efficiency, cost control, operation convenience and the like, and the specific advantages are as follows: The embodiment of the present application realizes experience upgrading through three technical optimizations: first, Three.js customization modification adapts to GPU, solves the rendering pipeline and driving compatibility problem, improves the fluency of digital person expression and action, and avoids texture tearing, shadow fault and the like; second, TTS / ASR large model operator reconstruction and calculation logic optimization eliminates the performance loss of the translation layer, the ASR recognition response is more timely, the naturalness of TTS synthesized voice is significantly improved, the recognition accuracy in dialect and noisy environment is greatly improved, and the problem of "answering irrelevant questions" is avoided; third, multi-modal data synchronization optimization, voice recognition results and action driving data are efficiently transmitted through a message queue, the problem of "voice and lip movement out of sync" is completely solved, the digital person interaction is closer to the real human communication scene, and the user perception and satisfaction are significantly improved.

[0069] The embodiment of the present application realizes precision and efficiency double improvement through technical innovation: at the CPU adaptation level, the instruction set native compilation is adopted instead of the translation layer to eliminate the inference delay and avoid voice recognition lag, and large-size digital person models are loaded quickly through memory cache layer optimization; at the GPU and large model adaptation level, the large model mixed precision inference is realized based on the GPU precision control interface, the inference speed is greatly improved under the premise of ensuring the ASR recognition accuracy and TTS voice naturalness, the digital person changes from "slow response" to "real-time interaction", and the core performance indicators completely meet the scene demand of financial and other scenes with strict requirements on precision and efficiency.

[0070] The embodiment of the present application realizes a significant cost saving through systematic design: in terms of labor cost, the middleware adaptation layer encapsulates a unified calling interface to shield the interface differences of different databases and message queues, without the need to develop separately for different middleware, reducing the adaptation development workload by more than 70%; in terms of time cost, the internal network deployment integrated package integrates rendering dependent libraries, GPU offline drivers, large model pre-training files and other resources to realize "one-key deployment", the deployment time is shortened from more than 24 hours of the prior art to within 2 hours, and the lightweight operation and maintenance module supports automatic fault alarm and recovery, and the fault repair time is shortened from 2 hours to within 15 minutes; in terms of material cost, the Three.js lightweight rendering framework has low demand for hardware resources, and high-end hardware does not need to be purchased to realize high-quality rendering, reducing hardware procurement and maintenance costs, so that small and medium-sized customers can also afford the deployment of digital person systems.

[0071] The embodiment of the application reduces the use threshold through humanized design: in the deployment operation, the integrated installation package automatically detects the intranet environment, the operating system version and the hardware model, completes the adaptive configuration of the driver and the dependent library, and does not need manual debugging, so that non-professionals can also complete the deployment; in the maintenance operation, the lightweight operation and maintenance module monitors the system running state in real time, intuitively displays the key indicators such as CPU / GPU usage and rendering frame rate, automatically triggers an alarm and provides recovery suggestions when a fault occurs, and ordinary operation and maintenance personnel can complete daily maintenance; the overall operation process is simplified from “professional and complex” to “dumb type” operation, which greatly reduces the technical threshold and promotes the rapid landing application of the digital human system in more industries.

[0072] In summary, compared with the prior art, the application realizes all-round optimization in user experience, precision efficiency, cost control and operation convenience, not only solves the core technical bottleneck of the 3D digital human system in the security creation environment, but also lays a foundation for the large-scale application of digital human technology in key industries such as finance and education through cost reduction and efficiency improvement and simplification of operation, and has significant technical advantages and practical application value.

[0073] Embodiment 2 The 3D digital human system integration system based on the security creation environment includes a 3D digital human system deployment interaction module, a 3D digital human creation preview module, a voice intent analysis and answer generation module, and a 3D digital human rendering and multi-modal interaction module; the 3D digital human system deployment interaction module is connected with the 3D digital human creation preview module, the 3D digital human creation preview module is connected with the voice intent analysis and answer generation module, and the voice intent analysis and answer generation module is connected with the 3D digital human rendering and multi-modal interaction module.

[0074] The 3D digital human system deployment interaction module is used for copying the integrated package of the 3D digital human system to a specified server through an offline mode, and performing interactive running through a terminal; The 3D digital human creation preview module is used for selecting a 3D digital human model through a creation interface of the 3D digital human system, configuring interaction parameters based on a digital human resource management library, and performing preview testing; The voice intent analysis and answer generation module is used for integrating a multi-channel voice intent analysis model based on the 3D digital human model, performing noise recognition and dialect matching, analyzing user intent, determining answer content from an answer knowledge base, and outputting answer voice according to the answer content; The 3D digital human rendering and multi-modal interaction module is used for rendering the 3D digital human model according to the answer content and the user intent, driving a multi-modal interaction service to feedback, and monitoring running state data in real time to perform abnormal alarm.

[0075] For example, Figure 2As shown, it is a system block diagram of the 3D digital human system integration system based on the Xinchuang environment, which can be used for executing Figure 1 The steps in the method embodiment are similar in implementation principle and technical effect, and will not be described here.

[0076] Embodiment 3 The embodiment of the present application is used to show an application of the 3D digital human system integration method based on the Xinchuang environment, which is applied to the service hall intranet environment, deploys a "3D digital human consultant" system, supports the service user to interact with the 3D digital human through the hall touch terminal, queries the service information such as social security payment, medical insurance reimbursement, and approval process, realizes "7*24 hours uninterrupted consultation", and replaces part of the manual window service.

[0077] The core goal of the embodiment of the present application is to adapt to the Xinchuang environment, which can realize smooth digital human rendering, that is, frame rate ≥ 30fps, natural voice interaction, that is, ASR dialect recognition rate ≥ 82%, simple deployment and maintenance, that is, non-professionals can complete basic operation and maintenance, realize continuous operation for 72 hours without failure, and stable performance with fault automatic recovery time ≤ 15 minutes.

[0078] The implementation steps of the 3D digital human system integration method based on the Xinchuang environment specifically include a creative terminal configuration stage and an intranet deployment stage.

[0079] The creative terminal configuration stage includes the following steps: creative terminal login and permission verification, digital human image and resource configuration, and preview test and running package generation.

[0080] Creative terminal login and permission verification, specifically: the technical personnel use the intranet PC to open the browser, and input the creative terminal address.

[0081] Digital human image and resource configuration, specifically: Select the basic model: enter the "visual editing module", click the left "model library", select the "consultant-female" 3D digital human model, such as Figure 3 The editing interface of the 3D digital human model shown in the figure; Configure the voice resource: enter the "resource management module-TTS tone library", select "neutral female voice", set the speech speed and tone; such as Figure 4 The 3D digital human model tone selection interface shown in the figure.

[0082] Preview test and running package generation, specifically: Preview test: click "preview test module-simulate interaction", input voice "I want to check how to pay social security payment", voice input supports microphone input or text input, preview 3D digital human: Rendering effect: no "plastic feeling" on the face, clear clothing texture, no shadow section; Voice interaction: ASR recognition is accurate, and the display is "query social security payment process", TTS synthesized voice is natural, and lip movement is synchronized with voice; Answer content: Correctly output "social security payment can be made through APP, offline window and bank deduction, and the process matches the knowledge base content; Generate a running package: After confirming that the preview is correct, click on the release, and open the terminal machine to interact with the released webpage.

[0083] The internal network deployment stage includes the following steps: deployment environment detection and touch terminal adaptation.

[0084] Deployment environment detection, specifically: the operation and maintenance personnel log in to the internal network server, copy the running package to the server for deployment; Touch terminal adaptation, specifically: the operation and maintenance personnel log in to 10 touch terminals, and access the released 3D digital human project website through the internal network; The operation and maintenance personnel monitor the system 7x24 hours automatically.

[0085] The application based on the above embodiment is verified for implementation effect, including function verification, performance verification and ease of use verification.

[0086] The function verification specifically includes: Rendering effect: the digital human facial expression is natural, without "plastic skin" feeling, the motion is smooth, without stuttering or dissection, and the frame rate is stable at 35fps or above; Voice interaction: ASR recognition supports English and Chinese, with an identification rate of 85%, TTS synthesized voice naturalness score of 4.4 points, and a full score of 5 points, with a voice and lip movement synchronization error of less than 0.1 seconds; Knowledge base matching: 98% of the consultation questions can accurately match the knowledge base, without "answering irrelevant questions".

[0087] The performance verification specifically includes: Stability: continuous operation for 72 hours, with an average CPU usage of 62% and an average GPU usage of 75%, without crashes or service interruptions; Response speed: from the public asking questions to the digital human answering, the average response time is 1.8 seconds; Fault recovery: simulate GPU failure by manually disabling one GPU, the system automatically switches to the standby GPU, and the recovery time is 8 seconds.

[0088] Ease of use verification: non-graphic professional operation and maintenance personnel complete full deployment within 2 hours according to the embodiment of the application, without additional debugging.

[0089] Through the introduction of the above-mentioned embodiments, the 3D digital human system integration package is copied to the designated server in an offline manner, and is run interactively through the terminal; a 3D digital human model is selected through the creation interface of the 3D digital human system, interactive parameters are configured based on the digital human resource management library, and preview testing is performed; a multi-channel speech intent analysis model is integrated based on the 3D digital human model, noise recognition and dialect matching are performed, user intent is analyzed, and the answer content is determined from the answer knowledge base, and the answer speech is output according to the answer content; the 3D digital human model is rendered according to the answer content and the user intent, the multi-modal interactive service feedback is driven, and the running state data is monitored in real time to perform abnormal alarm.

[0090] The application realizes the landing goal of simple creation, convenient deployment and stable running of the digital human of the signal creation through the three-layer architecture design of "creation-running-adaptation", reduces the creation threshold of the digital human through the webpage creation end, solves the compatibility problem of CPU and GPU through the bottom signal creation adaptation layer, and finally realizes the landing goal of simple creation, convenient deployment and stable running of the digital human of the signal creation, which is different from the traditional digital human system relying on the engine and unable to adapt to the signal creation environment; the application integrates the multi-channel speech intent analysis model, noise recognition and dialect matching, improves the robustness and accuracy of speech recognition, enables the 3D digital human to understand the user intent in a complex environment, and enhances the naturalness and intelligence of the interactive experience; the application outputs the speech in combination with the answer knowledge base and drives the 3D digital human model to render the multi-modal feedback, realizes the synchronization of speech, expression and action, provides an immersive and personalized interactive experience, and improves the user satisfaction and service efficiency.

[0091] The above is only a specific embodiment of the application, but the protection scope of the application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the application, which should be included in the protection scope of the application.

[0092] Finally, the above-mentioned is only the preferred solution of the application, and is not used to limit the application, any modification, equivalent replacement, improvement, etc. within the spirit and principle of the application should be included in the protection scope of the application.

Claims

1. A method for integrating a 3D digital human system based on an information technology innovation environment, characterized in that: Includes the following steps: Copy the 3D digital human system integration package to the designated server offline and run it interactively via the terminal; Select a 3D digital human model through the creation interface of the 3D digital human system, configure the interaction parameters based on the digital human resource management library, and conduct a preview test; Based on a 3D digital human model, a multi-channel speech intent parsing model is integrated to perform noise recognition and dialect matching, parse user intent, determine the answer content from the answer knowledge base, and output the answer speech based on the answer content. The system renders the 3D digital human model based on the answers and user intent, drives multimodal interactive service feedback, and monitors the operational status data in real time to issue anomaly alerts.

2. The 3D digital human system integration method based on the information technology innovation environment according to claim 1, characterized in that, The 3D digital human system's creation interface allows users to select a 3D digital human model, which includes: drag-and-drop configuration of the 3D digital human model image, providing facial expression binding tools and an action editing panel, the ability to configure lip movement synchronization parameters in relation to speech rhythm, and support for customizing action duration through a preset daily action library.

3. The method for integrating a 3D digital human system based on an information technology innovation environment according to claim 1, characterized in that, Based on a 3D digital human model and integrating a multi-channel speech intent parsing model, noise recognition and dialect matching are performed to parse user intent and determine the answer content from an answer knowledge base. The answer speech is then output based on the answer content. The specific steps are as follows: The user's voice signal is acquired through a voice acquisition terminal, and the user's voice signal is preprocessed to output the first voice signal. The recognition accuracy of the first speech signal in noisy environments is improved by using a multi-channel speech intent parsing model, and a second speech signal is output. The dialect recognition pre-classifier combines the second speech signal to query and match the dialect type in the dialect speech database. Based on the dialect type, the exclusive dialect recognition sub-model is determined to output the third speech signal. The user's intent is analyzed based on the third voice signal, and the answer content is determined from the answer knowledge base. The answer voice is then output based on the answer content through TTS.

4. The method for integrating a 3D digital human system based on an information technology innovation environment according to claim 3, characterized in that, The recognition accuracy of the first speech signal in noisy environments is improved by using a multi-channel speech intent parsing model, and a second speech signal is output. The specific steps are as follows: Multi-channel speech audio is acquired through a microphone array, and the multi-channel speech audio is segmented based on a fixed time period to obtain the audio segment to be detected; Extract the spectral features of the audio segment to be detected, and trim the non-speech audio segments of the audio segment to be detected based on the spectral features to obtain the remaining speech audio segments; among them, the non-speech audio segments include the silence spectrum segment and the noise spectrum segment; Based on the noise spectrum segment, noise characteristics are captured, and based on the noise characteristics, noise in the remaining speech audio segment is detected to determine whether it affects speech recognition. If it does not affect speech recognition, the remaining audio segments will be output as the second speech signal; if it does affect speech recognition, the remaining audio segments will be marked as difficult-to-recognize audio segments. Based on the speech audio segments preceding and following the difficult-to-identify audio segments, a second speech recognition detection is performed on the difficult-to-identify audio segments.

5. The 3D digital human system integration method based on the information technology innovation environment according to claim 4, characterized in that, The noise features are captured based on the noise spectrum band, and the noise in the remaining speech audio segment is detected based on these features to determine whether it affects speech recognition. The specific steps are as follows: Noise characteristics are captured based on the noise spectrum, which includes a first noise characteristic and a second noise characteristic. The time-series transformation value of the noise spectrum is then calculated. As the first noise characteristic, among which... For the xth noise spectrum segment The power spectrum value corresponding to the (n+1)th frame of the frequency index. For the xth noise spectrum segment The power spectrum value corresponding to the nth frame of each frequency index is used to calculate the mean and variance of the noise spectrum segment as the second noise feature. A primary speech recognition detection model is constructed based on noise characteristics to detect whether noise in the remaining speech audio segments affects speech recognition.

6. The method for integrating a 3D digital human system based on an information technology innovation environment according to claim 4, characterized in that, Based on the speech audio segments preceding and following the difficult-to-identify audio segments, a secondary speech recognition detection is performed on the difficult-to-identify audio segments. The specific steps include: For difficult-to-identify audio segments Perform ASR processing to obtain the confidence score for each frame; Extracting difficult-to-identify audio segments Previous audio segment The next audio segment The audio segments that are difficult to identify Marked as an ambiguous segment; By as well as The conditions are input into the Transformer language model to predict the candidate filling content for the ambiguous segment. The candidate filling content is then calculated using a text embedding model. as well as Based on the overall semantic consistency score, the candidate content corresponding to the highest consistency score is selected as the filling content; The filled content is then output again as a second speech signal.

7. The method for integrating a 3D digital human system based on an information technology innovation environment according to claim 3, characterized in that, The specific steps for parsing the user's intent based on the third voice signal are as follows: Based on the third speech signal, ASR recognition is performed to generate first-level speech text, and natural language processing is performed on the first-level speech text to obtain first intent features; The intent classification model is used to initially classify the first intent features and output the user's first intent category. Combining the context of the first-level speech text with the first intent category, the user's second user features are determined, and the user's intent is determined based on the second user features.

8. The method for integrating a 3D digital human system based on an information technology innovation environment according to claim 1, characterized in that, The 3D digital human model is rendered based on the answer and the user's intent, driving multimodal interactive service feedback. The multimodal interactive service feedback includes driving the 3D digital human model to generate corresponding expressions and actions based on the answer and the user's intent, and synthesizing speech through TTS to achieve lip-sync.

9. The method for integrating a 3D digital human system based on an information technology innovation environment according to claim 1, characterized in that, Real-time monitoring of operational status data triggers anomaly alerts. Specifically, the monitoring of operational status data includes CPU utilization, GPU utilization, and rendering frame rate. The operational status data is monitored in real time and compared with preset thresholds. If the operational status data is greater than or equal to the preset threshold, an anomaly alert will be triggered.

10. A 3D digital human system integration system based on an information technology innovation environment, applied to the 3D digital human system integration method based on an information technology innovation environment as described in any one of claims 1-9, characterized in that, The system includes a 3D digital human system deployment and interaction module, a 3D digital human creation and preview module, a voice intent parsing and response generation module, and a 3D digital human rendering and multimodal interaction module; The 3D digital human system deployment and interaction module is used to copy the 3D digital human system integration package to a designated server offline and run it interactively through a terminal. The 3D digital human creation and preview module is used to select a 3D digital human model through the creation interface of the 3D digital human system, configure interactive parameters based on the digital human resource management library, and conduct preview tests. The speech intent parsing and response generation module is used to integrate a multi-channel speech intent parsing model based on a 3D digital human model, perform noise recognition and dialect matching, parse user intent, determine response content from the response knowledge base, and output response speech based on the response content. The 3D digital human rendering and multimodal interaction module is used to render the 3D digital human model based on the answer content and user intent, drive multimodal interaction service feedback, and monitor the running status data in real time to issue anomaly alarms.

Images