3D digital human system integration method and system based on xinchuang environment

Abstract

The application relates to the technical field of software development, and particularly discloses a 3D digital human system integration method and system based on a Xinchuang environment, which copies a 3D digital human system integration package to a specified server in an offline mode, and performs interactive operation through a terminal; a 3D digital human model is selected through a 3D digital human system compiling interface, interactive parameters are configured based on a digital human resource management library, and preview testing is performed; a multi-channel voice intention analysis model is integrated based on the 3D digital human model, noise recognition and dialect matching are performed, user intention is analyzed, answering content is determined from an answering knowledge base, answering voice is output according to the answering content, the 3D digital human model is rendered according to the answering content and the user intention, a multi-modal interactive service feedback is driven, and running state data is monitored in real time to perform abnormality alarm; the application can realize high performance, high stability and easy-to-land application of the 3D digital human system in the Xinchuang environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of software development, more particularly, the present application relates to a 3D digital human system integration method and system based on a signal creation environment. BACKGROUND

[0002] With the development of artificial intelligence and computer graphics, 3D digital humans are gradually applied to government services, online education, financial consulting, virtual customer service and other scenarios. Existing 3D digital human systems usually rely on commercial rendering engines and voice interaction large models optimized based on CUDA architecture, and can achieve relatively natural real-time interaction effects in foreign GPU and general operating system environments. However, in the context of domestic substitution, the 3D digital human system exposes a series of core problems when running in the signal creation environment.

[0003] In terms of GPU rendering, the existing rendering engine is different from the GPU in the rendering pipeline, the driving interface and the instruction set, which leads to a significant decline in rendering performance, frequent rendering errors such as texture tearing and shadow discontinuity, and extremely high compatibility adaptation difficulty. Secondly, in terms of CPU architecture and system integration, the 3D digital human system needs to be deeply coupled with the CPU, the operating system, the database and the middleware. However, the current CPU architecture is diversified, the existing operating system interface is not unified, and the domestic database and message queue have deficiencies in interface compatibility and transaction consistency, which can easily cause problems such as loss of interaction data, delay in multi-modal data synchronization and performance decline, and the overall integration complexity increases significantly. In addition, in terms of deployment and maintenance, different operating systems, hardware devices and network environments can affect the stability and compatibility of the digital human system. Especially in the customer intranet environment, problems such as missing dependent libraries and driver mismatch frequently occur, which leads to a long deployment period, high maintenance cost and the need for manual debugging by professional personnel, which brings great obstacles to actual landing. More prominently, in the voice interaction link, the existing speech synthesis (TTS) and speech recognition (ASR) large models are developed and optimized based on CUDA architecture, and when directly migrated to the GPU, problems such as low efficiency of computing power scheduling, decline in model accuracy, unnatural speech synthesis, low dialect recognition rate and recognition failure in noisy environments often occur, which seriously affects the interaction experience.

[0004] Therefore, it is necessary to provide a 3D digital human system integration method and system based on a signal creation environment to solve the above technical problems. In order to solve the above problems, a technical solution is provided. SUMMARY

[0005] In order to overcome the above-mentioned defects of the prior art, the present application provides a 3D digital human system integration method and system based on a signal creation environment, which is used to solve the problems of poor rendering adaptation, difficult system integration, complex deployment and maintenance, and difficult migration of voice large models in the application process of the existing 3D digital human system.

[0006] To achieve the above object, the present application provides the following technical solutions:

[0007] The 3D digital human system integration method based on the Xinheng environment comprises the following steps:

[0008] Copy the integration package of the 3D digital human system to the designated server through an offline mode, and perform interactive operation through a terminal;

[0009] Select a 3D digital human model through the creation interface of the 3D digital human system, configure interactive parameters based on the digital human resource management library, and perform preview testing;

[0010] Integrate a multi-channel speech intent analysis model based on the 3D digital human model, perform noise recognition and dialect matching, analyze user intent, determine the answer content from the answer knowledge base, and output the answer speech according to the answer content;

[0011] Render the 3D digital human model according to the answer content and user intent, drive the multi-modal interactive service feedback, and monitor the running state data in real time to perform abnormal alarm.

[0012] As a further scheme of the present application, the 3D digital human model is selected through the creation interface of the 3D digital human system, specifically comprising: supporting drag-and-drop configuration of the 3D digital human model image, providing an expression binding tool and a motion editing panel, being able to associate speech rhythm to configure lip motion synchronization parameters, and supporting custom motion duration through a preset daily motion library.

[0013] As a further scheme of the present application, the multi-channel speech intent analysis model is integrated based on the 3D digital human model, noise recognition and dialect matching are performed, user intent is analyzed, the answer content is determined from the answer knowledge base, and the answer speech is output according to the answer content, specifically comprising the following steps:

[0014] Obtain the user speech signal through a speech acquisition terminal, and pre-process the user speech signal to output a first speech signal;

[0015] According to the multi-channel speech intent analysis model, the recognition accuracy of the first speech signal in a noise environment is improved, and a second speech signal is output;

[0016] Query the matching dialect type in the dialect speech library by combining the second speech signal through a dialect recognition pre-classifier, determine the exclusive dialect recognition sub-model according to the dialect type, and output a third speech signal;

[0017] According to the third speech signal, the user intent is analyzed, the answer content is determined from the answer knowledge base, and the answer speech is output according to the answer content through TTS.

[0018] As a further scheme of the present application, the recognition accuracy of the first speech signal in a noisy environment is improved according to a multi-channel speech intention analysis model, and a second speech signal is output, and the specific steps are as follows:

[0019] The multi-channel speech audio is collected by a microphone array, and the multi-channel speech audio is segmented based on a fixed time period to obtain a to-be-detected audio segment.

[0020] The spectral features of the to-be-detected audio segment are extracted, the non-speech audio segment of the to-be-detected audio segment is cropped according to the spectral features, and a remaining speech audio segment is obtained; wherein the non-speech audio segment includes a mute spectrum segment and a noise spectrum segment.

[0021] The noise features are captured according to the noise spectrum segment, and the noise in the remaining speech audio segment is detected according to the noise features to determine whether the noise has an impact on speech recognition.

[0022] If the noise has no impact on speech recognition, the remaining speech audio segment is output as a second speech signal; if the noise has an impact on speech recognition, the remaining speech audio segment is marked as a difficult-to-recognize audio segment.

[0023] The difficult-to-recognize audio segment is subjected to secondary speech recognition detection based on a speech audio segment of a previous time period and a speech audio segment of a subsequent time period.

[0024] As a further scheme of the present application, the noise features are captured according to the noise spectrum segment, and the noise in the remaining speech audio segment is detected according to the noise features to determine whether the noise has an impact on speech recognition, and the specific steps are as follows:

[0025] The noise features include first noise features and second noise features, and the time sequence transformation value of the noise spectrum is calculated As the first noise features, wherein, is a power spectrum value corresponding to the n+1th frame of the xth noise spectrum segment, is a power spectrum value corresponding to the nth frame of the xth noise spectrum segment, The mean value and the variance of the noise spectrum segment are calculated as the second noise features.

[0026] A primary speech recognition detection model is constructed according to the noise features to detect the noise in the remaining speech audio segment and determine whether the noise has an impact on speech recognition.

[0027] As a further scheme of the present application, the difficult-to-recognize audio segment is subjected to secondary speech recognition detection based on a speech audio segment of a previous time period and a speech audio segment of a subsequent time period, and the specific steps include:

[0028] The difficult-to-recognize audio segment is subjected to secondary speech recognition detection based on a speech audio segment of a previous time period and a speech audio segment of a subsequent time period. ​obtaining the confidence of each frame by ASR processing;

[0029] extracting the difficult-to-recognize audio segment the speech audio segment of the previous period the speech audio segment of the next period the difficult-to-recognize audio segment marked as a fuzzy segment;

[0030] by inputting and into the Transformer language model as conditions, predicting the candidate filling content of the fuzzy segment, calculating the consistency scores of the candidate filling content and and the overall semantics by a text embedding model, and selecting the candidate filling content corresponding to the highest consistency score as the filling content;

[0031] based on the filling content, outputting the second speech signal again.

[0032] As a further scheme of the present application, the specific steps of analyzing the user's intention according to the third speech signal are as follows:

[0033] generating a first-level speech text by ASR recognition according to the third speech signal, and obtaining a first intention feature by natural language processing on the first-level speech text;

[0034] preliminarily classifying the first intention feature by an intention classification model, outputting the first intention category of the user, determining a second user feature of the user in combination with the context of the first-level speech text and the first intention category, and determining the user's intention according to the second user feature.

[0035] As a further scheme of the present application, the 3D digital human model is rendered according to the answer content and the user's intention, and a multi-modal interactive service feedback is driven, wherein the multi-modal interactive service feedback includes generating corresponding expressions and actions by the 3D digital human model according to the answer content and the user's intention, and synthesizing speech and realizing lip movement synchronization by TTS.

[0036] As a further scheme of the present application, the running state data is monitored in real time to generate an abnormal alarm, specifically: the running state data includes CPU usage, GPU usage, and rendering frame rate, the running state data is monitored in real time, the running state data is compared with a preset threshold, and if the running state data is greater than or equal to the preset threshold, an abnormal alarm is triggered.

[0037] The 3D digital human system integration system based on the signal creation environment includes a 3D digital human system deployment interaction module, a 3D digital human creation preview module, a speech intention analysis and answer generation module, and a 3D digital human rendering and multi-modal interaction module.

[0038] The 3D digital human system deployment interaction module is used for copying the integrated package of the 3D digital human system to a specified server in an offline manner, and performing interactive operation through a terminal;

[0039] The 3D digital human creation preview module is used for selecting a 3D digital human model through a creation interface of the 3D digital human system, configuring interaction parameters based on a digital human resource management library, and performing preview testing;

[0040] The voice intention analysis and answer generation module is used for integrating a multi-channel voice intention analysis model based on a 3D digital human model, performing noise recognition and dialect matching, analyzing user intention, determining answer content from an answer knowledge base, and outputting answer voice according to the answer content;

[0041] The 3D digital human rendering and multi-modal interaction module is used for rendering the 3D digital human model according to the answer content and the user intention, driving multi-modal interaction service feedback, and monitoring running state data in real time to perform abnormal alarm.

[0042] The 3D digital human system integration method and system based on the Xinchuang environment have the following technical effects and advantages: the integrated package of the 3D digital human system is copied to a specified server in an offline manner, and interactive operation is performed through a terminal; a 3D digital human model is selected through a creation interface of the 3D digital human system, interaction parameters are configured based on a digital human resource management library, and preview testing is performed; a multi-channel voice intention analysis model is integrated based on a 3D digital human model, noise recognition and dialect matching are performed, user intention is analyzed, answer content is determined from an answer knowledge base, and answer voice is output according to the answer content; the 3D digital human model is rendered according to the answer content and the user intention, multi-modal interaction service feedback is driven, and running state data is monitored in real time to perform abnormal alarm.

[0043] The present application has a three-layer architecture design of "creation-operation-adaptation", which not only reduces the threshold of digital human creation through the web creation end, but also solves the compatibility problem of CPU and GPU through the underlying Xinchuang adaptation layer, finally realizes the goal of simple creation, convenient deployment and stable operation of Xinchuang digital human, which is different from the traditional digital human system relying on engine and unable to adapt to Xinchuang environment; the present application integrates multi-channel voice intention analysis model, noise recognition and dialect matching, improves the robustness and accuracy of voice recognition, enables 3D digital human to understand user intention in complex environment, and enhances the naturalness and intelligence of interactive experience; combined with the output of voice from the answer knowledge base and the rendering of 3D digital human model, the present application realizes the synchronization of voice, expression and action, provides immersive and personalized interactive experience, and improves user satisfaction and service efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1A flowchart of a 3D digital human system integration method based on a Xinyuan environment provided by an embodiment of the present application is shown in FIG.

[0045] Figure 2 A system block diagram of a 3D digital human system integration system based on a Xinyuan environment provided by an embodiment of the present application is shown in FIG.

[0046] Figure 3 An editing interface of a 3D digital human model provided by an embodiment of the present application is shown in FIG.

[0047] Figure 4 A timbre selection interface of a 3D digital human model provided by an embodiment of the present application is shown in FIG. DETAILED DESCRIPTION

[0048] The technical solutions in the present application will be described in detail below with reference to the accompanying drawings of the present application. Obviously, the described technical solutions are only a part of the present application, but not all. Based on the technical solutions in the present application, all other technical solutions obtained by those skilled in the art without creative efforts fall within the scope of the present application.

[0049] Embodiment 1

[0050] As shown in FIG. Figure 1 A flowchart of a 3D digital human system integration method based on a Xinyuan environment provided by an embodiment of the present application is shown in FIG. Figure 1 The execution subject of the method shown in FIG. can be a software and / or a hardware device. The execution subject of the present application can include but is not limited to at least one of the following: a user device, a network device, etc. The user device can include but is not limited to a computer, a smart phone, a personal digital assistant (PDA), and the above-mentioned electronic devices, etc. The network device can include but is not limited to a single network server, a server group composed of multiple network servers, or a cloud composed of a large number of computers or network servers based on cloud computing, wherein cloud computing is a kind of distributed computing, which is a super virtual computer composed of a loose coupled computer group. The present embodiment does not make any limitation. Steps 1 to 4 are included, and the details are as follows:

[0051] Step 1, copy the integration package of the 3D digital human system to the designated server through an offline mode, and interactively run through a terminal;

[0052] The present embodiment effectively solves the distribution and installation problems of the running package in the Internet isolation environment through the offline copying mode, and ensures the controllability and security of the system on the server. At the same time, the integration package of the 3D digital human system is directly loaded through the terminal, avoiding the traditional complex installation and network adaptation problems, and realizing the rapid delivery and plug-and-play deployment of the digital human in the government office.

[0053] The 3D digital human system integration package is a running unit generated in the creation stage and the preview test stage, has the ability of direct deployment and running in the domestic software and hardware environment, not only contains rendering resources of the 3D digital human model, including model files, action library, expression library, Shader translation files, but also contains multi-channel voice analysis modules related to voice interaction, dialogue logic configuration files and running environment configuration files, including driving interface parameters adapted to CPU / GPU and operating system compatibility parameters; the "plug and play" type deployment is realized, the safety and controllability in the signal creation environment are ensured, and the complexity of deployment and maintenance is significantly reduced, so that the digital human can be quickly applied to scenes such as government offices, education classrooms and medical diagnosis.

[0054] Step 2, select a 3D digital human model through the creation interface of the 3D digital human system, configure interaction parameters based on the digital human resource management library, and perform preview test;

[0055] In the construction process of a certain smart education platform, the developer uses the visual creation interface provided by the 3D digital human system to select and customize a classroom assistant type 3D digital human model. The specific operation is as follows: through the drag-and-drop configuration function, the developer selects a young teacher image with affinity in the interface, then sets up basic expressions such as "nodding", "smiling", "concentrating" and the like for the model using the expression binding tool, and sets up "gesture when explaining" and "board writing action" through the action editing panel, and at the same time, automatically associates the lip movement synchronization parameter with the voice rhythm to ensure the consistency of the mouth shape and the explanation content.

[0056] The developer calls existing resources in the digital human resource management library. The library supports importing 3D models in FBX format, meeting the rendering requirements of Three.js; at the same time, it selects a female voice in English and Chinese in the TTS tone library that is adapted to the education scene, so that the explanation is closer to the real classroom; in the ASR recognition library, the "education question and answer" recognition scene is selected, so that the digital human can understand and answer the knowledge point questions raised by the students. In addition, the developer also uploads a custom teaching knowledge base so that the digital human can call professional subject knowledge for interaction.

[0057] After the configuration is completed, the developer sets the node flow of "student question-digital human answer" through the dialogue logic configuration panel, and selects the CPU model and GPU model in the running parameter preset module, and the system will automatically match the adaptation strategy. Finally, the developer clicks the preview test function to real-time view the rendering effect and voice interaction effect of the digital human when explaining, and troubleshoots possible rendering errors or voice synchronization problems in the simulated intranet running environment. After the test is completed, the system automatically outputs a 3D digital human system that can run on the server, laying a foundation for subsequent integration and deployment.

[0058] Step 3, based on the 3D digital human model, integrate multi-channel speech intent analysis model, perform noise recognition and dialect matching, analyze user intent, and determine answer content from the answer knowledge base, output answer speech according to the answer content;

[0059] In the embodiment of the application, a multi-modal interactive terminal based on a 3D digital human model is deployed at the front desk of a hospital. When a patient comes to the front desk, the patient asks the digital human for consultation through voice, for example, "I want to make an appointment for a pediatrician." Due to the noisy environment of the waiting hall, there are background broadcasts and crowd conversations. The system first preprocesses the patient's voice using the integrated multi-channel speech intent analysis model. After the multi-channel signals collected by the microphone array are processed in segments, the system automatically identifies and clips the non-voice segments, extracts noise spectrum features, separates and suppresses the environmental noise, and outputs a clearer second voice signal.

[0060] In the speech recognition link, it is detected that the patient uses Mandarin with a dialect accent, and the recognition accuracy is affected. At this time, the system automatically calls the dialect recognition pre-classifier, compares the second voice signal with the dialect voice library, confirms that the voice belongs to a certain dialect accent, switches to the corresponding dialect recognition sub-model, and outputs a third voice signal to ensure the accuracy of speech recognition. The third voice signal is analyzed for intent, the keywords "registration" and "pediatrics" are extracted, the context of the conversation is combined, and the final intent of the user is determined to be "register for a child's medical visit". Then, the corresponding business process information is retrieved from the medical knowledge base, and the answer content is generated: "You can register for the pediatrician at the registration window on the first floor, or you can directly register for the card on the self-service machine."

[0061] Finally, the 3D digital human synthesizes the answer content into natural speech through TTS, and drives the model to generate corresponding lip synchronization and friendly expressions, and real-time broadcasts the answer to the patient, realizing high-robustness voice interaction experience in complex acoustic environment.

[0062] Step 4, render the 3D digital human model according to the answer content and user intent, drive the multi-modal interactive service feedback, and real-time monitor the running state data for abnormal alarm.

[0063] The embodiment of the application first identifies the problem type of the user according to the user voice input through a multi-channel voice intention analysis model, while suppressing environmental noise to ensure the analysis accuracy. After the intention is identified, the system drives the 3D digital human model to render, synchronously outputs the facial expression, gesture action and voice feedback, and provides intuitive and natural interactive experience to the user. At the same time, the background monitoring module collects the running state data in real time, including voice recognition delay, model rendering frame rate, system CPU / GPU usage, etc., and triggers an abnormal alarm based on the preset threshold, for example, high recognition delay or decreased rendering frame rate, to ensure the stable operation of the service. The whole process not only realizes the closed-loop interaction from voice analysis to digital human feedback, but also guarantees the reliability of the system in a high-concurrency and multi-task environment through the state monitoring and alarm mechanism.

[0064] Preferably, the 3D digital human model is selected based on a creation interface, specifically including: supporting drag-and-drop configuration of the 3D digital human model image, providing expression binding tools and action editing panels, being able to associate voice rhythm to configure lip movement synchronization parameters, supporting custom action duration through a preset daily action library, and not needing to manually write rendering codes, which solves the problem of traditional creation relying on professional engines.

[0065] Preferably, the interaction parameters are configured based on a digital human resource management library, the digital human resource management library includes an FBX format import supporting library, a 3D model library adapted to the rendering requirements of Three.js, a TTS voice library supporting Chinese and English emotional voice selection, an ASR recognition library preset with general and industry-specific recognition scenarios, and supports user uploading of own compliant resources, including personalized 3D digital human exclusive costumes and industry knowledge base;

[0066] The configuration of the interaction parameters is realized through a dialogue logic configuration panel and a running parameter preset module. The dialogue logic configuration panel is used to provide visual node dragging, set the rules of user questioning and digital human response, and support answer knowledge base. The running parameter preset module can select the appropriate CPU model and GPU model, target operating system, and automatically match the underlying adaptation strategy.

[0067] Preferably, the running package of the 3D digital human model is output for preview testing, which specifically includes: supporting real-time preview of digital human rendering effect and voice interaction effect, simulating an internal network running environment, and troubleshooting rendering errors, voice asynchronization and other problems in advance.

[0068] It should be noted that a Shader translation layer is added in the running package generation stage, which can automatically translate the GLSL shader code required by the digital human rendering into the instruction set supported by the GPU and perform cache optimization to avoid runtime compilation overhead.

[0069] This invention employs a 3D digital human model configuration and integration method based on a creation interface in the virtual assistant system of the service hall. Staff first select and configure the digital human avatar through a visual creation interface. The interface supports drag-and-drop operation, allowing users to complete the basic avatar setup without needing professional rendering engine development skills. The system's expression binding tool automatically binds common facial expressions to animation skeletons, while the action editing panel allows adjustment of action amplitude and duration via a timeline, and can synchronize with speech rhythm to automatically generate lip movement effects that match mouth shapes. By calling preset daily action libraries, such as nodding, gestures, and smiling, staff can quickly combine natural interactive actions suitable for government service scenarios and customize the duration of each action, thus significantly reducing the barrier to entry for traditional development relying on professional engine scripts.

[0070] During the interaction parameter configuration phase, staff customized the system based on the digital human resource management library. This library supports the import of 3D model files in formats such as FBX and automatically adapts to Three.js rendering requirements, ensuring efficient rendering of the model in a browser environment. Simultaneously, the system has a built-in TTS voice library, providing multiple emotional voice options in both Chinese and English to ensure the digital human possesses a friendly and approachable voice. The ASR recognition library has preset recognition optimization strategies for general and government service scenarios, enabling the digital human to accurately understand the user's intent in their inquiries. Furthermore, users can upload their own compliant resources, such as configuring personalized clothing for the digital human or importing knowledge bases from specific fields, achieving customized expansion of the digital human. Through the dialogue logic configuration panel, staff can intuitively set the logical rules for user questions and digital human responses by dragging and dropping nodes, and link external knowledge bases to achieve multi-turn dialogues. The runtime parameter preset module supports selecting different CPU models, GPU models, and target operating systems; the system can automatically match the corresponding underlying adaptation strategies, simplifying the complexity of cross-platform deployment.

[0071] During the runtime package generation phase, the system supports preview testing, allowing users to preview the digital human's rendering and voice interaction effects in real time on a browser. This allows for early detection of potential rendering errors or lip-sync issues in a simulated intranet environment. To facilitate this process, a shader translation layer is introduced to automatically translate the GLSL shader code required for Three.js rendering into the GPU-supported low-level instruction set. Combined with caching optimization strategies, this avoids the performance overhead of runtime dynamic compilation. This mechanism ensures the stability and real-time performance of the digital human rendering even on GPU platforms within a domestic IT environment. Ultimately, a 3D digital human runtime package containing rendering logic, voice interaction logic, and hardware adaptation strategies is output and can be directly copied to the intranet server of the government service hall for practical application.

[0072] Preferably, based on the 3D digital human model, a multi-channel speech intention analysis model is integrated, noise recognition and dialect matching are performed, user intention is analyzed, answer content is determined from an answer knowledge base, and answer speech is output according to the answer content, and the specific steps are as follows:

[0073] A user speech signal is acquired through a voice acquisition terminal, and a first speech signal is output by pre-processing the user speech signal;

[0074] The recognition accuracy of the first speech signal in a noisy environment is improved according to a multi-channel speech intention analysis model, and a second speech signal is output;

[0075] A dialect recognition pre-classifier is used to query a matching dialect type in a dialect speech library according to the second speech signal, a specific dialect recognition sub-model is determined according to the dialect type, and a third speech signal is output;

[0076] User intention is analyzed according to the third speech signal, and answer content is determined from an answer knowledge base, and answer speech is output according to the answer content through TTS.

[0077] In the consultation scene of the service hall, a multi-channel speech intention analysis system based on a 3D digital human model is deployed. The user interacts with the digital human through the voice acquisition terminal in the hall. First, the microphone array will acquire the user's speech signal in real time, and complete the mute clipping, noise segment rejection and endpoint detection through the terminal side preprocessing module, output the first speech signal, and provide clean input for subsequent recognition.

[0078] According to the characteristics of dense flow and noisy environment in the government office, the system calls a multi-channel speech intention analysis model, uses the spatial features and spectral features of multi-channel audio signals to realize beamforming and adaptive noise suppression, thereby improving the recognition accuracy of the first speech signal in a high-noise environment, and generating a second speech signal. This step effectively filters the background broadcast sound and crowd conversation sound, and ensures the clarity of the user's speech content.

[0079] After obtaining the second speech signal, the system will automatically call a dialect recognition pre-classifier to compare and analyze the speech signal with a preset dialect speech library to determine the specific dialect category used by the user. For example, when the user's speech is detected to have obvious Wu dialect features, the system automatically selects the corresponding Wu dialect recognition sub-model to transcribe the second speech signal with higher accuracy, and outputs a third speech signal, thereby significantly improving the recognition accuracy in a multi-dialect environment.

[0080] Subsequently, user intention analysis is performed based on the third voice signal, for example, when the user asks, "What materials do I need to prepare for social security transfer?", the intention analysis module matches the voice transcription result with the preset government affair consultation semantic library, determines that the user intention belongs to the "social security transfer material consultation" category, and the system retrieves the corresponding answer from the background answer knowledge base to obtain a complete answer content. The answer content is converted into natural and fluent voice by TTS, and is bound with the lip movement of the digital person, and the 3D digital person model performs voice broadcast in a lively and natural way on the large screen. For example, the digital person will smile and respond to the user with mouth movement: "Hello, you need to prepare an ID card, a social security card and a transfer voucher issued by the original participating insurance place. It effectively solves the problems of noise interference, low dialect recognition rate and unnatural interaction, and ensures the efficient application of 3D digital person in actual government service scenarios.

[0081] Preferably, the recognition accuracy of the first voice signal in a noisy environment is improved according to a multi-channel voice intention analysis model, and a second voice signal is output, and the specific steps are as follows:

[0082] The multi-channel voice audio is collected by a microphone array, and the multi-channel voice audio is segmented based on a fixed time period to obtain a to-be-detected audio segment;

[0083] The frequency spectrum features of the to-be-detected audio segment are extracted, and the non-voice audio segment of the to-be-detected audio segment is cut according to the frequency spectrum features to obtain a remaining voice audio segment; wherein the non-voice audio segment includes a mute spectrum segment and a noise spectrum segment;

[0084] Noise features are captured according to the noise spectrum segment, and whether the noise in the remaining voice audio segment affects voice recognition is detected according to the noise features;

[0085] If the voice recognition is not affected, the remaining voice audio segment is output as a second voice signal; if the voice recognition is affected, the remaining voice audio segment is marked as a difficult-to-recognize audio segment;

[0086] The difficult-to-recognize audio segment is detected by secondary voice recognition based on the voice audio segment of the previous period and the voice audio segment of the next period.

[0087] The embodiment of the present application performs spectrum analysis on each to-be-detected audio segment, extracts its frequency spectrum features, and automatically cuts off the non-voice segment such as long-time mute segment and environmental noise segment according to these features, thereby obtaining the remaining voice audio segment containing user voice. Then, the system further detects the remaining voice audio segment using the noise features extracted from the noise spectrum segment to determine whether the noise in the remaining voice audio segment will substantially affect the accuracy of voice recognition.

[0088] If the detection result shows that the noise does not interfere with the speech recognition, the remaining speech audio segment is directly output as a second speech signal, and enters the subsequent dialect matching and intent analysis process. For example, when the environment is relatively quiet, the user asks "I want to query the medical insurance payment record", and the system can successfully obtain a clear speech signal and correctly recognize it.

[0089] However, in another scenario, when there are crowd conversations and broadcast playing in the hall at the same time, the system detects that the remaining speech audio segment is significantly affected by noise, which may reduce the recognition accuracy, and automatically marks the segment as a "difficult-to-recognize audio segment". At this time, the system does not immediately output the result, but combines the speech audio of the previous period and the next period of the difficult-to-recognize audio segment to perform secondary speech recognition detection. By correlating and analyzing the acoustic features and semantic information of the context, the missing or ambiguous content in the user's speech is inferred.

[0090] For example, when the user says "I want [ambiguous segment] to transfer social security", due to background noise interference, the system cannot completely recognize the middle segment. However, by combining the semantics of the previous period "I want" and the next period "social security transfer", the secondary detection module can infer that the ambiguous segment should be "handle", and finally output the correct result "I want to handle social security transfer". As can be seen from this example, the embodiments of the present application not only can efficiently output clear speech recognition results in a quiet environment, but also can improve recognition accuracy through context combination and secondary detection mechanism in a noisy environment, thereby ensuring stable, natural and reliable voice interaction between the 3D digital person and the user.

[0091] Preferably, noise features are captured according to the noise frequency spectrum, and whether the noise in the remaining speech audio segment affects speech recognition is detected according to the noise features. The specific steps are as follows:

[0092] Noise features are captured according to the noise frequency spectrum, and the noise features include first noise features and second noise features. The time sequence transformation value of the noise frequency spectrum is calculated As the first noise feature, wherein, is the power spectrum value corresponding to the n+1 frame of the xth noise frequency spectrum segment, is the power spectrum value corresponding to the n frame of the xth noise frequency spectrum segment, and the mean and variance of the noise frequency spectrum are calculated as the second noise feature, respectively. A primary speech recognition detection model is constructed according to the noise features to detect whether the noise in the remaining speech audio segment affects speech recognition, and the formula of the primary speech recognition detection model is:

[0093] ; in the formula,

[0094] ​​is a noise detection factor, is a power spectrum value corresponding to the nth frame of the kth frequency index in the xth noise spectrum segment, is a power spectrum value corresponding to the n+1th frame of the kth frequency index in the xth noise spectrum segment, is a mean value of the xth noise spectrum segment, is a variance of the xth noise spectrum segment, is a minimum frame index in the noise frame set, is a total number of frame sequence indexes, is a frequency index, is a frame sequence index. It should be noted that, ,

[0095] is a complex spectrum value corresponding to the nth frame of the kth frequency index in the xth noise spectrum segment. In actual application, the noise detection factor obtained by calculation is used to determine whether the noise will affect the speech recognition. For example, in a relatively quiet scene, the broadcast sound and the background low-frequency noise are detected as a noise spectrum segment, and the noise detection factor value is low, so the system determines that the influence on recognition is small, and therefore the remaining speech signal is directly sent to the speech recognition engine.

[0096] However, in another scenario, when the user asks a question in the hall, someone nearby talks loudly, causing the power spectrum of the noise spectrum segment to change dramatically, the mean value is high and the variance is large, and the calculated noise detection factor exceeds the set threshold, determining that the noise will significantly affect the accuracy of speech recognition. At this time, the system marks the segment as a "difficult-to-recognize audio segment" and triggers the subsequent secondary recognition strategy, combining the context semantics or calling a deep neural network enhancement model for noise compensation.

[0097] Preferably, the difficult-to-recognize audio segment is subjected to secondary speech recognition detection based on a speech audio segment of a previous period and a speech audio segment of a subsequent period, and the specific steps include:

[0098] the difficult-to-recognize audio segment is subjected to ASR processing to obtain the confidence of each frame;

[0099] the difficult-to-recognize audio segment is subjected to ASR processing to obtain the confidence of each frame; the difficult-to-recognize audio segment is subjected to ASR processing to obtain the confidence of each frame;

[0100] the difficult-to-recognize audio segment is subjected to ASR processing to obtain the confidence of each frame; the difficult-to-recognize audio segment is subjected to ASR processing to obtain the confidence of each frame; the difficult-to-recognize audio segment is subjected to ASR processing to obtain the confidence of each frame;

[0101] the difficult-to-recognize audio segment is subjected to ASR processing to obtain the confidence of each frame; ​​​​As a condition input into the Transformer language model, the candidate filling content of the fuzzy segment is predicted, and the overall semantic consistency score of the candidate filling content and the and the candidate filling content corresponding to the highest consistency score is selected as the filling content;

[0102] Based on the filling content, the second speech signal is output again.

[0103] It should be noted that if the speech audio segment before the difficult-to-identify audio segment and the speech audio segment after the difficult-to-identify audio segment also have difficult-to-identify audio segments, the system will retreat to the nearest recognized speech segment or retreat to the nearest recognized speech segment, which is used to determine the filling content.

[0104] In the embodiment of the application, the speech audio segment before the difficult-to-identify audio segment is "today we", and the speech audio segment after the difficult-to-identify audio segment is "meet to discuss the project". After the Transformer language model predicts the candidate filling content of the fuzzy segment, the candidate filling content corresponding to the highest consistency score is "want", and the filling content is "want".

[0105] In the intelligent voice customer service system of a certain call center, users often have conversations with digital humans in noisy environments, such as subway stations or shopping malls. Due to environmental noise interference, the recognition confidence of some speech segments is too low, and these segments are marked as "difficult-to-identify audio segments". For example, in a voice interaction, after ASR processing of the difficult-to-identify audio segment, it is found that the frame-level confidence of the output text is generally lower than the preset threshold, so it is determined that this segment is a fuzzy segment. In order to avoid interruption of recognition, the speech segment before the difficult-to-identify audio segment and the speech segment after the difficult-to-identify audio segment are further extracted and input as context conditions into the Transformer-based language prediction model. The model predicts possible candidate filling content based on the context. For example, the user's continuous speech is "I want to check … today's … weather", and the "today's" part is identified as fuzzy. The system uses "I want to check" and "weather" as conditions to predict multiple candidate segments, such as "today's", "tomorrow's", and "this week's".

[0106] Subsequently, a text embedding model such as BERT or Sentence-BERT is called to calculate the overall semantic consistency score of the candidate filling content and the candidate filling content. Finally, the system selects "today's" with the highest consistency score as the optimal candidate content and outputs it as the recognized result after filling.

[0107] It should be noted that in another case, if the adjacent time period of the difficult-to-identify audio segment And Also determined as a fuzzy segment, the system will further backtrack, looking forward to the nearest correctly identified speech segment, or looking backward to the nearest correctly identified speech segment, for providing more stable context conditions. In this way, even in the case of continuous multiple audio segments with identification difficulties, the system can still rely on context prediction to fill in the missing content, thereby ensuring the coherence of the dialogue and the integrity of the identification.

[0108] Preferably, the specific steps of analyzing the user's intention according to the third voice signal are as follows:

[0109] According to the third voice signal, a first-level voice text is generated by ASR identification, and a first intention feature is obtained by natural language processing on the first-level voice text;

[0110] The first intention feature is preliminarily classified by an intention classification model, and a first intention category of the user is output, the second user feature of the user is determined in combination with the context of the first-level voice text and the first intention category, and the user's intention is determined according to the second user feature.

[0111] It is worth noting that the third voice signal is input into an automatic speech recognition system to generate a corresponding first-level voice text, and the first intention feature is extracted by natural language processing on the first-level voice text, including semantic vectors, keywords and related entity information;

[0112] The first intention feature is input into an intention classification model for preliminary classification to obtain a first intention category of the user, and the second user feature of the user is further inferred based on the context information of the first-level voice text and the first intention category, including user identity type, interest preference and current use scenario information;

[0113] The first intention category and the second user feature are combined and input into a final intention reasoning model to generate the final intention of the user, thereby realizing multi-stage analysis, understanding and accurate intention recognition of the user's voice.

[0114] In a smart service hall, the user proposes business consultation to the 3D digital person terminal through voice. First, the third voice signal of the user is received and input into an automatic speech recognition system to generate a first-level voice text. For example, the user says "I want to apply for a residence card".

[0115] Next, the system performs natural language processing on the first-level speech text to extract the first intent feature. At this stage, the system extracts the keywords "residence permit" and "process" through word segmentation and entity recognition, and obtains the corresponding semantic vectors through a semantic vectorization model. Simultaneously, it identifies the relevant entities "document type = residence permit" and "operation = process". These constitute the first intent feature.

[0116] Subsequently, the system inputs the first intent feature into the intent classification model for preliminary classification, obtaining the user's first intent category as "document processing". Based on this, the system combines the contextual information of the first-level speech text with the first intent category to further infer the user's second user features. For example, based on the dialogue history, the system identifies the user's identity type as "migrant worker", their interest preference as "fast document processing", and their current usage scenario as "offline consultation at the government service hall".

[0117] Finally, the first intent category "document processing" and the second user characteristics, such as user identity type, interests and preferences, and usage scenarios, are input into the final intent reasoning model to generate the user's final intent, namely "to process residence permits for migrant workers". This final intent is transmitted to the business processing module, and the digital human then calls the government knowledge base to return the corresponding answer: "Hello, residence permits can be processed at the window on the first floor of this hall. You need to prepare your ID card, employment contract and rental contract."

[0118] Through the above process, not only can the user's surface needs be accurately identified, but also deeper intent inference can be made by combining the user's identity and context information. This achieves multi-stage voice understanding and accurate intent recognition, ensuring the naturalness and personalization of the interaction process.

[0119] Preferably, the 3D digital human model is rendered based on the answer content and the user's intent, driving multimodal interactive service feedback. The multimodal interactive service feedback includes driving the 3D digital human model to generate corresponding expressions and actions based on the answer content and the user's intent, and synthesizing speech through TTS to achieve lip-movement synchronization.

[0120] The Three.js rendering of 3D digital human models involves adjusting the video memory read / write logic and optimizing the shader compilation rules through a GPU rendering optimization plugin. This supports real-time loading of digital human models and smooth rendering of expressions and movements, avoiding the performance loss and errors associated with traditional engines adapting to GPUs.

[0121] Real-time monitoring of operational status data triggers anomaly alerts. Specifically, the monitoring of operational status data includes CPU utilization, GPU utilization, and rendering frame rate. The operational status data is monitored in real time and compared with preset thresholds. If the operational status data is greater than or equal to the preset threshold, an anomaly alert will be triggered.

[0122] In a smart bank hall, a user asks a digital human terminal for consultation through voice: "I want to check this month's credit card bill." The system first analyzes the user's voice and generates the answer content "Your credit card bill this month is 5680 yuan, you can choose full repayment or installment repayment" in combination with the knowledge base. After obtaining the answer content and the user's intention, the system starts the multi-modal interaction service feedback, and drives the 3D digital human model to generate corresponding expressions and actions. Specifically, the digital human face presents a smiling expression, the hands are accompanied by natural swinging, the voice is synthesized through the TTS engine when the voice is broadcast, and the synchronization of voice and lip movement is realized by driving the lip movement parameters in real time, so as to generate a 3D digital human model.

[0123] At the same time, the system monitors the running state data in real time, including CPU usage, GPU usage and rendering frame rate. If the monitoring result shows that the CPU usage is more than 85%, or the GPU usage is close to full load, or the rendering frame rate is less than 30fps, the system will immediately trigger an abnormal alarm and prompt the operation and maintenance personnel to check the hardware or optimize the model performance parameters. For example, during a peak period, due to the interaction of multiple users with the digital human at the same time, the GPU usage soared to 95%, and the system automatically triggered an alarm information, and automatically reduced the rendering resolution of part of the non-critical actions in the background, so as to maintain the fluency of the overall interaction.

[0124] Through the above implementation process, the digital human can stably run in the GPU environment, not only ensuring the natural interaction effect of voice and action expression, but also realizing dynamic monitoring and abnormal alarm processing of the running state, ensuring the high availability and robustness of the system in complex environment.

[0125] Based on the method steps of the embodiment of the application, compared with the prior art, the application realizes significant breakthroughs in user experience, precision efficiency, cost control, operation convenience and the like, and the specific advantages are as follows:

[0126] The embodiment of the application realizes experience upgrading through three technical optimizations: first, Three.js customization modification adapts to GPU, solves the rendering pipeline and driving compatibility problem, improves the fluency of digital human expression and action, and avoids texture tearing, shadow fault and other defects; second, TTS / ASR large model operator reconstruction and calculation logic optimization eliminates the performance loss of translation layer, ASR recognition response is more timely, TTS synthesized voice naturalness is significantly improved, dialect and noisy environment recognition accuracy is greatly improved, and "answering irrelevant questions" is avoided; third, multi-modal data synchronization optimization, voice recognition result and action driving data are efficiently transmitted through the message queue, the "voice and lip movement are not synchronized" problem is completely solved, the digital human interaction is closer to the real human communication scene, and the user perception and satisfaction are significantly improved.

[0127] The embodiment of the application realizes the improvement of both precision and efficiency through technical innovation: at the CPU adaptation level, the instruction set native compilation is adopted to replace the translation layer, the inference delay is eliminated, the voice recognition is avoided from lagging, and the large-size digital human model is optimized through memory cache layering to realize fast loading; at the GPU and large model adaptation level, the large model mixed precision inference is realized based on the GPU precision control interface, the inference speed is greatly improved under the premise of ensuring the ASR recognition accuracy and the TTS voice naturalness, the digital human changes from'slow response' to'real-time interaction', and the core performance indicators completely meet the needs of scenes such as finance which have strict requirements on precision and efficiency.

[0128] The embodiment of the application realizes a substantial cost saving through systematic design: in terms of labor cost, the middleware adaptation layer encapsulates a unified calling interface, shields the interface differences of different databases and message queues, does not need to be developed separately for different middleware, and reduces the adaptation development workload by more than 70%; in terms of time cost, the internal network deployment integrated package integrates rendering dependent libraries, GPU offline drivers, large model pre-training files and other resources, realizes 'one-key deployment', and shortens the deployment time from more than 24 hours of the prior art to within 2 hours, while the lightweight operation and maintenance module supports automatic fault alarm and recovery, and the fault repair time is shortened from 2 hours to within 15 minutes; in terms of material cost, the Three.js lightweight rendering framework has low demand for hardware resources, and high-quality rendering can be realized without the need to purchase high-end hardware, reducing hardware procurement and maintenance costs, so that small and medium-sized customers can also afford the deployment of the digital human system of the signal creation.

[0129] The embodiment of the application reduces the use threshold through humanized design: in terms of deployment operation, the integrated installation package automatically detects the internal network environment, the operating system version and the hardware model, completes the adaptation and configuration of the driver and the dependent library, does not need manual debugging, and non-professionals can also complete the deployment; in terms of maintenance operation, the lightweight operation and maintenance module monitors the system running state in real time, intuitively displays key indicators such as CPU / GPU usage and rendering frame rate, automatically triggers an alarm and provides recovery suggestions when a fault occurs, and ordinary operation and maintenance personnel can complete daily maintenance; the overall operation process is simplified from 'professional and complex' to 'dumb type' operation, greatly reducing the technical threshold, and promoting the rapid landing application of the digital human system of the signal creation in more industries.

[0130] In summary, compared with the prior art, the application realizes all-round optimization in user experience, precision and efficiency, cost control and operation convenience, not only solves the core technical bottleneck of the 3D digital human system in the signal creation environment, but also lays a foundation for the large-scale application of digital human technology in key industries such as finance and education through cost reduction and efficiency improvement and operation simplification, and has significant technical advantages and practical application value.

[0131] Embodiment 2

[0132] The 3D digital human system integration system based on the Xinchuang environment comprises a 3D digital human system deployment interaction module, a 3D digital human creation preview module, a voice intention analysis and answer generation module and a 3D digital human rendering and multi-modal interaction module; the 3D digital human system deployment interaction module is connected with the 3D digital human creation preview module, the 3D digital human creation preview module is connected with the voice intention analysis and answer generation module, and the voice intention analysis and answer generation module is connected with the 3D digital human rendering and multi-modal interaction module.

[0133] The 3D digital human system deployment interaction module is used for copying an integration package of the 3D digital human system to a designated server in an offline mode and performing interactive operation through a terminal.

[0134] The 3D digital human creation preview module is used for selecting a 3D digital human model through a creation interface of the 3D digital human system, configuring interaction parameters based on a digital human resource management library and performing preview testing.

[0135] The voice intention analysis and answer generation module is used for integrating a multi-channel voice intention analysis model based on the 3D digital human model, performing noise recognition and dialect matching, analyzing user intention, determining answer content from an answer knowledge base and outputting answer voice according to the answer content.

[0136] The 3D digital human rendering and multi-modal interaction module is used for rendering the 3D digital human model according to the answer content and the user intention, driving a multi-modal interaction service to feed back and monitoring running state data in real time to perform abnormal alarm.

[0137] As shown in Figure 2 , it is a system block diagram of the 3D digital human system integration system based on the Xinchuang environment, which can be used for executing the steps in the method embodiment shown in Figure 1 , and the implementation principle and technical effects are similar, which will not be repeated here.

[0138] Embodiment 3

[0139] The embodiment of the present application is used to show an application case of the 3D digital human system integration method based on the Xinchuang environment, which is applied to a service hall intranet environment, deploys a 3D digital human consultant system and supports users to interact with the 3D digital human through a hall touch terminal to query service information such as social security payment, medical insurance reimbursement and approval process, so as to realize "7x24 hours uninterrupted consultation" and replace part of manual window service.

[0140] The core goal of the embodiment of the application is to adapt to the signal creation environment, so as to realize fluent digital person rendering, i.e., frame rate ≥ 30 fps, natural voice interaction, i.e., ASR dialect recognition rate ≥ 82%, simple deployment and maintenance, i.e., non-professionals can complete basic operation and maintenance, realize stable performance of continuous operation for 72 hours without failure, and automatic recovery time of fault ≤ 15 minutes.

[0141] The implementation steps of the 3D digital person system integration method based on the signal creation environment specifically include a creation terminal configuration stage and an intranet deployment stage.

[0142] The creation terminal configuration stage includes the following steps: creation terminal login and permission verification, digital person image and resource configuration, and preview test and running package generation.

[0143] The creation terminal login and permission verification specifically includes that a technician uses an intranet PC to open a browser, and inputs a creation terminal address.

[0144] The digital person image and resource configuration specifically includes that:

[0145] Selecting a basic model: entering a visual editing module, clicking a left side model library, selecting a 3D digital person model of a consultant-female, and like a 3D digital person model editing interface shown in FIG. 6; Figure 3

[0146] Configuring voice resources: entering a resource management module-TTS tone library, selecting a neutral female voice, and setting a speech speed and a tone; like a 3D digital person model tone selection interface shown in FIG. 7. Figure 4

[0147] Preview test and running package generation specifically includes that:

[0148] Preview test: clicking a preview test module-simulated interaction, inputting a voice "I want to know how to pay social security", supporting microphone input or text input for voice input, and previewing a 3D digital person:

[0149] Rendering effect: no "plastic feeling" on a face, clear clothing texture, and no shadow section;

[0150] Voice interaction: ASR recognition is accurate, a display is "query social security payment process", TTS synthesized voice is natural, and lip movement is synchronized with voice;

[0151] Answer content: correctly outputting "social security payment can be realized through an APP, offline window, and bank deduction, and process matching knowledge base content;

[0152] Generating a running package: after confirming that the preview is correct, clicking publishing, and opening a terminal machine to use the published webpage for interaction.

[0153] ​​The intranet deployment stage includes the following steps: deployment environment detection, and touch terminal adaptation.

[0154] The deployment environment detection specifically includes: an operation and maintenance personnel logging in an intranet server, copying a running package to the server for deployment.

[0155] The touch terminal adaptation specifically includes: the operation and maintenance personnel logging in 10 touch terminals, and accessing a 3D digital human project website published through the intranet.

[0156] The operation and maintenance personnel daily monitor the system to automatically run 7x24 hours.

[0157] Based on the application of the above embodiment, implementation effect verification is performed, including function verification, performance verification, and ease of use verification.

[0158] The function verification specifically includes:

[0159] The rendering effect: the digital human facial expression is natural, without skin “plastic sense”, the motion is smooth, without stuttering or dissection, and the frame rate is stable at above 35 fps.

[0160] The voice interaction: the ASR recognition supports Chinese and English, the recognition rate is 85%, the TTS synthesized voice naturalness score is 4.4 points, wherein the full score is 5 points, and the voice and lip synchronization error is less than or equal to 0.1 seconds.

[0161] The knowledge base matching: 98% of the consultation questions can be accurately matched with the knowledge base, without “irrelevant answer”.

[0162] The performance verification specifically includes:

[0163] The stability: continuously running for 72 hours, the average CPU usage rate of the server is 62%, the average GPU usage rate is 75%, without crash or service interruption.

[0164] The response speed: from the question of the masses to the answer of the digital human, the average response time is 1.8 seconds.

[0165] The fault recovery: by manually disabling 1 GPU to simulate GPU failure, the system automatically switches to the standby GPU, and the recovery time is 8 seconds.

[0166] The ease of use verification: the non-graphic professional operation and maintenance personnel complete the full deployment within 2 hours according to the embodiment of the application, without additional debugging.

[0167] Through the introduction of the above embodiments, the application copies the integration package of the 3D digital human system to the designated server through an offline mode, and runs interactively through the terminal; the 3D digital human model is selected through the creation interface of the 3D digital human system, the interaction parameters are configured based on the digital human resource management library, and preview testing is performed; a multi-channel speech intent analysis model is integrated based on the 3D digital human model, noise recognition and dialect matching are performed, user intent is analyzed, and the answer content is determined from the answer knowledge base, and the answer speech is output according to the answer content; the 3D digital human model is rendered according to the answer content and the user intent, the multi-modal interactive service feedback is driven, and the running state data is monitored in real time to perform abnormal alarm.

[0168] The application realizes the landing goal of simple creation, convenient deployment and stable running of the digital human through the three-layer architecture design of "creation-running-adaptation", reduces the creation threshold of the digital human through the webpage creation end, solves the compatibility problem of CPU and GPU through the bottom layer of the signal creation adaptation layer, and finally realizes the landing goal of simple creation, convenient deployment and stable running of the signal creation digital human, which is different from the traditional digital human system relying on the engine and unable to adapt to the signal creation environment; the application integrates the multi-channel speech intent analysis model, noise recognition and dialect matching, improves the robustness and accuracy of speech recognition, enables the 3D digital human to understand the user intent in a complex environment, and enhances the naturalness and intelligence of the interactive experience; the application outputs the speech combined with the answer knowledge base and drives the 3D digital human model to render the multi-modal feedback, realizes the synchronization of speech, expression and action, provides an immersive and personalized interactive experience, and improves the user satisfaction and service efficiency.

[0169] The above is only a specific embodiment of the application, but the protection scope of the application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the application, which should be included in the protection scope of the application.

[0170] Finally, the above is only a preferred solution of the application, and is not used to limit the application, any modification, equivalent replacement, improvement, etc. within the spirit and principles of the application should be included in the protection scope of the application.

Claims

1. A 3D digital human system integration method based on a Xinheng environment, characterized in that, The method comprises the following steps: Copy the integrated package of the 3D digital human system to a designated server in an offline manner, and perform interactive operation through a terminal; Select a 3D digital human model through the creation interface of the 3D digital human system, configure interaction parameters based on a digital human resource management library, and perform preview testing; Integrate a multi-channel speech intent analysis model based on the 3D digital human model, perform noise recognition and dialect matching, analyze user intent, determine answer content from an answer knowledge base, and output answer speech according to the answer content, the specific steps being as follows: Obtain user speech signals through a speech collection terminal, and pre-process the user speech signals to output first speech signals; According to the multi-channel speech intent analysis model, improve the recognition accuracy of the first speech signals in a noisy environment, and output second speech signals, the specific steps being as follows: Collect multi-channel speech audio through a microphone array, and segment the multi-channel speech audio based on a fixed time period to obtain audio segments to be detected; Extract the spectral features of the audio segments to be detected, and cut the non-speech audio segments of the audio segments to be detected according to the spectral features to obtain remaining speech audio segments; wherein the non-speech audio segments include silent spectrum segments and noise spectrum segments; According to the noise spectrum segments, capture noise features, and detect whether the noise in the remaining speech audio segments affects speech recognition according to the noise features, the specific steps being as follows: According to the noise spectrum segment capturing noise features, the noise features include first noise features and second noise features, by calculating the time sequence transform value p x (k, n+1)-p x (k, n) as the first noise features, wherein, p x (k, n+1) is the power spectrum value corresponding to the kth frequency index and the n+1th frame in the xth noise spectrum segment, p x (k, n) is the power spectrum value corresponding to the kth frequency index and the nth frame in the xth noise spectrum segment, and the mean value and the variance of the noise spectrum segment are calculated respectively as the second noise features; According to the noise features, construct a first speech recognition detection model to detect whether the noise in the remaining speech audio segments affects speech recognition; If the speech recognition is not affected, the remaining speech audio segments are output as second speech signals; if the speech recognition is affected, the remaining speech audio segments are marked as difficult-to-recognize audio segments; Based on the speech audio segments of the previous period and the speech audio segments of the next period of the difficult-to-recognize audio segments, perform secondary speech recognition detection on the difficult-to-recognize audio segments, the specific steps including: For the unidentified audio segment X t Do ASR processing to obtain the confidence of each frame; extracting the difficult-to-recognize audio segment X t the speech audio segment X of the preceding time period t-1 the speech audio segment X of the following time period t+1 marking the difficult-to-recognize audio segment X t as an ambiguous segment By inputting X t-1 and X t+1 as conditions into the Transformer language model, predicting the candidate filling content of the ambiguous segment, calculating the consistency scores of the candidate filling content and X t-1 and X t+1 with the overall semantics respectively through the text embedding model, and selecting the candidate filling content corresponding to the highest consistency score as the filling content; Based on the filled content, output the second speech signals again; Through a dialect recognition pre-classifier, query a matching dialect type in a dialect speech library combined with the second speech signals, determine a dedicated dialect recognition sub-model according to the dialect type, and output third speech signals; According to the third speech signals, analyze user intent, and determine answer content from an answer knowledge base, and output answer speech according to the answer content through TTS; According to the answer content and the user intent, render the 3D digital human model, drive a multi-modal interactive service to feed back, and monitor running state data in real time to perform abnormal alarm.

2. The 3D digital human system integration method based on the Xinchuang environment according to claim 1, characterized in that, Select a 3D digital human model through the creation interface of the 3D digital human system, specifically including: support drag-and-drop configuration of the 3D digital human model image, provide expression binding tools and action editing panels, can associate speech rhythm to configure lip movement synchronization parameters, and support custom action duration through a preset daily action library.

3. The 3D digital human system integration method based on the Xinchuang environment according to claim 1, characterized in that, The specific steps of analyzing user intent according to the third speech signals are as follows: Perform ASR recognition on the third speech signals to generate first-level speech text, and perform natural language processing on the first-level speech text to obtain first intent features; The first intention feature is preliminarily classified through the intention classification model, and a first intention category of the user is output; second user features of the user are determined in combination with a context of the first-level voice text and the first intention category; and the user intention is determined according to the second user features.

4. The 3D digital human system integration method based on the Xinchuang environment according to claim 1, characterized in that, The 3D digital human model is rendered according to the answer content and the user intention, and a multi-modal interactive service feedback is driven, wherein the multi-modal interactive service feedback includes generating corresponding expressions and actions of the 3D digital human model according to the answer content and the user intention, and synthesizing speech through TTS and realizing lip movement synchronization.

5. The 3D digital human system integration method based on the Xinchuang environment according to claim 1, characterized in that, Real-time monitoring of running state data for abnormal alarm, specifically: monitoring running state data including CPU usage, GPU usage, rendering frame rate, respectively, real-time monitoring of running state data, comparing running state data with preset threshold, if running state data is greater than or equal to preset threshold, will trigger abnormal alarm.

6. The system integration system of the 3D digital human system based on the Xinchuang environment, applied to the system integration method of the 3D digital human system based on the Xinchuang environment as claimed in any one of claims 1-5, characterized in that, The system includes a 3D digital human system deployment interaction module, a 3D digital human creation preview module, a voice intention analysis and answer generation module, and a 3D digital human rendering and multi-modal interaction module. The 3D digital human system deployment interaction module is used to copy the integrated package of the 3D digital human system to the designated server through the offline mode, and to interact and run through the terminal. The 3D digital human creation preview module is used to select a 3D digital human model through the creation interface of the 3D digital human system, configure interaction parameters based on the digital human resource management library, and perform preview testing. The voice intention analysis and answer generation module is used to integrate a multi-channel voice intention analysis model based on the 3D digital human model, perform noise recognition and dialect matching, analyze the user intention, and determine the answer content from the answer knowledge base, and output the answer voice according to the answer content. The 3D digital human rendering and multi-modal interaction module is used to render the 3D digital human model according to the answer content and the user intention, drive the multi-modal interactive service feedback, and real-time monitor the running state data for abnormal alarm.

Images