Large-model-driven digital human real-time interaction method and interaction equipment based on mine

Abstract

The invention provides a large-model-driven digital human real-time interaction method and interaction equipment based on a mine, and belongs to the technical field of mine intellectualization. Building a voice recognition module, providing a voice recognition process through the voice recognition module, obtaining an audio file at a specified position of a current server, recognizing the audio file, and outputting a text; a mine large model module is configured, a text-to-voice module is configured, a digital human generation module is built, and the digital human generation module comprises a training set processing unit, a digital human training function unit, a digital human generation unit and a digital human image generation model. And configuring an audio and video synthesis stream pushing module, synthesizing the video information and the voice information by the audio and video synthesis stream pushing module, realizing audio and picture synchronization, and configuring the audio and video synthesis stream pushing module as stream pushing information. The digital human supports multi-dimensional interaction with the user in voice, text, chart and the like, and the user can dialogue with the system through a natural language to obtain more intuitive and convenient information feedback.

Description

Technical Field

[0001] The present invention belongs to the technical field of mine intelligence, and in particular relates to a digital human real-time interaction method and interaction equipment driven by a large mine model. Background Art

[0002] In the mining field, it is generally necessary to popularize or explain the mining operating environment, mining safety knowledge and related mining knowledge to meet the needs of intelligent mining use.

[0003] The traditional mine operation environment, mine safety knowledge and mining-related knowledge explanations rely on manual explanations, which makes it impossible to fully introduce mine-related information. Although some mines currently use computers to introduce through screen projection. However, relevant personnel need to read text to obtain it, which lacks interactivity and intuitiveness. Moreover, the mine environment is complex and the amount of data is large, so it is impossible to fully introduce and explain it. Related technologies can often only provide simple command input, cannot achieve real-time voice interaction, lack personalization, and are difficult to meet the use needs of mines. Summary of the invention

[0004] The present invention provides a real-time digital human interaction method driven by a large model of a mine, which can solve the problem of single interaction dimension of large model application and single application function of digital human in the mining industry. It provides users in the mining industry with rich large model application capabilities, and provides decision support for users in combination with mining data, while providing a good interactive experience of real-person dialogue.

[0005] Methods include: S101: Building a front-end module, which includes a text input unit, a voice input unit, a digital human video display unit, and a chart and text display unit; the voice input unit uploads the acquired voice information to the server; S102: Building a speech recognition module, providing a speech recognition process through the speech recognition module, and after receiving a speech recognition call request, obtaining an audio file at a specified location of the current server for recognition and outputting text; S103: configuring a mine large model module, the mine large model module includes a demand analysis unit, a large model application library and a data processing unit; S104: configuring a text-to-speech module, where the text-to-speech module includes a training set processing unit, a training function unit, a speech service unit, and a text-to-speech model; S105: Building a digital human generation module, the digital human generation module includes a training set processing unit, a digital human training unit, a digital human generation unit and a digital human image generation model; S106: Configuring an audio and video synthesis streaming module, the audio and video synthesis streaming module synthesizes the video information and the voice information to achieve audio and video synchronization and configures the synthesized information as streaming information.

[0006] It should be further explained that in the method, a relational automation module is constructed to analyze and process the voice information in the method, as well as semantic understanding, keyword analysis and identifier analysis of the voice information, and output an application index k; In the large model application library LLMAB n Matching user needs in LLMAB k ; The matching method is: LLMAB k =LLMAB RAF (text) The matching relationship data based on information input and information output is: data = DDF(LLMAB k (text),k) The matching relationship data data includes structured data of digital human description text and drawn charts.

[0007] It should be further explained that the method further includes: person Perform sentence segmentation and automatic annotation; Using the speech-to-text model asr Mark and correct self-broken sentences; Get personalized character voice based on the following text-to-speech model: audio i ,text i =TTSM datasets (audio person , model asr ), i=[1…n]; Among them, audio i ,text i It is an audio segment and the corresponding text segment.

[0008] It should be further explained that in the method, based on audio i ,text i Generate text-to-speech model tts , the formula is as follows: model tts =TTSM train (audio i ,text i ), i=[1…n].

[0009] It should be further explained that step S104 also includes: converting the text information into audio speaking , drives the generation of the digital human's mouth shape features and serves as the digital human's voice. The formula is as follows: audio speaking =TTSM serve (data text , model tts ,data sample ) Among them, data sample The parameter is audio i ,text i An audio and text clip of TTSM serve Generate reference samples of cloned sounds.

[0010] It should be further explained that in step S105, the digital human generation module supports the image cloning of the character by providing a speech video of a preset length. person As training samples for digital humans; Among them, the speech video person Perform data processing, extract features, and obtain blink action data au45 ; Get the audio file data of the video aud , audio feature data aud_eo , video frame data images , segment portrait data head , extract background data bg , segment body data body , Get face data landmarks , Get face tracking data track ; The specific formula is as follows: data types =DHGM datasets (OpenFace|process,video person ).

[0011] It should be further explained that step S105 is also based on data types Training of digital humans, the training methods include the following: model head =DHGM train (data types ) model head-lip =DHGM train (datatypes , model head ,fineturn) model dh =DHGM train (data types , model head-lip ,body) Among them, model head and model head-lip is the intermediate model for training, model dh It is the final trained digital human model.

[0012] It should be further explained that in the method, the speech audio converted by the digital human based on the text-to-speech module speaking The formula to drive the digital human to generate the corresponding digital speech mouth shape and head movement is as follows: RP,video speaking =DHGM generate (model dh ,audio speaking ) The audio and video synthesis push streaming module converts the video frame video speaking and speech audio speaking Synthesis, to achieve audio and video synchronization AVSSM synthesis , and realize the real-time streaming function, and realize the synchronous synthesis of audio and video digital human video dh , the formula is as follows: video dh =AVSSM synthesis (video speaking , audio speaking ,RP).

[0013] It is further necessary to explain that, in the method, the front-end module receives and displays the digital human description text, structured data of the drawn chart, and the digital human video; The method also builds a GB28181 platform, accesses audio and video to synchronize digital human video, and supports multi-protocol video distribution.

[0014] According to another embodiment of the present application, an interactive device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the real-time digital human interactive method driven by a large mine model are implemented.

[0015] It can be seen from the above technical solutions that the present invention has the following advantages: The real-time digital human interaction method driven by a large model of a mine provided by the present invention realizes interaction in various forms such as text, voice, video and charts by building a front-end module, thereby improving the intuitiveness and interactivity of information display. The problem of lack of intuitiveness and interactivity in existing mines is solved. The present invention realizes real-time processing and analysis of mine environmental data by configuring a large model module of a mine, and utilizes a large model application library and a data processing unit, thereby improving the accuracy and efficiency of data processing. In addition, by building a voice recognition module and a text-to-speech module, real-time recognition and conversion of voice information are realized, providing a more convenient way for remote monitoring and interaction of mines. The drawback that the prior art can only provide simple command input and cannot realize real-time voice interaction is solved. By building a digital human generation module, utilizing a digital human training unit and a digital human image generation model, a personalized digital human image can be generated according to user needs, thereby improving the fun and practicality of digital human interaction. By configuring an audio and video synthesis and streaming module, real-time synthesis and streaming of video information and voice information are realized, the synchronization of audio and video is ensured, and the fluency and user experience of remote interaction are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0017] Figure 1 A flow chart of the real-time interaction method of digital humans driven by a large mine model; Figure 2 A flow chart of an embodiment of a digital human real-time interaction method driven by a large mine model; Figure 3 Schematic diagram of interactive equipment. DETAILED DESCRIPTION

[0018] The real-time digital human interaction method driven by a large model of a mine provided in this application is mainly to solve the problem that existing digital humans in the mining field can only output voice according to a predetermined mode, have a single function, and cannot perform effective automated learning to achieve real-time interaction.

[0019] The method of the present application obtains user voice or text input, performs demand analysis based on the user input and calls the corresponding large model application, and converts the results of each application into the corresponding data type and digital human speech text through the data processing function.

[0020] Among them, this application can convert the digital human speech text into audio information and drive the generation of digital human video frames. The video frames and audio frames are synthesized synchronously, pushed through the RTMP protocol and connected to the GB28181 platform for multi-video protocol distribution. At the same time, the streamlit front-end output module displays unstructured data and the digital human's analysis and summary of it. It can be seen that this application can call different large-scale mining model applications according to user needs, drive the real-time interaction between digital humans and users in multiple dimensions such as voice, text and charts, and can easily obtain and intuitively display the required information, helping mining industry users to make decisions quickly.

[0021] The specific steps of the real-time digital human interaction method driven by a large model of a mine will be described in detail below. For the purpose of illustration rather than limitation, specific details such as specific system structures and technologies are proposed to facilitate a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details.

[0022] The phrases such as "one embodiment" or "some embodiments" described in the present application mean that the specific features, structures or characteristics described in the embodiment are included in one or more embodiments of the present application. Therefore, the phrases such as "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments" etc. that appear in different places in the present application do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] See also Figure 1 The figure is a flow chart of a real-time digital human interaction method driven by a large model of a mine in a specific embodiment, the method comprising: S101: Build a front-end module, which includes a text input unit, a voice input unit, a digital human video display unit, and a chart and text display unit; the voice input unit uploads the acquired voice information to the server.

[0025] In some embodiments, the text input unit is used to allow the user to input text information through a keyboard, a touch screen, and related input devices, and the text input unit is used to interact with a digital human.

[0026] Optionally, a front-end module to front-end module SFM is built based on streamlit.

[0027] Specifically, you can use HTML form elements to implement text input functions.

[0028] The voice input unit allows the user to input voice information through a microphone or related equipment, and upload the input voice information to the server for voice recognition.

[0029] Optionally, a third-party speech recognition library can also be used to implement the speech input function.

[0030] The digital human video display unit is used to display the digital human's video output, including the digital human's movements, expressions and voice. HTML5 can be used here to implement the video display function.

[0031] The chart and text display unit is used to display charts and text information related to the mine, providing additional data support and visualization effects. Here you can use chart libraries such as D3.js, or customize charts and text displays.

[0032] S102: Build a speech recognition module, provide a speech recognition process through the speech recognition module, and after receiving a speech recognition call request, obtain the audio file at the specified location of the current server for recognition and output text.

[0033] In some embodiments, an open source speech recognition toolkit may be used to build a speech recognition module.

[0034] Optionally, you can deploy a local ASR model or call the cloud to implement the speech recognition function.

[0035] In this embodiment, the voice recognition process provided is based on the voice recognition module providing an API interface for the front-end module to call. When the front-end module sends a voice recognition request, the voice recognition module should start the recognition process and return the recognition result.

[0036] After receiving the speech recognition call request, the speech recognition module should obtain the audio file from the specified location of the server, use the speech recognition algorithm to recognize the audio file, and output the recognized text information.

[0037] The speech recognition module involved in this embodiment can realize the conversion from speech to text, improve the convenience and naturalness of user interaction, and realize the speech interaction function for the digital human.

[0038] S103: Configuring a large mine model module, which includes a demand analysis unit, a large model application library, and a data processing unit.

[0039] In some embodiments, the demand parsing unit is used to parse the user's text or voice-to-text and extract the demand. The demand parsing unit can obtain the user's intention and provide guidance for subsequent interactions.

[0040] The large model application library contains various large models related to mines, which may involve visual large models, industry large models, etc., which are used to process and analyze mine data.

[0041] The large model application library can provide professional knowledge and skills related to mines and support the intelligent interaction of digital humans.

[0042] The data processing unit is used for tasks such as data preprocessing, feature extraction, and model training. The data processing unit can ensure the quality and accuracy of input data and improve the performance and effect of large models.

[0043] It can be seen that this embodiment realizes the intelligent application of mine-related knowledge and skills, and improves the interactive ability and intelligence level of digital humans.

[0044] S104: configuring a text-to-speech module, where the text-to-speech module includes a training set processing unit, a training function unit, a speech service unit, and a text-to-speech model.

[0045] In some embodiments, the training set processing unit is used to prepare and preprocess the training data, which can ensure the quality and diversity of the training data.

[0046] The training function unit is used to train the text-to-speech model, and can train a high-quality text-to-speech model.

[0047] The voice service unit can provide speech synthesis services, convert text into speech, provide voice output support for digital humans, and realize voice interaction functions.

[0048] The text-to-speech model can use the trained model for speech synthesis, converting text content into natural and fluent speech.

[0049] The text-to-speech module of this embodiment realizes the conversion from text to speech, improves the interactive ability of digital human, provides high-quality speech output, and enhances user experience.

[0050] S105: Building a digital human generation module, the digital human generation module includes a training set processing unit, a digital human training function unit, a digital human generation unit and a digital human image generation model.

[0051] In some embodiments, the training set processing unit is used to prepare and pre-process the digital human training data, which can ensure the quality and diversity of the digital human training data.

[0052] Digital human training functional unit: used to train digital human models, including movements, expressions, voices, etc., to train digital human models with realistic movements, expressions and voices.

[0053] Digital human generation unit: Generates digital humans using trained models and can generate customized digital humans based on user needs.

[0054] Digital human image generation model: Generates digital humans with different images according to user needs. It can provide a variety of digital human image options to meet the needs of different users.

[0055] The digital human generation module constructed in this embodiment realizes customized generation and real-time interaction of digital humans, and provides a variety of digital human image options to meet the mine's requirements for the use of digital humans.

[0056] S106: Configuring an audio and video synthesis streaming module, the audio and video synthesis streaming module synthesizes the video information and the voice information to achieve audio and video synchronization and configures the synthesized information as streaming information.

[0057] In some embodiments, video information and voice information may be synthesized using audio and video synthesis technology.

[0058] Optionally, configure the push streaming protocol to push the synthesized audio and video information to the specified address. This can achieve synchronous synthesis and push streaming of audio and video information to meet the needs of mine intelligence and improve the real-time and interactivity of digital human interaction in mines.

[0059] It can be seen that the real-time digital human interaction method driven by the large model of the mine provided in this embodiment solves the problem of single large model application interaction dimension and single digital human application function in the mining industry by building a front-end module, a speech recognition module, a large mine model module, a text-to-speech module, a digital human generation module, and an audio and video synthesis streaming module. The real-time interaction function of the digital human is realized, providing users in the mining industry with rich large model application capabilities, and providing decision support for users in combination with mining data.

[0060] On the basis of the above embodiments, in order to further improve the reliability and richness of the digital human real-time interaction method based on the mine model driven by the above embodiments, the following is an implementable method. In the following embodiments, the digital human real-time interaction method based on the mine model driven by the above embodiments can be combined, and different mine model applications can be called according to user needs to drive the real-time interaction between the digital human and the user in multiple dimensions such as voice, text and charts. Figure 2 As shown, a more specific implementation method will be given below.

[0061] S211: Build the front-end module SFM based on streamlit.

[0062] In this embodiment, the front-end module SFM includes functions such as text input, voice input, audio, digital human video display, and chart and text display.

[0063] Optionally, the voice input function is implemented through streamlit_webrtc technology, and the voice input is uploaded to the specified location of the server after completion.

[0064] S212: Building a voice recognition module VRM.

[0065] This embodiment provides speech recognition services by building a speech recognition platform. After receiving a call request, the audio file at the specified location of the current server is obtained for recognition and the text is output.

[0066] Optionally, the speech recognition module supports access configuration of ASR speech recognition and Whisper speech recognition.

[0067] S213: Build the mine large model module MLLMM.

[0068] The mine model module MLLMM in this embodiment includes a demand analysis unit RAF and a large model application library LLMAB. n And data processing unit DDF. Among them, the large model application is LLMAB k .

[0069] In this embodiment, the requirement analysis unit RAF is constructed based on the Qwen2 model, prompt words and identifier analysis components, and is used to analyze and process the text of step one or step two.

[0070] The requirement parsing unit RAF outputs the large model application index k through the semantic understanding (su), keywords (key) and identifiers (flag) of the text, and selects LLMAB n LLMAB is the most suitable for solving user needs k .

[0071] The specific formula is as follows LLMAB k =LLMAB RAF (text) In some specific embodiments, the large model application library LLMAB n Support application access. This embodiment may include applications such as intelligent question and answer, knowledge question and answer, and text quick reading. Different applications have different inputs and outputs. The specific formula is as follows: data = DDF(LLMAB k (text),k) Among them, data contains the digital human description text (datatext ) and structured data for drawing charts (data json ).

[0072] S214: Build a text-to-speech module TTSM, which includes a training set processing function (TTSM datasets ), training function (TTSM train ), service function (TTSM serve ) and text-to-speech models tts ).

[0073] The text-to-speech module TTSM of this embodiment is based on gpt-sovits and supports visual zero-code training and reasoning.

[0074] Among them, the text-to-speech module TTSM also supports text-to-speech services such as edgets, gpt-sovits, and cosyvoice.

[0075] In some embodiments, according to the needs of using digital humans in the mine, the required parameters such as voice quality, speaking speed, volume, and intonation can be set. Collect and organize the text and corresponding voice data used to train the text-to-speech model. When performing text-to-speech conversion, the collected text data can be annotated and normalized. Extract key information in the text, such as semantics, intonation, speaking speed, etc. Start the text-to-speech module TTSM and input the preprocessed data into the selected model for training. During the training process, continuously adjust the parameters and structure of the model to optimize the conversion effect. Evaluate the trained model, including tests on voice quality, speaking speed, volume, intonation, etc. Based on the evaluation results, fine-tune the model to improve its performance and stability.

[0076] The text-to-speech module TTSM in this embodiment can convert the generated voice data into MP3 or WAV format, or can send the voice data to the client in a streaming manner or perform other processing.

[0077] In some embodiments, the text-to-speech module TTSM supports voice cloning of characters by providing a 5-minute audio person ) can achieve rapid cloning with a similarity of up to 95%.

[0078] Specifically, first of all, audio person Sentence segmentation and automatic annotation are performed, and then the automatically annotated sentence segments are annotated and corrected. The automatic annotation uses a speech-to-text model (model asr ). Finally, after training, a text-to-speech model with personalized character voice can be obtained. The formula is as follows: audioi ,text i =TTSM datasets (audio person , model asr ), i=[1…n] Here's the audio i ,text i For audio clips and corresponding text clips, the text clips require us to make corrective annotations.

[0079] In some embodiments, based on the audio i ,text i You can use the training function of TTSM to generate the required text-to-speech model tts , the formula is as follows: model tts =TTSM train (audio i ,text i ), i=[1…n] As an example, TTSM can text Convert to audio speaking ), which is used to drive the generation of the digital human's mouth shape features and serve as the digital human's voice.

[0080] The formula is as follows: audio speaking =TTSM serve (data text , model tts ,data sample ) Among them, data sample The parameter is audio i ,text i An audio and text clip of TTSM serve Generate reference samples of cloned sounds.

[0081] S215: Configure the digital human generation module DHGM.

[0082] The digital human generation module DHGM includes a training set processing unit (DHGM datasets ), Digital Human Training Unit (DHGM train ), Digital Human Generation Unit (DHGM generate ) and digital human image generation model (model dh ).

[0083] The digital human generation module DHGM of this embodiment is based on the digital human generation method of er-nerf, and supports the training data processing of digital humans, the customization of digital human images, etc.

[0084] The digital human generation module DHGM of this embodiment also supports the generation methods of digital human such as musetalk, wav2lip, etc. The digital human generation module DHGM supports the image cloning of characters.

[0085] It should be noted that the training set processing unit can collect, organize and pre-process the data set for digital human training. The digital human training unit can use the data provided by the training set processing unit to train the digital human so that the digital human can simulate the behavior, expression and voice of real people to meet the use environment of the mine. The PyTorch deep learning framework can be used here to improve the training efficiency. The digital human generation unit can generate a digital human with the image and behavior of a mine personnel according to user needs.

[0086] The digital human image generation model can configure the customization and generation of digital human images. Digital humans with similar images can be generated based on the mining scenes and mining environments input by mining personnel. The digital human image generation model supports three-dimensional modeling technology. It can be achieved through computer vision and graphics algorithms.

[0087] For example, by providing a 5-minute video of a speech person ) can be used as a training sample for digital humans. person The speech should be clearly pronounced and punctuated, with a clear distinction between the background and the character's clothing. The video should be about 5 minutes long, with a size of 512×512 and a frame rate of 25fps.

[0088] Specifically, first of all, the video person For data processing, you need to use OpenFace tools to process the video person Perform feature extraction and obtain blink action data (data au45 ), and then use the process tool to obtain the audio file of the video (data aud )、audio features(data aud_eo )、video frame splitting (data images ), segmentation portrait (data head ), extract background (data bg ), split body (data body ), get face (data landmarks ), get face tracking data (data track )wait.

[0089] The specific formula is as follows: data types =DHGM datasets (OpenFace|process,video person ) In this embodiment, types include data such as au45, aud, aud_eo, images, head, bg, body, landmarks, and tarck.

[0090] It should be noted that au45 refers to the facial action data of the digital human. For example, the action number can be specified based on the number "45". The facial muscle movement of the digital human is realized. aud is the abbreviation of "audio". aud_eo is an abbreviation based on "audio" and "eye-opening" or "eye-tracking". images refers to image data. head refers to the position, direction, shape and other information of the digital human's head. b refers to the background content in the image or video. body refers to the position, posture, shape and other information of the digital human's body. landmarks refer to the feature points of the digital human or the key point shape features automatically detected in the image or video. tarck represents the tracking instruction for the digital human.

[0091] In some embodiments, based on data types To train a personalized digital human, specifically, it is necessary to train the digital human's head, fine-tune the mouth movements, body and other parts.

[0092] The formula is as follows: model head =DHGM train (data types ) model head-lip =DHGM train (data types , model head ,fineturn) model dh =DHGM train (data types , model head-lip ,body) Among them, model head and model head-lip is the intermediate model for training, model dh It is the final trained digital human model.

[0093] In some embodiments, based on the digital human speech audio in step S214, speakingThe formula to drive the digital human to generate the corresponding digital speech mouth shape and head movement is as follows: RP,video speaking =DHGM generate (model dh ,audio speaking ) It should be noted that the video here speaking It is the video frame generated by DHGM only, and RP is the mapping relationship between the audio frame and the video frame.

[0094] S216: Configure the audio and video synthesis streaming module AVSSM.

[0095] Audio and video synthesis push streaming module AVSSM is used for video speaking and audio speaking Synthesis, to achieve audio and video synchronization function (AVSSM synthesis ), and realize the rtmp real-time streaming function (AVSSM rtmp ).

[0096] Specifically, video speaking By audio speaking The driver generates the corresponding relationship RP between the video frame and the audio frame in step 5 to realize the audio-visual synchronization synthesis of the digital human video (video dh ).

[0097] The specific formula is as follows: video dh =AVSSM synthesis (video speaking , audio speaking ,RP) In some embodiments, by constructing the rtmp_streaming method, video dh Real-time streaming.

[0098] S217: Based on step S211, the front-end module SFM receives the digital human description text (data text ), structured data for drawing charts (data json ) and the digital human video of step S216 (video dh ) for display purposes.

[0099] Specifically, the front-end module SFM is based on data json Draw various types of charts such as line charts, bar charts, etc. chart ), according to the video dh Perform real-time audio and video decoding.

[0100] In this embodiment, data text 、video dh and data chart Through the display of the front-end module SFM and the generated digital human video (video dh ) Combined with chart data chart ) Analyze and answer user needs in the mining industry.

[0101] In this embodiment, the GB28181 platform can be built on the basis of step S216 to access the video dh , and supports multi-protocol video distribution and multi-system access.

[0102] S218: Based on step S213, configure an intelligent question-answering program. If the user input fails to provide accurate query conditions, a counter-question strategy can be supported until the user inputs additional information that no longer causes ambiguity.

[0103] Specifically, the intelligent question-answering program analyzes user needs based on the user input text, generates SQL and queries the database by calling the langchain's SQL generation tool chain. During the generation process, if there is ambiguity between the user's input and the database field, it supports a feedback mechanism, requiring the user to provide additional information until the ambiguity is eliminated.

[0104] S219: Based on step S213, applications developed based on large models generally have occasional hallucination problems.

[0105] This embodiment sets up a self-correction mechanism, which automatically monitors the places where hallucinations may occur in the large model task flow by configuring the monitoring large model tool, records the hallucinations in the task nodes, and re-reasons and corrects them.

[0106] S220: On the basis of step S217, step S218 and step S219, the interaction results between the user and the digital human are saved locally for analysis and self-learning by the mine large model application module.

[0107] The steps involved in this embodiment can be implemented locally on the server, without relying on the Internet, and meet the data confidentiality requirements of the mining industry. The processing steps of the present invention are based on streaming processing, and the digital human driven by the large model responds quickly, and the user experience is good.

[0108] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0109] Based on the above-mentioned digital human real-time interaction method driven by a large mine model, this embodiment also provides an interaction device. Figure 3 A schematic diagram of the hardware structure of an interactive device for implementing various embodiments of the present invention.

[0110] The interactive device 500 includes, but is not limited to, components such as a processor 501, a network module 502, an audio output unit 503, an input unit 504, a display unit 506, a user input unit 507, an interface unit 508, and a memory 509. Those skilled in the art will appreciate that the interactive device structure involved in the embodiments of the present invention does not constitute a limitation on the interactive device, and the interactive device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0111] In the embodiments of the present invention, the interactive devices include but are not limited to digital robots, simulated robots, bionic robots, laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of the present application described and / or required herein.

[0112] In the embodiment of the present application, the processor 501 can be implemented by using at least one of an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a processor, a controller, a microcontroller, a microprocessor, and an electronic unit designed to perform the functions described herein. In some cases, such an implementation can be implemented in a controller. For software implementation, implementations such as processes or functions can be implemented with separate software modules that allow execution of at least one function or operation. The software code can be implemented by a software application (or program) written in any appropriate programming language, and the software code can be stored in a memory and executed by a controller.

[0113] The processor 501 provides the user with wireless Internet access through the network module 502, thereby realizing the transmission and communication of data information in this application.

[0114] The audio output unit 503 includes a speaker, a buzzer, a receiver, and the like.

[0115] The input unit 504 is used to receive audio or video signals. The input unit 504 may include a graphics processing unit (GPU) and a microphone, and the graphics processor processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The processed image frame can be displayed on the display unit 506. The image frame processed by the graphics processor can be stored in the memory 509 (or other storage medium) or sent through the network module 502. The microphone can receive sound and can process such sound into audio data.

[0116] The display unit 506 may include a display panel, and the display panel may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0117] The user input unit 507 may include, but is not limited to, a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0118] The memory 509 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0119] In addition, the interactive device 500 includes some functional modules not shown, which will not be described in detail here.

[0120] The interactive device 500 of this embodiment has multiple large model applications built in. According to the needs of the user, the system can analyze and call different large model applications by itself, and has the ability to fine-tune and access other applications. The interactive device 500 supports multi-dimensional interaction with users in voice, text, and charts. Users can communicate with the system through natural language to obtain more intuitive and convenient information feedback. The present invention can help users obtain the required business information by interacting with digital people in real time, thereby lowering the usage threshold for users in the mining industry. This application deeply analyzes historical data and real-time information through large model applications, thereby assisting managers in the mining industry in making quick decisions.

[0121] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A real-time digital human interaction method driven by a large mine model, characterized in that: Methods include: S101: Building a front-end module, which includes a text input unit, a voice input unit, a digital human video display unit, and a chart and text display unit; the voice input unit uploads the acquired voice information to the server; S102: Building a speech recognition module, providing a speech recognition process through the speech recognition module, and after receiving a speech recognition call request, obtaining an audio file at a specified location of the current server for recognition and outputting text; S103: configuring a mine large model module, the mine large model module includes a demand analysis unit, a large model application library and a data processing unit; S104: configuring a text-to-speech module, where the text-to-speech module includes a training set processing unit, a training function unit, a speech service unit, and a text-to-speech model; S105: Building a digital human generation module, the digital human generation module includes a training set processing unit, a digital human training unit, a digital human generation unit and a digital human image generation model; S106: Configuring an audio and video synthesis streaming module, the audio and video synthesis streaming module synthesizes the video information and the voice information to achieve audio and video synchronization and configures the synthesized information as streaming information.

2. The method for real-time digital human interaction based on a large model driven by a mine according to claim 1 is characterized in that: In the method, a relational automation module is constructed to analyze and process the voice information in the method, as well as semantic understanding, keyword parsing and identifier parsing of the voice information, and output an application index k; In the large model application library LLMAB n Matching user needs in LLMAB k ; The matching method is: <h2 style=";text-align:left;direction:ltr">LLMAB<h2 style=";text-align:left;direction:ltr"> k <h2 style=";text-align:left;direction:ltr"> = LLMAB<h2 style=";text-align:left;direction:ltr"> RAF (text) The matching relationship data based on information input and information output is: data=DDF(LLMAB k (text),k) The matching relationship data data includes structured data of digital human description text and drawn charts.

3. The method for real-time digital human interaction based on a large model driven by a mine according to claim 1 is characterized in that: The method also includes: person Perform sentence segmentation and automatic annotation; Using the speech-to-text model asr Mark and correct self-broken sentences; Get personalized character voice based on the following text-to-speech model: audio i ,text i =TTSM datasets (audio person , model asr ), i=[1…n]; Among them, audio i ,text i is an audio segment and a corresponding text segment.

4. The method for real-time digital human interaction based on a large model driven by a mine according to claim 3 is characterized in that: In the method, based on audio i ,text i Generate text-to-speech model tts , the formula is as follows: model tts =TTSM train (audio i ,text i ), i=[1…n]。 5. The method for real-time digital human interaction based on a large model driven by a mine according to claim 1 is characterized in that: Step S104 also includes: converting the text information into audio speaking , drives the generation of the digital human's mouth shape features and serves as the digital human's voice. The formula is as follows: audio speaking =TTSM serve (data text , model tts ,data sample ) Among them, data sample The parameter is audio i ,text i An audio and text clip of TTSM serve Generate reference samples of cloned sounds.

6. The method for real-time digital human interaction based on a large model driven by a mine according to claim 1 is characterized in that: In step S105, the digital human generation module supports the image cloning of the character by providing a speech video of a preset length. person As training samples for digital humans; Among them, the speech video person Perform data processing, extract features, and obtain blink action data au45 ; Get the audio file data of the video aud , audio feature data aud_eo , video frame data images , segment portrait data head , extract background data bg , segment body data body , Get face data landmarks , Get face tracking data track ; The specific formula is as follows: data types =DHGM datasets (OpenFace|process,video person )。 7. The method for real-time digital human interaction based on a large model driven by a mine according to claim 6 is characterized in that: Step S105 is also based on data types Training of digital humans, the training methods include the following: model head =DHGM train (data types ) model head-lip =DHGM train (data types , model head ,fineturn) model dh =DHGM train (data types , model head-lip ,body) Among them, model head and model head-lip is the intermediate model for training, model dh It is the final trained digital human model.

8. The real-time digital human interaction method based on a large model driven by a mine according to claim 1 or 2, characterized in that: In the method, the digital human speech audio converted by the text-to-speech module speaking The formula to drive the digital human to generate the corresponding digital speech mouth shape and head movement is as follows: RP,video speaking =DHGM generate (model dh ,audio speaking ) The audio and video synthesis push module converts the video frame video speaking and speech audio speaking Synthesis, to achieve audio and video synchronization AVSSM synthesis , and realize the real-time streaming function, and realize the synchronous synthesis of audio and video digital human video dh , the formula is as follows: video dh =AVSSM synthesis (video speaking , audio speaking ,RP)。 9. The method for real-time digital human interaction based on a large model driven by a mine according to claim 1 is characterized in that: In the method, the front-end module receives and displays the digital human description text, structured data of the drawn chart, and the digital human video; The method also builds a GB28181 platform, accesses audio and video to synchronize digital human video, and supports multi-protocol video distribution.

10. An interactive device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the real-time digital human interaction method driven by a large model of a mine are implemented as described in any one of claims 1 to 9.

Images