Digital twin voice real-time interaction method based on retrieval enhancement

Abstract

The invention is applicable to the technical field of voice interaction, and particularly relates to a retrieval enhancement-based digital twin voice real-time interaction method, which comprises the following steps of: during interaction, performing voice input through a voice input module to obtain voice input data, understanding the voice input data through a semantic understanding module, extracting a user instruction, and outputting the user instruction to a server; the RAG retrieval enhancement generation module is used for processing a user instruction, obtaining a corresponding knowledge fragment and generating reply data, a natural language instruction is converted into an instruction sequence of a digital twin system through a standardized interface, and the feedback module is used for constructing a system interface and carrying out image display on a system state and feedback information. According to the method, the interaction experience is remarkably improved, the user can directly control the digital twin system through voice, additional manual operation is not needed, and the operation convenience and efficiency are improved.

Description

Technical Field

[0001] The present invention belongs to the field of voice interaction technology, and in particular relates to a digital twin voice real-time interaction method based on retrieval enhancement. Background Art

[0002] With the rapid development of artificial intelligence technology, digital twin technology has become a bridge connecting the physical world and the virtual world, and is widely used in manufacturing, urban planning, smart buildings and other fields. The core of digital twin technology is to build a virtual model corresponding to the real-world entity, and realize accurate simulation and prediction of the entity state through real-time data interaction.

[0003] At present, in terms of the interaction of digital twin systems, traditional control methods mainly rely on mouse and keyboard input or touch screen operation. Although these interaction methods are intuitive and easy to use, they have limitations in some application scenarios. For example, in a noisy working environment, operators may not be able to conveniently use hand operations to interact; in addition, in complex system management scenarios, traditional human-computer interaction interfaces often cannot meet the needs of rapid response. On the other hand, with the development of speech recognition technology, real-time speech-to-text technology has achieved a high accuracy rate and can realize natural language communication between people and machines. However, existing speech recognition technology is mostly used for simple command execution or information query, and is rarely used in the control of complex systems, especially in combination with digital twin systems. Application cases are scarce.

[0004] Retrieval-enhanced generation technology is a natural language processing technology that combines retrieval and generation. It can retrieve information from the corresponding action function knowledge base before the model generates text to improve the accuracy of user instruction execution. Although RAG technology has achieved good application results in fields such as intelligent customer service and automatic document writing, its application in digital twin systems is still in the exploratory stage.

[0005] In summary, the existing technology has the following shortcomings in the interactive control of digital twin systems: lack of efficient and natural interactive means, especially inconvenient operation in complex environments; the existing speech-to-text technology has not been fully utilized in the control of digital twin systems; the application of RAG technology in digital twin systems is not yet mature, and there is a lack of effective integration solutions. Summary of the invention

[0006] The purpose of the present invention is to provide a real-time digital twin voice interaction method based on retrieval enhancement, aiming to solve the problem that the existing speech-to-text technology has not been fully utilized in the control of the digital twin system.

[0007] The present invention is implemented as follows: a real-time interactive method of digital twin voice based on retrieval enhancement, the method comprising: a voice input module, a semantic understanding module, a RAG retrieval enhancement generation module, a digital twin system control interface and a feedback module. During interaction, voice input is performed through the voice input module to obtain voice input data, the voice input data is understood through the semantic understanding module, user instructions are extracted, the user instructions are processed through the RAG retrieval enhancement generation module, the corresponding knowledge fragments are obtained, and reply data is generated, a standardized interface is constructed through the digital twin system control interface, and natural language instructions are converted into instruction sequences of the digital twin system through the standardized interface. The feedback module is used to construct a system interface and to display the system status and feedback information in a graphical manner.

[0008] Preferably, the voice input module uses a microphone as a voice input device, and converts the sound recorded by the voice input device into text through a voice conversion engine.

[0009] Preferably, when audio is recorded via a voice input device, ambient noise is processed via a noise suppression algorithm.

[0010] Preferably, the semantic understanding module adopts a large language model, through which the text converted by the voice input device is semantically understood, and the target object and operation type are extracted therefrom.

[0011] Preferably, the RAG retrieval enhancement generation module includes a vector knowledge base, wherein the vector database contains all action functions, parameters, descriptions and rules, and the vector database is used to store parameters and rules in various scenarios.

[0012] Preferably, the RAG retrieval enhancement generation module uses a multi-way query algorithm and a re-ranking algorithm to locate knowledge fragments related to user instructions.

[0013] Preferably, the RAG retrieval enhancement generation module uses the GPT series large language model or the Qwen series large language model to generate a JSON string based on the retrieved information.

[0014] Preferably, the feedback module also includes a voice feedback mechanism to provide information to the user through sound.

[0015] Preferably, the feedback module also includes an evaluation mechanism, through which users evaluate the system.

[0016] Preferably, the feedback module also includes a correction guidance mechanism, which is used to guide the user to correct errors or abnormal situations.

[0017] The digital twin voice real-time interaction method based on retrieval enhancement provided by the present invention can achieve the following effects:

[0018] The interactive experience is significantly improved. Users can directly control the digital twin system through voice without additional manual operation, which improves the convenience and efficiency of operation.

[0019] Instruction assistance function: RAG technology can provide decision support based on the constructed digital twin vector knowledge base, helping the system to make more reasonable judgments and perform the best actions in complex user instruction situations;

[0020] User feedback adaptability: the system can score different users’ actions and automatically adjust behavior strategies to improve overall applicability and flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flowchart of a retrieval-enhanced digital twin voice real-time interaction method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0023] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first xx script may be referred to as a second xx script, and similarly, a second xx script may be referred to as a first xx script.

[0024] Retrieval-enhanced generation technology is a natural language processing technology that combines retrieval and generation. It can retrieve information from the corresponding action function knowledge base before the model generates text to improve the accuracy of user instruction execution. Although RAG technology has achieved good application results in fields such as intelligent customer service and automatic document writing, its application in digital twin systems is still in the exploratory stage.

[0025] In summary, the existing technologies have the following deficiencies in the interactive control of digital twin systems: lack of efficient and natural interactive means, especially inconvenient operation in complex environments; existing speech-to-text technology has not been fully utilized in the control of digital twin systems; the application of RAG technology in digital twin systems is not yet mature and lacks effective integration solutions. Therefore, developing a digital twin system screen control method that can effectively integrate real-time speech-to-text technology and RAG retrieval enhancement generation technology has important research value and broad application prospects.

[0026] like Figure 1 As shown, it is a flowchart of the real-time interactive method of digital twin voice based on retrieval enhancement provided by an embodiment of the present invention. The method includes: a voice input module, a semantic understanding module, a RAG retrieval enhancement generation module, a digital twin system control interface and a feedback module. When interacting, voice input is performed through the voice input module to obtain voice input data, the voice input data is understood by the semantic understanding module, and user instructions are extracted. The user instructions are processed by the RAG retrieval enhancement generation module to obtain corresponding knowledge fragments and generate reply data. A standardized interface is constructed through the digital twin system control interface, and natural language instructions are converted into instruction sequences of the digital twin system through the standardized interface. The feedback module is used to construct a system interface and to display the system status and feedback information in a graphical manner.

[0027] In this embodiment, the voice input module and the semantic understanding module are responsible for receiving the user's voice commands and converting them into text form. Advanced speech recognition technology is used to ensure a high-accuracy voice-to-text process. The user's original text obtained by voice-to-text conversion is analyzed to extract the user's intentions and key information. The voice input module and the semantic understanding module can use natural language processing technology to understand complex commands and perform action segmentation on texts containing multiple commands.

[0028] For the voice input module, in terms of hardware equipment, a high-quality microphone is selected as the voice input device to ensure that the captured voice signal is clear and accurate; in terms of software algorithms, mature speech recognition technology is used for speech-to-text processing to ensure high accuracy and low latency; in terms of noise suppression, noise suppression algorithms (such as spectral subtraction or deep learning models) are applied to reduce the impact of environmental noise on speech recognition; in terms of wake-up word recognition, a specific wake-up word (such as "Hello, Xiaozhu") is set to activate the voice control system.

[0029] In the semantic understanding module, in terms of text processing, the large language model prompt engineering is used to understand the actions and parameters in the user's instructions and perform action segmentation; in terms of dialogue management, the interaction history between the user and the system is tracked to better understand the context. In terms of intent recognition, the fine-tuned model can distinguish different types of commands, such as queries, control operations, etc.; in terms of entity extraction, the fine-tuned model extracts key information from user instructions, such as target objects, operation types, etc., and classifies entity information.

[0030] Based on the user's instructions, the RAG retrieval enhancement generation module retrieves relevant information from the pre-built vector knowledge base, and generates the corresponding action function and parameter JSON string according to the search results, which can ensure the accuracy of the information provided by the system. The RAG retrieval enhancement generation module builds a vector database containing all the action functions, parameters, descriptions and rules of the digital twin, which is used to store parameters and rules in various situations; in terms of retrieval algorithms, efficient retrieval algorithms (such as multi-way query, reordering, etc.) are used to quickly locate knowledge fragments related to user instructions; in terms of generation models, large language models (such as GPT series, Qwen series, etc.) are used to generate corresponding JSON strings based on the retrieved information; in terms of context fusion, the user's current context information is combined with the search results to generate answers that are more in line with actual needs. The system first matches the relevant functions such as screen adjustment and weather adjustment that best meet the user's instructions from the digital twin action function vector library, splices them with text prompts and passes them into the large language model to determine the parameters and instructions and generate a JSON string.

[0031] The digital twin system control interface defines a set of standard communication protocols (such as RESTful API) to ensure efficient interaction between systems. In terms of command mapping, a command mapping table is established to convert natural language commands into command sequences that the digital twin system can understand. In terms of security, a security authentication mechanism is implemented to protect the system from unauthorized access. The system sends JSON commands to the digital twin system to execute corresponding actions.

[0032] Feedback module, visual feedback, including user-friendly system interface, graphically displaying system status and feedback information; multimodal feedback, in addition to system interface feedback, can also provide users with more information through sound; user experience, allowing users to score the system to ensure the accuracy of operation; error handling, providing clear error prompts to guide users on how to correct errors or abnormal situations. The system displays the corresponding action on the interface, and will give feedback to the user through voice if the user action is contradictory or irrelevant to the current system.

[0033] It should be understood that, although each step in the flow chart of each embodiment of the present invention is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0034] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0035] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0036] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0037] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A digital twin voice real-time interaction method based on retrieval enhancement, characterized in that: The method includes: a voice input module, a semantic understanding module, a RAG retrieval enhancement generation module, a digital twin system control interface and a feedback module. When interacting, voice input is performed through the voice input module to obtain voice input data, the voice input data is understood by the semantic understanding module, and user instructions are extracted, the user instructions are processed by the RAG retrieval enhancement generation module, the corresponding knowledge fragments are obtained, and reply data is generated, a standardized interface is constructed through the digital twin system control interface, and natural language instructions are converted into instruction sequences of the digital twin system through the standardized interface. The feedback module is used to construct a system interface and to display the system status and feedback information in a graphical manner.

2. The method for real-time interaction of digital twin voice based on retrieval enhancement according to claim 1 is characterized in that: The voice input module uses a microphone as a voice input device, and converts the sound recorded by the voice input device into text through a voice conversion engine.

3. The method for real-time interaction of digital twin voice based on retrieval enhancement according to claim 2 is characterized in that: When recording audio through a voice input device, ambient noise is processed through a noise suppression algorithm.

4. The method for real-time interaction of digital twin voice based on retrieval enhancement according to claim 2 is characterized in that: The semantic understanding module uses a large language model to perform semantic understanding on the text converted by the voice input device and extract the target object and operation type from it.

5. The method for real-time digital twin voice interaction based on retrieval enhancement according to claim 1 is characterized in that: The RAG search enhancement generation module includes a vector knowledge base, wherein the vector database contains all action functions, parameters, descriptions and rules, and the vector database is used to store parameters and rules in various scenarios.

6. The method for real-time interaction of digital twin voice based on retrieval enhancement according to claim 1 is characterized in that: The RAG retrieval enhancement generation module uses a multi-way query algorithm and a re-ranking algorithm to locate knowledge fragments related to user instructions.

7. The method for real-time digital twin voice interaction based on retrieval enhancement according to claim 1 is characterized in that: The RAG retrieval enhancement generation module uses the GPT series large language model or the Qwen series large language model to generate a JSON string based on the retrieved information.

8. The method for real-time digital twin voice interaction based on retrieval enhancement according to claim 1 is characterized in that: The feedback module also includes a voice feedback mechanism to provide information to the user through sound.

9. The method for real-time digital twin voice interaction based on retrieval enhancement according to claim 1 is characterized in that: The feedback module also includes an evaluation mechanism, through which users evaluate the system.

10. The method for real-time digital twin voice interaction based on retrieval enhancement according to claim 1 is characterized in that: The feedback module also includes a correction guidance mechanism, which is used to guide users to correct errors or abnormal situations.

Images