Intelligent voice interaction robot and virtual simulation platform based on knowledge graph

Abstract

The invention relates to an intelligent voice interaction robot based on a knowledge graph and a virtual simulation platform, and aims to solve the problems that in the prior art, a voice interaction system is insufficient in semantic understanding depth, response content is disjointed from virtual scene feedback and the like. The voice input of a user is converted into a query text through a voice recognition technology, then deep semantic analysis is performed on the text by utilizing a knowledge graph, related entities and association relationships thereof are accurately positioned through character string and semantic matching, and a natural language response text with coherent context is dynamically generated according to retrieved knowledge. And in the virtual simulation platform, the system synchronously drives the mouth shape, the expression and the action of the virtual character and triggers the animation and the environment change of the related object in the scene so as to ensure that the voice content is highly consistent with the visual presentation. According to the method, voice processing, knowledge graph and virtual simulation technologies are deeply fused, so that the accuracy and naturalness of interaction are remarkably improved, and the robustness and personalized service level of the system are enhanced.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, and in particular to intelligent voice interaction robots and virtual simulation platforms based on knowledge graphs. Background Technology

[0002] In the current technological field, intelligent voice interaction systems have been widely used in virtual simulation environments to achieve natural human-computer dialogue. However, most existing systems rely on predefined dialogue flows and fixed response libraries, lacking the ability to deeply understand complex semantics. This results in insufficient accuracy in responding to open-ended user queries and difficulty in generating natural language responses that are coherent with the context. Furthermore, traditional methods typically treat speech recognition, semantic parsing, and scene rendering as independent modules, lacking overall coordination, leading to problems such as high interaction response latency and mismatch between virtual character behavior and voice content.

[0003] In existing technologies, speech recognition modules typically convert speech into text based on acoustic and language models, but often neglect the dynamic integration of dialogue history and domain knowledge, making recognition results prone to errors in specialized contexts. Simultaneously, semantic parsing often employs keyword matching or template filling methods, which struggle to effectively handle polysemous words and complex sentence structures, resulting in incomplete or weakly correlated information retrieved from knowledge graphs. In virtual simulation environments, existing solutions typically trigger animations through simple scripts, lacking fine-grained analysis of actions, states, and objects within the speech content. This leads to inconsistencies between virtual scene feedback and speech responses, reducing user immersion.

[0004] Furthermore, existing systems have significant limitations in error handling and personalized interaction. When speech recognition confidence is low or the knowledge graph query yields no results, the system typically returns a generic error message, lacking a targeted guidance mechanism. User identification relies heavily on independent authentication modules, failing to deeply integrate with the voice interaction process and limiting the generation of adaptive responses based on user history and preferences. Therefore, there is an urgent need for an integrated interaction solution that can deeply integrate speech recognition, knowledge graph query, and virtual scene rendering to improve the accuracy of semantic understanding, the coherence of responses, and the real-time and consistent nature of scene feedback. Summary of the Invention

[0005] To achieve the above objectives, this application provides the following technical solution: According to the first aspect of this invention, the present invention claims protection for a knowledge graph-based intelligent voice interaction method, applied to a robot or virtual simulation platform, the method comprising: S1, the analog voice signal emitted by the user is collected in real time through the audio input device, and the analog voice signal is converted into a digital voice signal; S2, the digital speech signal is input into the speech recognition engine. The speech recognition engine analyzes the acoustic features and language model of the speech signal, segments the continuous speech into phoneme sequences, and maps the phoneme sequences into word sequences based on a dictionary to generate query text. S3, the query text is segmented and tagged with parts of speech. After removing stop words, key terms are extracted. The key terms are matched with the tags of entities in the knowledge graph using string matching and semantic matching to obtain matching entities. All direct attributes and relationships of the matching entities, as well as adjacent entities connected by relationships, are retrieved from the knowledge graph. S4. Based on the obtained entities and relationships, determine the response topic, select the sentence structure corresponding to the relationship type, insert the entity name into the sentence template, adjust pronouns and conjunctions to ensure contextual coherence, and generate natural language response text. S5, the natural language response text is input into the speech synthesizer, the speech synthesizer parses the natural language response text into a phoneme sequence, adds prosodic markers to control pitch, duration and stress, and generates a digital speech waveform signal; S6. In the virtual simulation platform, the digital voice waveform signal is converted into an audio stream, which drives the virtual character's lip movements to synchronize with the voice, parses the keywords in the response text, triggers the animation script of the object in the virtual scene, and adjusts the lighting, camera angle and background elements to match the response content.

[0006] Further, step S3 includes: The key terms and knowledge graph entity labels in the query text are converted into feature vectors using a pre-trained semantic vector model, and the cosine similarity between the feature vectors is calculated as the semantic similarity. Set a similarity threshold, select entities with semantic similarity higher than the similarity threshold as candidate matching entities, and sort the candidate matching entities by similarity. Retrieve all direct relation triples of the candidate matching entity from the knowledge graph, including relation type and target entity, and further retrieve the direct attributes and relations of the target entity.

[0007] Further, step S4 includes: Analyze the acquired entities and relationships to identify primary and secondary entities, as well as the direction and type of relationships; Based on the relationship type, a matching template is retrieved from a predefined response template library, which is categorized by relationship type, and each template contains variable placeholders and fixed text; Insert the main entity name into the subject placeholder of the template, insert the object placeholder from the entity name into the template, and insert the relation description into the predicate part of the template; Perform part-of-speech and syntactic analysis on the filled text to ensure subject-verb agreement and correct tense, and adjust the use of articles and prepositions; The generated response text is compared with the relationships in the knowledge graph to confirm whether the response text accurately reflects the real relationships between entities. The validated text is formatted, punctuation marks and paragraph separators are added, and the final natural language response text is output.

[0008] Further, step S6 includes: Natural language processing techniques are used to parse the response text and identify action verbs, object nouns, and state descriptive words. The action verbs are mapped to specific animation sequences in the animation library of the virtual character, including gestures, facial expressions, and body movements; Based on the object name, locate the corresponding object in the virtual scene, and modify the object's color, size, or position based on the state description word; Generate scene control commands to specify the animation playback order, the timing and duration of object state changes; Control the virtual engine to render scene changes according to instructions, ensure that voice playback is synchronized with animation frames, and dynamically adjust the focal length and angle of the virtual camera to highlight key interactive areas; The current state of all elements in the virtual scene is detected in real time and compared with the expected state of the response text. If there is a deviation, the animation parameters are readjusted.

[0009] Furthermore, step S2 uses an end-to-end speech recognition model to directly map the speech signal into a text sequence without performing intermediate feature extraction, and also includes: The digital speech signal is input into a neural network model, which extracts global features through an encoder, generates character probability distributions frame by frame through a decoder, and outputs the most likely text sequence through a beam search algorithm.

[0010] Furthermore, step S5 employs waveform generation technology to convert the text sequence into a speech waveform and adds prosody control to adjust intonation and rhythm, specifically including: The text is input into the acoustic model to generate a Mel spectrogram. The spectrogram is then converted into a waveform signal using a vocoder, and pauses and accents are added based on the punctuation marks in the text.

[0011] Furthermore, it also includes dialogue management steps: Store the context information of the current interaction, including user query history and system responses, and retrieve the context in subsequent queries to optimize knowledge graph queries and response generation, specifically including: Maintain a context cache that records entities and relationships from recent interactions. When a new query is entered, the relevant entities are retrieved from the cache first.

[0012] Furthermore, the dialogue management steps include: The cache context uses a first-in-first-out queue to store the query text, response text, and related entities from the five most recent interactions; The context is retrieved, entities are extracted from the new query, and compared with entities in the cache. If there are overlapping entities, the cached relationships are used as additional information to input the knowledge graph query.

[0013] Furthermore, the method also includes an error handling step: when speech recognition or knowledge graph query fails, a default prompt speech is generated, and an error prompt animation is displayed in the virtual simulation platform, specifically including: When the speech recognition confidence level is below the threshold or there is no matching entity in the knowledge graph, an error handling process is triggered, a pre-recorded error message is played, and the virtual character displays a confused expression.

[0014] Furthermore, the method also includes a user authentication step: User identity is identified through voice biometrics, and personalized knowledge graph queries and virtual scene configurations are made based on the identity information, specifically including: Extract voiceprint features from the speech signal, compare them with the user's voiceprint in the database, and after successful authentication, load the user's preference settings and historical interaction records.

[0015] This application relates to a knowledge graph-based intelligent voice interaction robot and virtual simulation platform, aiming to address issues such as insufficient semantic understanding depth and disconnect between response content and virtual scene feedback in existing voice interaction systems. It converts user voice input into query text using speech recognition technology, then performs deep semantic analysis of the text using a knowledge graph. Through string and semantic matching, it accurately locates relevant entities and their relationships, dynamically generates context-coherent natural language response text based on the retrieved knowledge, and converts it into voice output. In the virtual simulation platform, the system synchronously drives the virtual character's lip movements, facial expressions, and actions, triggering animations of relevant objects and environmental changes in the scene, ensuring a high degree of consistency between voice content and visual presentation. This invention significantly improves the accuracy and naturalness of interaction, and enhances the system's robustness and personalized service level by deeply integrating voice processing, knowledge graphs, and virtual simulation technologies. Attached Figure Description

[0016] Figure 1 A flowchart illustrating the workflow of a knowledge graph-based intelligent voice interaction method claimed in this application embodiment; Figure 2A second flowchart illustrating the claimed intelligent voice interaction method based on knowledge graphs, as specified in this application embodiment; Figure 3 A third flowchart illustrating the claimed intelligent voice interaction method based on knowledge graphs, as described in this application embodiment; Figure 4 The fourth flowchart of a knowledge graph-based intelligent voice interaction method claimed in this application is shown in the embodiment of the present application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0018] The terms "first," "second," and "third" in this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. All directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative positional relationships and movements between components in a specific orientation (as shown in the figures). If the specific orientation changes, the directional indications also change accordingly. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0019] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0020] According to the first embodiment of this invention, the present invention claims protection for an intelligent voice interaction method based on knowledge graphs, applicable to robots or virtual simulation platforms, see reference. Figure 1The method includes: S1, the analog voice signal emitted by the user is collected in real time through the audio input device, and the analog voice signal is converted into a digital voice signal; S2, the digital speech signal is input into the speech recognition engine. The speech recognition engine analyzes the acoustic features and language model of the speech signal, segments the continuous speech into phoneme sequences, and maps the phoneme sequences into word sequences based on a dictionary to generate query text. S3, the query text is segmented and tagged with parts of speech. After removing stop words, key terms are extracted. The key terms are matched with the tags of entities in the knowledge graph using string matching and semantic matching to obtain matching entities. All direct attributes and relationships of the matching entities, as well as adjacent entities connected by relationships, are retrieved from the knowledge graph. S4. Based on the obtained entities and relationships, determine the response topic, select the sentence structure corresponding to the relationship type, insert the entity name into the sentence template, adjust pronouns and conjunctions to ensure contextual coherence, and generate natural language response text. S5, the natural language response text is input into the speech synthesizer, the speech synthesizer parses the natural language response text into a phoneme sequence, adds prosodic markers to control pitch, duration and stress, and generates a digital speech waveform signal; S6. In the virtual simulation platform, the digital voice waveform signal is converted into an audio stream, which drives the virtual character's lip movements to synchronize with the voice, parses the keywords in the response text, triggers the animation script of the object in the virtual scene, and adjusts the lighting, camera angle and background elements to match the response content.

[0021] In this embodiment, the user is situated in a virtual simulation environment that mimics a history museum exhibition hall. The user wishes to interact with a virtual tour guide robot via voice to obtain information about the exhibits. The system includes audio input devices (such as microphones), a voice processing module, a knowledge graph database, a virtual simulation engine, and audio output devices.

[0022] A user, interacting with the virtual simulation platform, utters a voice query: "Please introduce Leonardo da Vinci from the Renaissance period." The audio input device captures the user's voice signal in real time, converting the analog speech into a digital speech signal. Experimental testing used various environmental noise conditions (such as low noise and medium noise) to verify the stability of the voice acquisition. The results show that the system effectively filters out background interference, ensuring a clear voice signal. Furthermore, referring to Figure 2 Step S3 includes: The key terms and knowledge graph entity labels in the query text are converted into feature vectors using a pre-trained semantic vector model, and the cosine similarity between the feature vectors is calculated as the semantic similarity. Set a similarity threshold, select entities with semantic similarity higher than the similarity threshold as candidate matching entities, and sort the candidate matching entities by similarity. Retrieve all direct relation triples of the candidate matching entity from the knowledge graph, including relation type and target entity, and further retrieve the direct attributes and relations of the target entity.

[0023] In this embodiment, the text query "Please introduce Leonardo da Vinci during the Renaissance" is segmented and tagged with parts of speech, stop words (such as "please" and "introduce") are removed, and the key terms "Renaissance period" and "da Vinci" are extracted. The system calculates the semantic similarity between these terms and entity tags in the knowledge graph: using a pre-trained semantic vector model, terms and tags are converted into vectors, cosine similarity is calculated, and only entities with similarity higher than a preset threshold are selected (such as "Renaissance" and "Leonardo da Vinci"). Then, the direct attributes (such as da Vinci's birth date and occupation) and relationships (such as "belongs to the Renaissance period" and "created the Mona Lisa") of the matching entities are retrieved, and adjacent entities (such as "Mona Lisa") are obtained. In experimental testing, the knowledge graph contains tens of thousands of entities, the query response time is within an acceptable range, and the similarity threshold setting effectively reduces false matches.

[0024] Furthermore, referring to Figure 3 Step S4 includes: Analyze the acquired entities and relationships to identify primary and secondary entities, as well as the direction and type of relationships; Based on the relationship type, a matching template is retrieved from a predefined response template library, which is categorized by relationship type, and each template contains variable placeholders and fixed text; Insert the main entity name into the subject placeholder of the template, insert the object placeholder from the entity name into the template, and insert the relation description into the predicate part of the template; Perform part-of-speech and syntactic analysis on the filled text to ensure subject-verb agreement and correct tense, and adjust the use of articles and prepositions; The generated response text is compared with the relationships in the knowledge graph to confirm whether the response text accurately reflects the real relationships between entities. The validated text is formatted, punctuation marks and paragraph separators are added, and the final natural language response text is output.

[0025] In this embodiment, the system parses the acquired entities and relationships, identifying the core entity as "Leonardo da Vinci" and the relationship types as "belongs to" and "created". A corresponding template is selected from the response template library; for example, for the "belongs to" relationship, the template is "[entity] is a famous figure of [period]."; for the "created" relationship, the template is "He created [work]". The entity name is inserted as a placeholder to form the initial response text: "Leonardo da Vinci was a famous figure of the Renaissance. He created the Mona Lisa." Then, a grammar check is performed, adjusting word order and conjunctions to ensure fluency, such as adding the conjunction "in addition," resulting in: "Leonardo da Vinci was a famous figure of the Renaissance. In addition, he created the Mona Lisa." A logical verification step checks the consistency between the response text and the knowledge graph; after confirming that there are no errors, the final response text is output. Experimental tests evaluate the naturalness and accuracy of the response text, and the results show that the response logic is reasonable and meets user expectations.

[0026] Furthermore, referring to Figure 4 Step S6 includes: Natural language processing techniques are used to parse the response text and identify action verbs, object nouns, and state descriptive words. The action verbs are mapped to specific animation sequences in the animation library of the virtual character, including gestures, facial expressions, and body movements; Based on the object name, locate the corresponding object in the virtual scene, and modify the object's color, size, or position based on the state description word; Generate scene control commands to specify the animation playback order, the timing and duration of object state changes; Control the virtual engine to render scene changes according to instructions, ensure that voice playback is synchronized with animation frames, and dynamically adjust the focal length and angle of the virtual camera to highlight key interactive areas; The current state of all elements in the virtual scene is detected in real time and compared with the expected state of the response text. If there is a deviation, the animation parameters are readjusted.

[0027] In this embodiment, the virtual simulation platform receives voice waveform signals and response text. The text analysis sub-step parses the text, identifying action verbs (e.g., "introduction"), object nouns (e.g., "Da Vinci," "Mona Lisa"), and state descriptive words (e.g., "famous"). The action mapping sub-step maps "introduction" to an animation sequence of the virtual character, such as a virtual tour guide making gestures and smiling expressions. The object update sub-step adjusts the virtual scene according to "Mona Lisa," highlighting the painting in the exhibition hall. The scene instruction generation sub-step controls the animation playback order and the timing of object state changes. The rendering synchronization sub-step ensures that the voice playback is synchronized with the virtual character's lip movements and adjusts the camera's perspective to focus on the painting. The state monitoring sub-step checks scene elements in real time to ensure consistency with the response content. Experimental tests verify synchronization and visual effects; user feedback indicates that the virtual environment responds promptly and provides a strong sense of immersion.

[0028] Furthermore, step S2 uses an end-to-end speech recognition model to directly map the speech signal into a text sequence without performing intermediate feature extraction, and also includes: The digital speech signal is input into a neural network model, which extracts global features through an encoder, generates character probability distributions frame by frame through a decoder, and outputs the most likely text sequence through a beam search algorithm.

[0029] In this embodiment, a digital speech signal input end-to-end speech recognition model is used. This model directly processes the speech signal without intermediate feature extraction, mapping the speech sequence to a text sequence through an encoder-decoder structure. Experimental tests use diverse speech samples (such as different accents and speaking speeds) to evaluate the conversion accuracy. The results confirm the text query and the original speech meaning. Figure 1 As a result, the error rate is low.

[0030] Furthermore, step S5 employs waveform generation technology to convert the text sequence into a speech waveform and adds prosody control to adjust intonation and rhythm, specifically including: The text is input into the acoustic model to generate a Mel spectrogram. The spectrogram is then converted into a waveform signal using a vocoder, and pauses and accents are added based on the punctuation marks in the text.

[0031] In this embodiment, the text-input speech synthesizer employs waveform generation technology. The synthesizer parses the text into a phoneme sequence, adds prosodic control (such as adding pauses and stresses based on punctuation marks), and generates a digital speech waveform signal. In experimental testing, the clarity and naturalness of the synthesized speech were verified through subjective evaluation, with most users finding the speech output fluent and easy to understand.

[0032] Furthermore, it also includes dialogue management steps: Store the context information of the current interaction, including user query history and system responses, and retrieve the context in subsequent queries to optimize knowledge graph queries and response generation, specifically including: Maintain a context cache that records entities and relationships from recent interactions. When a new query is entered, the relevant entities are retrieved from the cache first.

[0033] Furthermore, the dialogue management steps include: The cache context uses a first-in-first-out queue to store the query text, response text, and related entities from the five most recent interactions; The context is retrieved, entities are extracted from the new query, and compared with entities in the cache. If there are overlapping entities, the cached relationships are used as additional information to input the knowledge graph query.

[0034] In this embodiment, the system stores context information of the current interaction, such as user query history and system responses. The context caching mechanism uses a first-in, first-out queue to store entities and relationships from the five most recent interactions. When a user subsequently queries "What other works did he create?", the system retrieves the "da Vinci" entity from the cache, prioritizing the use of cached information to optimize the knowledge graph query and directly retrieving other works by da Vinci. Experimental tests show that context usage improves query efficiency and reduces repetitive work.

[0035] Furthermore, the method also includes an error handling step: when speech recognition or knowledge graph query fails, a default prompt speech is generated, and an error prompt animation is displayed in the virtual simulation platform, specifically including: When the speech recognition confidence level is below the threshold or there is no matching entity in the knowledge graph, an error handling process is triggered, a pre-recorded error message is played, and the virtual character displays a confused expression.

[0036] In this embodiment, during testing, a simulated speech recognition failure scenario (such as unclear user speech) was implemented. When the system detected that the speech recognition confidence level was below a threshold, error handling was triggered: a default prompt message, "Sorry, I didn't hear you clearly, please repeat," was generated, and the virtual character displayed a confused expression and gestures. Experiments verified that error handling effectively guided the user to re-enter the information.

[0037] Furthermore, the method also includes a user authentication step: User identity is identified through voice biometrics, and personalized knowledge graph queries and virtual scene configurations are made based on the identity information, specifically including: Extract voiceprint features from the speech signal, compare them with the user's voiceprint in the database, and after successful authentication, load the user's preference settings and historical interaction records.

[0038] In this embodiment, the system identifies users through voice biometrics. At the start of the interaction, the system extracts the user's voiceprint features and compares them with pre-registered voiceprints in the database. Upon successful authentication, user preferences (such as language style) and historical records are loaded, enabling personalized knowledge graph queries and virtual scene configurations. Experimental tests showed high authentication accuracy, and the personalized settings enhanced the user experience.

[0039] The following describes a more specific embodiment: Users enter a virtual simulated history museum exhibition hall containing multiple exhibits (such as paintings and sculptures). Users interact with a virtual guide robot via voice. The system integrates audio input devices (such as microphones), a voice processing module, a knowledge graph database (storing entities such as historical figures, works, and events), and a virtual simulation engine (rendering virtual scenes and characters). After system startup, the virtual environment is initialized, knowledge graph data is loaded, and the system awaits user input.

[0040] A user, interacting with the virtual simulation platform, utters a voice query: "Please provide a detailed introduction to Leonardo da Vinci, a representative figure of the Renaissance, and his major works." The audio input device captures the user's voice signal in real time, and its internal analog-to-digital converter converts the analog voice waveform into a digital signal. During the conversion process, the system employs gain control to ensure stable signal strength and uses filtering techniques to reduce interference from environmental noise (such as background music in an exhibition hall). Experimental testing simulated different noise levels (such as low, medium, and high noise) to verify the voice acquisition quality. The results show that the system effectively preserves the voice content, and the signal clarity meets the requirements for subsequent processing.

[0041] An end-to-end speech recognition model based on digital speech signal input. This model directly processes speech sequences without the need for intermediate acoustic feature extraction steps. The model analyzes the temporal patterns of the speech signal through an encoder, and the decoder maps the phoneme sequence to a word sequence based on a language model, generating the text query "Please provide a detailed introduction to Leonardo da Vinci, a representative figure of the Renaissance, and his major works." During processing, the model considers the coherence and context of the speech to avoid errors caused by accent or speech rate variations. Experimental tests using diverse speech samples (such as different dialects and speech rates) evaluate the conversion accuracy. The results show that the text accurately reflects the meaning of the original speech. Figure 1 As a result, many of the error cases stem from extreme noise environments.

[0042] The text query first undergoes natural language preprocessing: a word segmentation tool divides the sentence into word sequences (such as "Renaissance", "period", "da Vinci"), part-of-speech tagging identifies nouns and verbs, stop words (such as "please" and "introduce") are removed, and key terms "Renaissance period", "da Vinci", and "representative works" are extracted.

[0043] Semantic similarity calculation sub-step: The pre-trained semantic vector model converts key terms and knowledge graph entity labels into vector representations and calculates cosine similarity. For example, the label similarity between "da Vinci" and the entity "Leonardo da Vinci" is the highest.

[0044] Entity selection sub-step: Set a similarity threshold, and select only entities with a similarity higher than the threshold as candidates. In this example, "Da Vinci" and "Renaissance" are selected as matching entities.

[0045] The relation extraction sub-step involves retrieving direct attributes (e.g., Leonardo da Vinci's birthplace and occupation) and relations (e.g., "belonging to" the Renaissance period, "creating" the "Mona Lisa") of matching entities from the knowledge graph, and obtaining adjacent entities (e.g., the year the "Mona Lisa" was created). In experimental testing, the knowledge graph contained a large number of entities, the query response time was within a reasonable range, and the threshold setting effectively filtered out irrelevant entities.

[0046] Entity Relationship Resolution Sub-step: Analyze the acquired entities and relationships, identify the core entity as "Da Vinci", the secondary entities as "Renaissance Period" and "Mona Lisa", and the relationship types include "belongs to" and "created".

[0047] Template selection sub-step: Retrieve a matching template from the predefined response template library. For example, for the "belongs to" relationship, select the template "[Entity] is a key figure in [Period]."; for the "creator" relationship, select the template "Its representative works include [works]". Text filling sub-step: Insert entity names into template placeholders to form preliminary text: "Leonardo da Vinci was a key figure in the Renaissance. His masterpieces include the Mona Lisa." The grammar check sub-step involves performing part-of-speech and syntactic analysis on the text, adjusting word order and conjunctions. For example, adding the conjunction "in addition" and modifiers creates: "Leonardo da Vinci was a key figure in the Renaissance, and in addition, his masterpieces include the Mona Lisa." Logical verification sub-step: Compare the response text with the knowledge graph to confirm the accuracy of the relationship (e.g., whether Leonardo da Vinci was indeed from the Renaissance period).

[0048] Text output sub-step: Output the final response text: "Leonardo da Vinci was a key figure in the Renaissance, whose masterpieces include the Mona Lisa and The Last Supper." Experimental tests evaluated the naturalness of the response, and the results showed that the text was fluent and the information was accurate.

[0049] A text-input speech synthesizer employs waveform generation technology. The synthesizer parses the text into a phoneme sequence and adds prosodic control: pauses are inserted based on punctuation marks (e.g., pauses after periods), emphasis is added on keywords (e.g., "Da Vinci"), and intonation is adjusted to simulate the rhythm of natural speech. Finally, a vocoder generates a digital speech waveform signal. In experimental testing, the synthesized speech was verified through subjective evaluation; users considered the speech output clear and natural.

[0050] Text analysis sub-step: Parse the response text and identify the action verb "introduce", the object nouns "da Vinci" and "mona lisa" and the state descriptor "key".

[0051] Action mapping sub-step: Map the verb "introduce" to the animation sequence of the virtual character, such as the virtual tour guide making a gesture (pointing to the virtual painting), a facial expression (smiling), and a body movement (walking closer to the exhibit).

[0052] Object update sub-step: Based on the object name, locate the "Mona Lisa" painting in the virtual scene and adjust its attributes (such as highlighting and adding labels).

[0053] Scene instruction generation sub-step: Generate control commands, specify the animation playback order (e.g., gestures before movement) and the timing of object changes (e.g., highlighting when the voice plays "Mona Lisa").

[0054] Rendering synchronization sub-step: The virtual engine renders the scene in real time, ensuring that the voice output is synchronized with the character's lip movements, and adjusting the camera angle to focus on the painting.

[0055] Status monitoring sub-step: Continuously monitor the status of virtual elements (such as whether the artwork highlighting is effective), and re-trigger the animation if there is a deviation. Experimental tests verify synchronization, and user feedback indicates that the virtual environment responds promptly.

[0056] The system maintains a context cache to store recently interacted queries and responses. For example, it saves the entity "da Vinci" and the relation "belongs to the Renaissance" from the current query.

[0057] Context caching sub-step: Use a first-in-first-out queue to store the entities and relationships of the last five interactions.

[0058] Contextual retrieval sub-step: When a user subsequently queries "What other works did he create?", the system retrieves the "Da Vinci" entity from the cache and directly queries the creation relationships in the knowledge graph, without needing to re-identify the entity. Experimental tests show that using context improves query efficiency.

[0059] Simulate speech recognition failure scenarios (such as unclear user speech). When the system detects that the speech recognition confidence level is below a threshold, it triggers error handling: generating a default prompt voice, "Sorry, please repeat your question.", and controlling the virtual character to display confused expressions (such as frowning) and gestures (such as shrugging). In testing, error handling effectively guided the user to re-enter the information.

[0060] At the start of the interaction, the system identifies the user through voice biometrics. It extracts the user's voiceprint features (such as spectral characteristics) and compares them with a pre-registered voiceprint database. Upon successful authentication, it loads user preferences (such as detailed explanations of their likes), performs personalized knowledge graph queries (such as prioritizing relationships of interest to the user), and configures virtual scene settings (such as adjusting the appearance of the virtual character). In testing, the authentication accuracy was high, and personalization enhanced the user experience.

[0061] This embodiment comprehensively covers all the contents of claims 1 to 10 through detailed steps. The tests simulated various scenarios (such as complex queries, noisy environments, and multi-turn dialogues), and the evaluation metrics included speech recognition accuracy, response generation quality, and virtual simulation synchronization. All steps are logically coherent, forming a closed loop from speech input to virtual output. The system performed stably, meeting design expectations.

[0062] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0063] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

[0064] The specific embodiments of the invention have been described in detail above, but they are only examples, and this application is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications or substitutions to the invention are also within the scope of this application. Therefore, all equivalent changes, modifications, and improvements made without departing from the spirit and principles of this application should be covered within the scope of this application.

Claims

1. A knowledge graph-based intelligent voice interaction method, applied to robots or virtual simulation platforms, characterized in that, The method includes: S1, the analog voice signal emitted by the user is collected in real time through the audio input device, and the analog voice signal is converted into a digital voice signal; S2, the digital speech signal is input into the speech recognition engine. The speech recognition engine analyzes the acoustic features and language model of the speech signal, segments the continuous speech into phoneme sequences, and maps the phoneme sequences into word sequences based on a dictionary to generate query text. S3, the query text is segmented and tagged with parts of speech. After removing stop words, key terms are extracted. The key terms are matched with the tags of entities in the knowledge graph using string matching and semantic matching to obtain matching entities. All direct attributes and relationships of the matching entities, as well as adjacent entities connected by relationships, are retrieved from the knowledge graph. S4. Based on the obtained entities and relationships, determine the response topic, select the sentence structure corresponding to the relationship type, insert the entity name into the sentence template, adjust pronouns and conjunctions to ensure contextual coherence, and generate natural language response text. S5, the natural language response text is input into the speech synthesizer, the speech synthesizer parses the natural language response text into a phoneme sequence, adds prosodic markers to control pitch, duration and stress, and generates a digital speech waveform signal; S6. In the virtual simulation platform, the digital voice waveform signal is converted into an audio stream, which drives the virtual character's lip movements to synchronize with the voice, parses the keywords in the response text, triggers the animation script of the object in the virtual scene, and adjusts the lighting, camera angle and background elements to match the response content.

2. The intelligent voice interaction method based on knowledge graph as described in claim 1, characterized in that, Step S3 includes: The key terms and knowledge graph entity labels in the query text are converted into feature vectors using a pre-trained semantic vector model, and the cosine similarity between the feature vectors is calculated as the semantic similarity. Set a similarity threshold, select entities with semantic similarity higher than the similarity threshold as candidate matching entities, and sort the candidate matching entities by similarity. Retrieve all direct relation triples of the candidate matching entity from the knowledge graph, including relation type and target entity, and further retrieve the direct attributes and relations of the target entity.

3. The intelligent voice interaction method based on knowledge graph as described in claim 1, characterized in that, Step S4 includes: Analyze the acquired entities and relationships to identify primary and secondary entities, as well as the direction and type of relationships; Based on the relationship type, a matching template is retrieved from a predefined response template library, which is categorized by relationship type, and each template contains variable placeholders and fixed text; Insert the main entity name into the subject placeholder of the template, insert the object placeholder from the entity name into the template, and insert the relation description into the predicate part of the template; Perform part-of-speech and syntactic analysis on the filled text to ensure subject-verb agreement and correct tense, and adjust the use of articles and prepositions; The generated response text is compared with the relationships in the knowledge graph to confirm whether the response text accurately reflects the real relationships between entities. The validated text is formatted, punctuation marks and paragraph separators are added, and the final natural language response text is output.

4. The intelligent voice interaction method based on knowledge graph as described in claim 1, characterized in that, Step S6 includes: Natural language processing techniques are used to parse the response text and identify action verbs, object nouns, and state descriptive words. The action verbs are mapped to specific animation sequences in the animation library of the virtual character, including gestures, facial expressions, and body movements; Based on the object name, locate the corresponding object in the virtual scene, and modify the object's color, size, or position based on the state description word; Generate scene control commands to specify the animation playback order, the timing and duration of object state changes; Control the virtual engine to render scene changes according to instructions, ensure that voice playback is synchronized with animation frames, and dynamically adjust the focal length and angle of the virtual camera to highlight key interactive areas; The current state of all elements in the virtual scene is detected in real time and compared with the expected state of the response text. If there is a deviation, the animation parameters are readjusted.

5. The intelligent voice interaction method based on knowledge graph as described in claim 1, characterized in that, Step S2 uses an end-to-end speech recognition model to directly map speech signals into text sequences without intermediate feature extraction, and also includes: The digital speech signal is input into a neural network model, which extracts global features through an encoder, generates character probability distributions frame by frame through a decoder, and outputs the most likely text sequence through a beam search algorithm.

6. The intelligent voice interaction method based on knowledge graph as described in claim 1, characterized in that, Step S5 employs waveform generation technology to convert the text sequence into a speech waveform and adds prosody control to adjust intonation and rhythm, specifically including: The text is input into the acoustic model to generate a Mel spectrogram. The spectrogram is then converted into a waveform signal using a vocoder, and pauses and accents are added based on the punctuation marks in the text.

7. The intelligent voice interaction method based on knowledge graph as described in claim 1, characterized in that, It also includes dialogue management steps: Store the context information of the current interaction, including user query history and system responses, and retrieve the context in subsequent queries to optimize knowledge graph queries and response generation, specifically including: Maintain a context cache that records entities and relationships from recent interactions. When a new query is entered, the relevant entities are retrieved from the cache first.

8. The intelligent voice interaction method based on knowledge graph as described in claim 7, characterized in that, The dialogue management steps include: The cache context uses a first-in-first-out queue to store the query text, response text, and related entities from the five most recent interactions; The context is retrieved, entities are extracted from the new query, and compared with entities in the cache. If there are overlapping entities, the cached relationships are used as additional information to input the knowledge graph query.

9. The intelligent voice interaction method based on knowledge graph as described in claim 1, characterized in that, It also includes error handling steps: when speech recognition or knowledge graph query fails, a default prompt speech is generated and an error prompt animation is displayed in the virtual simulation platform, specifically including: When the speech recognition confidence level is below the threshold or there is no matching entity in the knowledge graph, an error handling process is triggered, a pre-recorded error message is played, and the virtual character displays a confused expression.

10. The intelligent voice interaction method based on knowledge graph as described in claim 1, characterized in that, It also includes user authentication steps: User identity is identified through voice biometrics, and personalized knowledge graph queries and virtual scene configurations are made based on the identity information, specifically including: Extract voiceprint features from the speech signal, compare them with the user's voiceprint in the database, and after successful authentication, load the user's preference settings and historical interaction records.

Images