Deep semantic parsing voice answering method and system, electronic device, and storage medium

Abstract

The present application relates to the field of natural language processing and speech recognition in the field of artificial intelligence, and discloses a deep semantic analysis voice answering method and system, an electronic device and a storage medium, comprising the following steps: offline preprocessing book content to construct a knowledge graph containing physical space and semantic information; real-time acquisition of user voice and eye movement data stream; through the fusion of the pronoun in the voice and the eye movement fixation point coordinates, the accurate mapping of the fuzzy physical reference is realized; based on the eye movement data stream analysis, the user's implicit cognitive state is inferred, and the knowledge graph is dynamically reconstructed; based on the user state and the dynamic knowledge graph, an active teaching dialogue strategy is generated; finally, the strategy is converted into a multi-modal response output of voice and augmented reality visual annotation. The present application solves the problem that the prior art cannot understand the user's fuzzy reference and implicit cognitive difficulty, and realizes active, intelligent and natural interactive teaching guidance.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and speech recognition technology in the field of artificial intelligence, and in particular to deep semantic parsing speech question answering methods, systems, electronic devices and storage media. Background Technology

[0002] In recent years, with the development of artificial intelligence technology, intelligent tutoring systems and reading aids integrating natural language processing and speech recognition functions have been widely used, enhancing the interactivity and accessibility of learning resources. These systems can, to a certain extent, automatically parse text content and support basic voice question-and-answer interaction.

[0003] However, existing technologies still have limitations when applied to in-depth reading and learning assistance scenarios for physical books. Most current solutions rely on keyword retrieval or preset template matching to understand book content. This approach results in a superficial understanding of knowledge, especially when faced with technical terms, complex formulas, and the logical connections between charts and text, making accurate analysis and expression difficult.

[0004] Furthermore, these systems typically employ a passive, responsive interaction model, requiring users to initiate requests through a fixed set of instructions or clear, complete natural language questions. This approach lacks naturalness and fluency, failing to understand the natural, conversational, and ambiguous instructions a user might give while reading a physics book. For instance, when a user points to a point in the book and asks what it is, the system lacks the ability to perceive the user's physical focus and therefore cannot respond effectively.

[0005] More importantly, existing technologies lack the ability to understand users' implicit cognitive states. The system cannot detect in real time whether users encounter difficulties during reading comprehension. For example, when a user shows confusion while trying to understand the intrinsic connection between two non-adjacent knowledge points, the system cannot proactively identify this cognitive bottleneck and therefore cannot provide forward-looking, targeted guidance and inspiration. This passive question-and-answer mode limits the system's role in improving the depth and efficiency of user learning, resulting in an overall user experience and final learning outcome that fall short of expectations. Summary of the Invention

[0006] The technical problem this invention aims to solve is that existing electronic reading aids are usually passively responsive, making it difficult to understand the vague, nonverbal interactive intentions of users when reading physical books, and even more difficult to proactively perceive the cognitive difficulties encountered by users when establishing knowledge connections, resulting in an unnatural interactive experience and low teaching aid efficiency.

[0007] To address the aforementioned technical problems, this invention provides a deep semantic parsing voice Q&A method, system, electronic device, and storage medium. By fusing and processing multimodal information such as speech and eye movement, it achieves accurate understanding of user intent and deep inference of user cognitive state, thereby providing proactive and personalized teaching guidance.

[0008] The first aspect of this invention provides a deep semantic parsing speech Q&A method, which is implemented through the following steps: S1. Offline preprocessing of the digital content of the target book is performed to construct an initial knowledge graph. The book content is deconstructed into multiple information units, each of which is assigned its precise location information in the physical space of the page (e.g., bounding box coordinates) and its own attribute information (e.g., content type and data). Further, through semantic parsing technology, core semantic concept nodes and symbol nodes are extracted from the information units. Finally, the information units, semantic concept nodes, and symbol nodes are collectively used as the node set of the graph, and edge sets are established based on their structural relationships, source relationships, semantic relationships, and symbolic semantic relationships, forming a knowledge network structure that integrates physical space information and multi-level semantic information.

[0009] S2. Collect users' multimodal data streams in real time through electronic devices, mainly including voice data streams and eye-tracking data streams.

[0010] S3. The acquired multimodal data streams are fused to achieve precise mapping from the user's fuzzy physical referential instructions to specific information units in the book. A specific technical solution for the multimodal data stream fusion process is as follows: After identifying demonstrative pronouns (such as "this" or "there") and their timestamps from the speech data stream, the system retrieves gaze events from the eye-tracking data stream within a very short time window near those timestamps. By obtaining the precise coordinates of these gaze events on the book page and calculating the spatial distance between these precise coordinates and the bounding box coordinates of each information unit, the system can determine that the information unit with the smallest distance is the user's target, thus accurately eliminating the uncertainty of physical referentiality.

[0011] S4. Based on continuous eye-tracking data stream analysis, the system infers the user's implicit cognitive state and dynamically reconstructs the knowledge graph when a specific cognitive state is triggered. Specifically, the system continuously monitors the user's saccade patterns. When it detects that the user's gaze repeatedly and rapidly moves between two physically non-adjacent information units, the system considers this a potential characteristic of cognitive confusion. At this point, the system calculates a confusion score that integrates the frequency of saccades and the duration of fixation on the two units. When this confusion score exceeds a preset threshold, the system determines the user's implicit cognitive state as difficulty in establishing relationships. This preset threshold is not a fixed empirical value, but a critical score determined through optimization based on pre-collected and statistically analyzed confusion score distributions of a large number of users who actually exhibit difficulty in establishing relationships, thus ensuring the accuracy of the determination. Once the system determines that the user is in a state of difficulty in establishing relationships, it generates a new temporary association edge between the nodes representing the two information units in the current knowledge graph and adds it to the edge set, so that the structure of the knowledge graph dynamically reflects the user's current cognitive challenge.

[0012] S5. Based on the user's explicit questions, the implicit cognitive state inferred by the system, and the dynamically reconstructed knowledge graph, an active teaching dialogue strategy is generated. For example, when the system detects that the user's cognitive state is difficulty in establishing relationships and that corresponding temporary connection edges already exist in the knowledge graph, the system will select and generate an exploratory question-and-answer dialogue action through a preset strategy function. This question-and-answer dialogue action aims to inspire the user to think about the potential connection between the two knowledge points, transforming passive answering into active guidance.

[0013] S6. The dialogue actions generated by the dialogue strategy are transformed into multimodal responses that the user can intuitively perceive and output. This step includes: using natural language generation and speech synthesis technology to convert the dialogue actions into clear speech signals for playback; simultaneously, if the dialogue action is associated with specific information units, the system will retrieve the bounding box coordinates of these units and control the augmented reality display device to highlight or draw virtual connecting lines at the corresponding physical locations of the physical book in the user's field of vision. This response method, which synchronizes voice guidance and visual focus, greatly reduces the user's comprehension load and improves interaction efficiency and learning effectiveness.

[0014] A second aspect of the present invention provides a deep semantic parsing voice Q&A system, the system comprising: A multimodal perception input module is used to acquire speech and eye-tracking data streams in real time; a data storage and communication module is used to store the initial knowledge graph constructed through preprocessing. The system comprises a central processing module and a multimodal response output module. The central processing module is the core of the system and is configured to execute the key processing steps of the aforementioned methods, including: The system processes multimodal data streams to achieve accurate referential resolution, infers the user's implicit cognitive state based on eye-tracking data analysis, dynamically reconstructs the knowledge graph under specific state triggers, and finally combines all information to generate an active teaching dialogue strategy. The multimodal response output module is responsible for converting the dialogue actions generated by the central processing module into speech and visual signals and presenting them to the user.

[0015] A third aspect of the present invention provides an electronic device whose hardware structure includes an input unit, an output unit, a memory, and a processor. The input unit integrates a microphone for acquiring speech and an eye-tracking sensor for acquiring eye-tracking data. The output unit integrates a speaker for playing speech and an augmented reality display unit for overlaying virtual visual information. The memory stores executable computer program instructions and an initial knowledge graph. The processor is configured to perform all the method steps described in the first aspect of the present invention when executing the computer program instructions in the memory.

[0016] A fourth aspect of the present invention provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, can implement any of the methods described in the first aspect of the present invention.

[0017] In summary, the present invention has at least one of the following beneficial technical effects: 1. This invention achieves precise mapping and natural interaction of users' ambiguous physical referential commands. By synchronously acquiring voice and eye-tracking data streams in real time and integrating automatic speech recognition, natural language understanding, and eye-tracking data, it can accurately identify demonstrative pronouns in the user's speech. Furthermore, by spatially matching the user's gaze coordinates within the corresponding time window with the bounding box coordinates of the book information unit, it effectively eliminates ambiguous referentials, transforming the user's colloquial, imprecise commands into precise target information units that the system can understand, thus improving the naturalness and efficiency of human-computer interaction.

[0018] 2. This invention can deeply infer the user's implicit cognitive state and provide proactive teaching strategies. By continuously monitoring the user's eye movement patterns and focusing on the user's repeated eye saccades between two non-adjacent information units, the system can calculate and evaluate a confusion score. When this confusion score exceeds a preset threshold determined through optimization based on a large amount of user data, the system can intelligently infer that the user is facing an implicit cognitive state of difficulty in establishing relationships. Based on this inference, the system further dynamically generates temporary association edges in the knowledge graph to reflect the user's confusion points, and proactively generates exploratory questioning dialogue actions accordingly, thereby transforming passive answering into proactive inspiration, guiding the user to establish knowledge connections, and improving the pertinence and effectiveness of teaching assistance.

[0019] 3. This invention provides a user-perceptible multimodal response to optimize the learning experience. Through natural language generation and speech synthesis technologies, proactive teaching dialogue strategies are converted into speech signal output, ensuring timely information delivery. Simultaneously, for dialogue actions involving specific information units, the system retrieves the bounding box coordinates of the corresponding information unit and controls the augmented reality display device to highlight or draw virtual connecting lines or other visual annotations on the physical book within the user's field of vision. This response method, which synchronizes voice guidance with visual focus, intuitively links the system's intent with the user's attention, effectively reducing cognitive load and enhancing the user's understanding and absorption efficiency of the answers. Attached Figure Description

[0020] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a block diagram of the system hardware structure of the present invention. Detailed Implementation

[0021] The following is in conjunction with the appendix Figure 1 - Appendix Figure 2 The present invention will be further described in detail below.

[0022] like Figure 1 As shown, this invention provides a deep semantic parsing voice question answering method, which includes the following steps: S1. Perform offline preprocessing on the target book content to construct an initial knowledge graph containing physical spatial information and semantic information. Step S1 first uses document layout analysis technology to divide the digitized content of the book into multiple information units, represented as a set of information units. Each information unit Each element is recorded with its unique identifier, content type (e.g., text paragraph, formula, chart), content data, and bounding box coordinates in the physical space of the page. Then, semantic parsing is performed on different types of information units: for text-type information units, a pre-trained language model is used to extract their semantic features; for formula-type information units, their symbolic structure is parsed and associated with the corresponding text description. Finally, the parsed concepts, entities, formulas, etc., are used as nodes, and the predefined semantic relationships between nodes are used as edges to construct an initial knowledge graph. ,in For the initial node set, This is the initial edge set.

[0023] S2. During the interaction between the user and the physical book, multimodal data streams are collected in real time and synchronously via electronic devices. The multimodal data streams include voice data streams recording the user's voice commands. and eye-tracking data streams that record user eye-tracking parameters. Eye-tracking data stream At each point in time Includes a set of parameters: ; in, The coordinates of the user's gaze point in the screen coordinate system; The user's pupil diameter; For the type of eye-tracking event, such as fixation or saccade, all acquired data is accompanied by a precise timestamp for subsequent alignment processing.

[0024] S3. Process the acquired multimodal data stream to achieve precise mapping from the user's fuzzy physical referential instructions to specific information units in the book. Step S3 first converts the speech data stream into text using automatic speech recognition technology, and then uses natural language understanding technology to identify the user's question intent and demonstrative pronouns from the text. (When at a certain time point...) When an indicator pronoun is detected, within the time window The system retrieves one or more gaze events from the eye-tracking data stream and obtains their gaze coordinates. By calculating the coordinates of the gaze point The spatial distance from the center point of the bounding box of all information units in the book is used to determine the target information unit with the smallest distance. This completes the resolution of the referential meaning.

[0025] S4. Based on continuous eye-tracking data stream analysis, infer the user's implicit cognitive state and dynamically reconstruct the knowledge graph under the trigger of specific cognitive states. The core of step S4 lies in identifying the user's difficulty in establishing relationships during the comprehension process. This occurs when the user's gaze is detected on two non-adjacent information units. and Calculate a confusion score when saccades repeatedly. The formula for calculating the confusion score is: ; in, The length of the analysis time window; In the time window Inside, the starting and ending points of the eye saccade trajectory fall on the information unit respectively. and Total number of occurrences within the bounding box; In the time window Inside, it occurs in the information unit and The average fixation duration on the screen; A logarithmic function is used to non-linearly scale the effect of average fixation time to reflect the level of cognitive effort invested by the user.

[0026] When the calculated confusion score Exceeding the preset threshold At that time, the current user's implicit cognitive state will be... The system has been determined to be difficult to establish relationships. This determination will trigger the knowledge graph dynamic reconstruction engine, which will then refactor the current knowledge graph. In the middle, it represents an information unit. and Generate a new temporary association edge between the two nodes. Weight of temporary associated edges The value is assigned to the calculated confusion score, i.e. By adding this temporary associated edge to the current edge set, we obtain the updated knowledge graph. , This indicates that by creating new temporary associated edges... Add to the current edge set In this way, an updated knowledge graph can be obtained. ,in, For knowledge graphs, For a set of nodes, For edge set, This is a temporary associated edge.

[0027] S5. Based on the user's explicit questions, inferred implicit cognitive states, and dynamically reconstructed knowledge graph, a proactive teaching dialogue strategy is generated. Step S5 uses a strategy function. Choose the optimal dialogue action .

[0028] ; Among them, the current state of the system Represented by a triple: For the user's explicit question content, This refers to the inferred implicit cognitive state. This is the knowledge graph at the current moment. When... It is difficult to establish a relationship and There are temporary associated edges in the middle. At that time, the policy function Output dialogue actions This is an exploratory question. The exploratory questioning action aims to generate a question about the potential relationship between two knowledge nodes connected by a temporary associative edge.

[0029] S6. Transform the dialogue actions generated in step five into user-perceptible multimodal responses and output them. Step S6 first uses natural language generation technology to convert the planned dialogue actions into fluent text. Then, it uses speech synthesis technology to convert the text into speech signals, which are then played through a speaker. Simultaneously with the speech output, if the dialogue action is associated with a specific information unit, such as the information unit associated with an exploratory question action... and The system then retrieves the bounding box coordinates of the two information units and controls the augmented reality display device to highlight or draw virtual connecting lines at the physical location of the physical book in the user's field of vision, so as to achieve synchronization between voice guidance and visual focus.

[0030] S1. Offline preprocessing of the target book content to construct an initial knowledge graph containing physical spatial and semantic information. The implementation of step S1 begins with obtaining a digital version of the target book (such as a scanned image or PDF file). Its specific process can be further broken down into the following core stages: The digitization and structuring of book content involves several steps. First, the digitized book undergoes preprocessing (e.g., noise reduction and skew correction). Then, document layout analysis techniques are used to divide the page into multiple content blocks with clearly defined type labels, such as titles, body text, formulas, and charts. Next, appropriate recognition engines (e.g., Optical Character Recognition (OCR) and mathematical formula recognition) are used to extract the content data for different types of content blocks. Simultaneously, the reading order is determined by analyzing the physical positional relationships of the content blocks. Finally, the processing results are integrated into a set of structured information units. Each information unit contains a unique identifier, content type, content data, page physical bounding box coordinates, and a reading order index.

[0031] Construction of the Digital Twin Model for the Book: Based on the structured information units described above, a digital twin model is constructed that precisely corresponds to the physical layout of the book. This process includes: establishing a unified page-level coordinate system for the book; assigning each information unit a hierarchical unique identifier (e.g., P15-Area3-Formula-1) that reflects its chapter, region, type, and other location; and constructing a hierarchical data structure that integrates non-hierarchical relationships such as chapter and table of contents hierarchy and cross-referencing between text and images. This digital twin model is ultimately stored as a structured document (e.g., JSON or XML), providing an accurate digital map for subsequent interactions.

[0032] Multimodal semantic parsing and linking: Deep semantic parsing of the content of each information unit in the digital twin model.

[0033] For text units, a pre-trained language model is used to perform named entity recognition and relation extraction to extract entities such as physical concepts and mathematical theorems and the semantic relationships between them, and to generate semantic vectors.

[0034] For formula units, by analyzing their adjacent textual context, the formula symbols and their textual definitions (such as symbols) are realized. The semantic implementation between concepts and external forces.

[0035] For graph units, object detection and topological relationship analysis are used to understand their internal structure and information (such as causal relationships in flowcharts). Finally, cross-modal semantic linking operations are performed to associate the same semantic entities identified in text, formulas, and graphs, forming a unified, semantically interconnected network.

[0036] Construction of the initial knowledge graph: Integrate all the parsed structured semantic information to construct a unified, static initial knowledge graph. .

[0037] Node set Generation: Based on the parsing results, three types of nodes are instantiated: content unit nodes (corresponding to each information unit), semantic concept nodes (corresponding to normalized conceptual entities, such as resultant external force), and symbol nodes (corresponding to symbols in formulas, such as...). ).

[0038] Edge set Generation: Various types of directed edges are established between the aforementioned nodes to represent their intrinsic connections. These mainly include: structural relationship edges (such as the inclusion relationship between chapters and paragraphs), source relationship edges (linking concept nodes to their source content unit nodes in the book), semantic relationship edges (such as the descriptive relationship between concepts), and symbolic semantic relationship edges (linking symbol nodes to the concept nodes they represent, such as…). (Representation of the relationship between the resultant external forces).

[0039] Through the above process, a heterogeneous initial knowledge graph is finally constructed. This initial knowledge graph not only encodes the explicit content structure of the book, but also integrates deep semantic relationships extracted from different modalities, providing a comprehensive knowledge foundation for subsequent dynamic interaction and intelligent reasoning.

[0040] During the user interaction phase, this invention achieves synchronous acquisition of multimodal data through step S2, which specifically includes: Parallel acquisition: Using an integrated sensor array in an electronic device, voice data streams of explicit user commands and eye-tracking data streams reflecting implicit physiological responses are acquired and recorded in parallel.

[0041] Voice data stream processing: A microphone array is used, and beamforming technology is employed to acquire the user's voice signal to improve the signal-to-noise ratio. This voice signal undergoes front-end processing, including voice activity detection, to generate digital audio segments with clearly defined start and end times.

[0042] Eye-tracking data stream processing: An eye-tracking module consisting of an infrared light source and an infrared camera captures images of the user's eyes. During processing, the eye-tracking module: Perform user-specific calibration to establish a mapping between raw eye readings and the device display coordinate system.

[0043] Real-time analysis of eye images to calculate the coordinates of the user's gaze point and pupil diameter .

[0044] Analyzing continuous raw fixation data allows for the identification and classification of higher-level eye movement events such as fixation and saccades. .

[0045] Cross-modal data synchronization mechanism: To ensure the precise temporal correspondence between the two independent data streams mentioned above, this invention employs a synchronization mechanism based on a unified clock source and timestamp. The synchronization mechanism includes: Source timestamp marking: Using a unified high-resolution hardware clock inside the device, a precise timestamp is bound to the earliest moment when the voice data and eye-tracking data are generated.

[0046] Data encapsulation and sorting: All timestamped data (regardless of source) is encapsulated into standardized data packets and sent to a central synchronization buffer queue sorted by timestamp.

[0047] Synchronous consumption: All downstream processing modules sequentially retrieve data packets from the head of this central synchronization buffer queue for processing.

[0048] Through this mechanism, even if there are latency differences in the processing links of different data channels, the system can ensure that the data processed at any time is the earliest event in time among all modalities, thus providing a reliable time consistency guarantee for subsequent cross-modal data fusion and analysis.

[0049] To achieve precise mapping from user's fuzzy physical reference instructions to specific information units in the book, this invention performs the following steps: First, a digital audio segment containing the user's speech, from a central synchronization buffer queue, is input into the Automatic Speech Recognition (ASR) engine. The ASR engine combines a domain-specific language model pre-built based on the entire text content of the target book, which improves the accuracy of recognizing technical terms specific to the book, and finally converts the audio signal into a text string.

[0050] The text string is then input into the Natural Language Understanding (NLU) engine, which performs two parallel core tasks: Intent recognition: Determine the core purpose of the user's question through a predefined intent classification system (such as including request definition, request explanation, etc.).

[0051] Slot filling: Extract key information fragments required to execute the intent from the text, and specially configure an indicator pronoun recognizer to detect whether the slot value contains words with physical referential functions such as this, that, and here.

[0052] After completing the above tasks, the NLU engine integrates the parsed results into a structured data object. This data object contains the original text, intent category, slot list, and a timestamp associated with the speech. When a slot value contains an indicator pronoun, a boolean referential marker is appended to that slot.

[0053] An eye-tracking-based dereference algorithm is activated only if the reference marker of the structured data object is true. The execution process of this dereference algorithm is as follows: Extract the timestamps corresponding to the demonstrative pronouns from the data object. and around Define a time window ;in and These are preset time offsets, used to define the effective eye-tracking data retrieval range before and after the voice command is given.

[0054] Next, the referential resolution algorithm retrieves all eye-tracking events whose timestamps fall within the time window from the central synchronization buffer queue, and then filters out all fixation events from these eye-tracking events to form a fixation event set. Each gaze event All include the screen coordinates of their center point. Duration and the timestamp of the time of occurrence .

[0055] To determine a unique target information unit from multiple candidate information units, this algorithm identifies each information unit within the current user's field of vision. Calculate a correlation score The association score comprehensively considers three factors: spatial proximity between the gaze point and the information unit, temporal proximity between the gaze event and the voice command, and gaze duration. The formula for calculating the association score is as follows: ; in, It is the total number of gaze events selected within the time window; It is the first The duration of a gaze event; It is a Gaussian function, and its specific form is: This function is used to weight the input values ​​based on their deviation from the center; the closer the input value is to the center, the closer the function's output value is to 1; the farther the input value is from the center, the closer the function's output value is to 0. The output value of the Gaussian function, The input value, in this invention, represents spatial distance or time difference. Standard deviation controls the rate of weight decay; Is calculating the first The coordinates of the center point of a gaze event With information unit coordinates of the center point of the bounding box The Euclidean distance between them; This is the standard deviation of the spatial Gaussian function, a parameter that controls the rate of weight decay of spatial distances. A smaller... The value means that only gaze points very close to the center of the information unit can receive a high weight; It is the first The moment of occurrence of a gaze event With voice command timestamps The absolute value of the time difference between them; This is the standard deviation of the time Gaussian function; this parameter controls the rate at which the weights decay from the time differences. A smaller one... The value means that only gaze events that are very close to the voice command in time will receive high weight.

[0056] After calculating the association score for all information units within the field of view, the algorithm performs a maximization operation, selecting the information unit with the highest association score as the final target information unit.

[0057] ; in, The final identified target information unit; A mathematical operation, referring to finding the function (here) that makes the function ( The parameter to obtain the maximum value (here is) ); The set of all visible information units within the current user's field of vision; Information unit The correlation score.

[0058] Finally, the determined target information unit is used. The unique identifier replaces the slot value marked as the reference in the input structured data object (for example, replacing this formula with P15-Area3-Formula-1), thereby completing the mapping from fuzzy physical reference to precise digital entity, and passing the updated structured data object to subsequent modules for processing.

[0059] The S4 step aims to infer a user's implicit cognitive state based on their eye-tracking data and dynamically reconstruct the knowledge graph accordingly. The S4 step mainly includes the following core components: Implicit cognitive state inference: This method continuously analyzes eye-tracking data streams to identify a type of information unit that represents a user's understanding of two spatially non-adjacent spaces. and ) specific eye-tracking patterns that cause difficulties, i.e., the user's gaze is on and Scan the area repeatedly.

[0060] To quantify the pattern, the system calculates a confusion score. This score combines data from a preset time window. Inside, and The frequency of eye saccades, and the user's... and The system measures the average duration of fixation. When the calculated confusion score exceeds a preset threshold, the system classifies the user's current cognitive state as difficulty in establishing relationships and generates a list containing... , Identifiers and cognitive state objects for confusion scores.

[0061] Dynamic knowledge graph reconstruction: When a cognitive state object that is difficult to establish relationships is received, the dynamic knowledge graph reconstruction engine is activated. This engine first reconstructs the current knowledge graph... In the middle, the information unit was located. and The corresponding semantic concept node ( and ).

[0062] Before creating a new connection, the engine will check... and Does a close semantic path already exist between them? If not, or if the path is too long, perform a reconstruction operation: in and A new temporary edge of type speculative association is created between them. At the same time, the calculated confusion score is assigned as the weight of this edge, which quantitatively reflects the difficulty for the user to establish this association.

[0063] Ultimately, this new edge is added to the current knowledge graph, generating an updated knowledge graph that includes inferences about the user's current cognitive state. It is then passed to the subsequent dialogue strategy generation module for use.

[0064] Before generating specific dialogue actions, the S5 step first constructs a comprehensive and structured representation of the current interaction state. This representation aims to integrate dynamically changing information from different sources into a unified input that can be processed by subsequent policy functions. At each decision point... Interaction status Defined as a triple: The first part Explicit questions on behalf of the user. Instead of raw speech or text, this is a structured data object generated after processing in step S3, which has undergone referential resolution. This data object explicitly contains the user's query intent (e.g., a request for a definition) and one or more target entity slots that have been precisely mapped to unique identifiers of specific information units in the book's digital twin model. To facilitate subsequent model processing, the structured data object is converted into a numerical vector. During this conversion, the intent category is one-hot encoded, while the unique identifier in the target entity slot is used to find its corresponding semantic concept node in the initial knowledge graph and extract the pre-computed semantic vector of that node.

[0065] Part Two This represents the user's implicit cognitive state inferred through the S4 step. It is also represented as a numerical vector. In most cases, when no specific cognitive difficulty pattern is detected, this vector is a predefined zero vector or a default value vector, indicating that the user's cognitive state is normal. When the implicit cognitive state inference model outputs a judgment result of difficulty in establishing relationships, this vector consists of information related to that state. Specifically, the system extracts the information unit pairs that triggered this judgment. The corresponding semantic concept node and The pre-computed semantic vectors are then compared with the normalized perplexity scores. The features are then concatenated to form a feature vector that can represent the current cognitive dilemma.

[0066] Part Three A knowledge graph representing the current moment. (Knowledge graph) It is an updated version that includes temporary associated edges generated by the knowledge graph dynamic reconstruction engine. Since the entire graph structure cannot be directly used as input, the system employs a graph neural network encoder to process the knowledge graph. The graph is processed to generate a fixed-dimensional global graph embedding vector. This graph neural network encoder aggregates the features of each node in the graph and the features of its neighbors, passing information layer by layer, ultimately outputting a single vector representation at the top level that captures the entire knowledge graph's topological structure and node semantic information. This vector representation is particularly effective in reflecting the impact of newly generated temporary connections on the overall graph structure.

[0067] Ultimately, at each decision-making point in time... The system concatenates the numerical vectors corresponding to the above three parts to form a single, high-dimensional comprehensive state vector. The integrated state vector It fully encapsulates the user's explicit commands, implicit cognitive state, and current global knowledge background, and passes them as a standardized input to the active teaching dialogue strategy function to calculate and select the next optimal dialogue action.

[0068] After obtaining the comprehensive state vector of the current interaction state. Subsequently, this method selects the next optimal dialogue action through a policy decision mechanism. The core of this mechanism is a policy function based on deep reinforcement learning. The goal of this function is to select an action that maximizes long-term teaching effectiveness given the current state. First, this method defines a hierarchical dialogue action space that includes various teaching interventions. Each action in this action space is an executable atomic operation, which can be categorized as follows: Response actions: These actions directly respond to the user's explicit questions. .include: For the target entity, directly provide the most relevant text or chart content retrieved from the knowledge graph.

[0069] Extract a definitional description of the target semantic concept.

[0070] Find and present instances in the knowledge graph that are associated with the target semantic concept.

[0071] Active intervention actions: These actions are driven by the user's implicit cognitive state. Triggers are designed to address potential learning barriers. These include: When it is detected that a user is having difficulty establishing a relationship between two concepts, a supplementary explanation is automatically generated to clarify the logical connection between the two concepts.

[0072] Find the precursor knowledge points of the target concept in the knowledge graph and suggest that the user review the precursor knowledge points first.

[0073] To provide an easier-to-understand analogy for an abstract or complex concept.

[0074] Dialogue management actions: These actions are used to maintain the smoothness of the conversation. They include: After providing a complex explanation, proactively ask the user if they have understood.

[0075] RequestClarification: When the user's voice command intent is ambiguous, request the user to clarify further.

[0076] Policy function Implemented as a deep neural network, we call it a policy network. The input layer of this policy network receives the concatenated combined state vector. The vector undergoes nonlinear transformations through multiple fully connected hidden layers before finally reaching the output layer. Each neuron in the output layer corresponds to the dialogue action space. An atomic action in a given state has an output value that represents the Q-value (action value) of performing the corresponding action in that state.

[0077] At each decision-making moment, the system will use the current state vector The input is fed into the policy network, and a forward propagation calculation is performed to obtain the Q-values ​​of all candidate actions. Subsequently, the system employs either a greedy strategy (in practical applications) or an epsilon-greedy strategy (during training and exploration) to select the action with the maximum Q-value. As a result of this decision.

[0078] The decision-making logic of this policy network is formed through offline training. During training, the system uses a reward function to evaluate the merits of each action. The reward function is designed to quantify the immediate effects of instructional interactions. For example, if a decrease in the user's confusion score is detected in subsequent eye-tracking data after the system performs the BridgeConcepts action, the action receives a high positive reward; conversely, if the user repeats the question after the system's response or the confusion score remains high, the action receives a negative reward. By training on a large amount of simulated or real user interaction data, the policy network can learn under what states (e.g., when the implicit cognitive state portion of the state vector is activated and the knowledge graph embedding vector reflects a speculative association between two concepts) choosing the BridgeConcepts action yields a higher long-term cumulative reward than choosing the AnswerDirectly action.

[0079] The final output of this strategy decision-making mechanism is the identifier of the selected optimal action and its required parameters (e.g., the parameters for the BridgeConcepts action are...). and (The identifier). This output is passed to the final natural language generation module to generate the final natural language response presented to the user.

[0080] The optimal dialogue action is determined by the strategic decision-making mechanism. After obtaining the relevant parameters, step S6 proceeds to the multimodal response generation stage. The primary task of this stage is to convert this abstract, symbolic action instruction into expressive natural language speech that the user can directly understand. This process is specifically divided into two closely connected sub-steps: natural language generation and speech synthesis.

[0081] The first step is Natural Language Generation (NLG). The input to this step is the selected dialogue action and its parameters, such as BridgeConcepts. Instead of using fixed, rigid text templates, the NLG engine dynamically constructs responses using a fine-tuned, generative, pre-trained language model.

[0082] The specific process is as follows: The NLG engine first, based on the action type and parameters, directs the input to the current knowledge graph. Initiate one or more queries to obtain all the material necessary to generate a response. For example, for the BridgeConcepts action, the engine will retrieve the definitions of concept A and concept B, the existing relationship path between them, and the content of the original information unit that triggered the user's confusion. The engine then encodes these retrieved discrete information fragments, along with the action type itself, into a structured prefixed input sequence. For example, this sequence could be formatted as: [Action: Bridge][ConceptA: Momentum][DefA: Product of mass and velocity][ConceptB: Impulse][DefB: Accumulation of force over time][Relation: Impulse theorem].

[0083] The structured prefix input sequence is fed into a generative language model. During offline training, the model has been fine-tuned on a corpus containing extensive instructional explanations and dialogues, learning how to translate this structured knowledge into fluent, coherent, and logically consistent natural language paragraphs. The model not only ensures the grammatical correctness of the generated text but also adjusts its interpretation strategy based on action type. For example, for the BridgeConcepts action, the model tends to generate sentences using phrases like "firstly…secondly…" or "the key difference is…", which convey comparative or progressive logic.

[0084] The next step is text-to-speech (TTS). This step converts the text string generated by the NLG engine into the final audio waveform. To make the interactive experience more natural and human-like, this method employs a parametric speech synthesis model that can control emotion and prosody.

[0085] This TTS model not only converts text into speech, but also accepts an additional style instruction as input. This instruction precisely controls the tone, rate, stress, and emotional nuances of the generated speech. The generation of the style instruction is directly related to the dialogue action selected in this decision. The types are associated. The system internally maintains a mapping table from action type to speech style. For example: When the action is AnswerDirectly, it maps to a neutral, declarative speech style to convey information objectively.

[0086] When the action is BridgeConcepts and is triggered by a high confusion score, it is mapped to a patient, soothing voice style to calm the user’s underlying anxiety and guide them to understand.

[0087] When the action is ConfirmUnderstanding (e.g., generating the text "Do you understand?"), it is mapped to a question with an upward inflection to conform to the language habits of asking questions.

[0088] During synthesis, the text string and selected style instructions are fed together into the encoder of the TTS model. The model generates an acoustic feature sequence based on the text content and modulates the prosodic information (such as pitch curves and duration models) in the sequence using the style instructions. Finally, the modulated acoustic features are converted into a high-quality audio waveform with predetermined emotional color and expression by a neural vocoder.

[0089] The resulting audio waveform, as part of a complete response to the user interaction, is played through the audio output device of the electronic device (such as a speaker or headphones), thus completing a closed-loop, proactive teaching dialogue interaction.

[0090] To achieve seamless coordination between auditory and visual information, step S6, while outputting natural language speech, generates visual enhancement effects in the user's field of vision through an augmented reality display device that are precisely synchronized with the speech content. This process is coordinated by a synchronous visual controller.

[0091] The first step in this process occurs in the Natural Language Generation (NLG) stage. The NLG engine, based on the selected dialogue actions... When generating explanatory text, in addition to outputting plain text content for speech synthesis, a series of non-speech, machine-readable synchronization signals are embedded within the internal structure of the text. These synchronization signals serve as time anchors, instructing the timing and type of subsequent visual effects.

[0092] The specific format of synchronization signaling is a label containing the instruction type and the target object. For example: When the system is about to interpret a formula, the NLG engine inserts a signaling message in the form of [HIGHLIGHT: P15-Area3-Formula-1] before the sentence that will mention the formula.

[0093] When the system executes the BridgeConcepts action to explain the relationship between two concepts, the NLG engine inserts a signaling message of the form [DRAW_LINK: P15-Area3-Formula-1, P22-Area1-Diagram-2] at the corresponding position of the explanatory statement.

[0094] After the interpretation is complete, a [CLEAR-ALL] signal is inserted to clear all visual effects.

[0095] The synchronous visual controller receives two inputs in parallel: one is the raw text containing complete synchronization signaling from the NLG engine; the other is a detailed word-timestamp map from the text-to-speech (TTS) engine. The map precisely records the start and end times of pronunciation for each word in the final generated audio waveform.

[0096] The synchronous visual controller first parses the raw text containing signaling, extracting a sequence of visual instructions arranged chronologically. For each visual instruction in the sequence (e.g., [HIGHLIGHT: P15-Area3-Formula-1]), the controller looks up the next word in the text that immediately follows the signaling instruction (e.g., the formula). Then, the controller looks up the initial pronunciation time of that word in a word-timestamp mapping table. .

[0097] Based on the retrieval of pronunciation times The controller converts the visual instruction into a rendering command with a precise execution timestamp and sends it to a rendering command queue. For example, the highlight instruction mentioned above would be converted into a command with the following content: at time... Start highlighting the information unit with the unique identifier P15-Area3-Formula-1. For better visual guidance, this execution timestamp can be set to a value higher than [previous value]. Slightly advance the offset by a small amount to ensure that the user's gaze is guided to the correct position before hearing the relevant content.

[0098] The augmented reality rendering engine continuously retrieves and executes commands from the rendering command queue. When a rendering command is received, the engine first uses the unique identifier contained in the command to retrieve the precise 3D spatial coordinates and bounding box information of the information unit in the digital twin model of the book. Subsequently, depending on the type of command, the engine overlays the corresponding virtual visual effects at the physical book page location corresponding to the 3D spatial coordinates.

[0099] Visual effects come in a variety of types, for example: For highlight instructions, the rendering engine renders a semi-transparent, softly glowing color overlay over the bounding box area of ​​the information unit.

[0100] For link instructions, the rendering engine dynamically draws an animated curve with particle flow effects between the center points of the two target information units to visually represent the logical connection between them.

[0101] For annotation instructions, the rendering engine can display a virtual information plaque containing supplementary text next to the target information unit.

[0102] By precisely binding visual instructions to the moment of speech pronunciation, this method ensures that when a user hears "Let me show you this formula," the corresponding formula in their field of vision is already highlighted. This significantly reduces the cognitive load on the user when locating and matching information while listening to explanations, achieving a high degree of synergy between auditory and visual information transmission, thereby enhancing children's reading engagement and learning outcomes. Thanks to the use of a high-precision position tracking sensor in the smart book page and real-time data synchronization with the AR device, a smoother and more realistic interactive experience between virtual content and physical book pages is achieved, greatly improving the user experience.

[0103] like Figure 2 As shown, the present invention also provides a deep semantic parsing voice Q&A system, which includes: a central processing module, a multimodal perception input module, a multimodal response output module, and a data storage and communication module.

[0104] The central processing unit (CPU) serves as the brain of the entire hardware system, responsible for executing the operating system, scheduling tasks, and running all the core algorithms involved in this invention. To provide powerful computing capabilities while maintaining low power consumption, this module adopts a heterogeneous computing architecture, specifically comprising the following processing units: Central Processing Unit (CPU): Employs a high-performance multi-core processor based on the ARM architecture. This unit is primarily responsible for running the device's main operating system, managing the allocation of various hardware resources, executing general logical control flows, and coordinating the work of other processing units. For example, it manages the enqueueing and dequeueing operations of data packets in the central synchronization buffer queue.

[0105] Graphics Processing Unit (GPU): Integrates a powerful mobile GPU. The core task of this unit is to handle all tasks related to graphics rendering, including but not limited to: generating virtual visual effects such as highlights, lines, and information boards in real time according to rendering commands sent by the synchronous vision controller, accurately aligning and rendering these virtual graphics with real-world coordinates provided by the environment tracking subunit, and finally outputting them to the visual display unit.

[0106] Neural Processing Unit (NPU): To execute deep learning models efficiently and with low latency, the system integrates a dedicated NPU. This unit is optimized for handling large-scale tensor operations. All computationally intensive AI models in this invention, including Automatic Speech Recognition (ASR) models, Natural Language Understanding (NLU) models, image-based eye-tracking algorithms, implicit cognitive state inference models, and policy decision networks, are deployed and run on this NPU. Through edge deployment, the system can complete the entire cycle from perception to decision within milliseconds, while ensuring the privacy of user data.

[0107] The multimodal sensing input module is the window through which the system perceives the user and the environment. It consists of a highly integrated array of sensors and is responsible for collecting all the raw data required to realize this invention.

[0108] Audio Acquisition Unit: A microphone array consisting of three to four MEMS (Micro-Electro-Mechanical Systems) microphones is integrated at different locations on the frame of the device (e.g., the front of the left and right temples and the bridge of the nose). This array works together to form a narrow beam pointing towards the user's mouth through multi-channel signal processing and beamforming algorithms, thus clearly picking up the user's voice even in noisy environments. The acquired analog signal is converted into a digital audio stream via a high signal-to-noise ratio 24-bit analog-to-digital converter (ADC).

[0109] Visual acquisition unit: This unit contains two functionally distinct sub-units: Eye-tracking subunit: Located inside the device's frame, facing the user's eyes, each eye is equipped with a high-frame-rate (e.g., 90Hz) infrared (IR) camera and two infrared LED emitters. During operation, the eye-safe infrared LEDs flash in a specific pattern, while the infrared cameras simultaneously capture the Purchiner reflection formed on the pupil and cornea. These images are directly fed into the NPU for real-time calculation of gaze direction, fixation point, and pupil diameter.

[0110] Environmental tracking subunit: A high-resolution wide-angle RGB camera is integrated in the center of the outer side of the device's frame. This camera is responsible for capturing the physical world scene directly in front of the user, and its data is mainly used for two purposes: first, to track the device's position and orientation in three-dimensional space in real time through Simultaneous Localization and Mapping (SLAM) algorithms, providing spatial anchors for the stable attachment of AR visual effects; second, to identify the pages of physical books and match them with the digital twin model of the book.

[0111] The multimodal response output module is responsible for presenting system decisions in a user-perceptible manner, enabling multi-channel and collaborative output of information.

[0112] Visual Display Unit: Employing a pair of advanced diffractive waveguide optical modules in conjunction with a micro-projector, this unit creates a highly transparent virtual display area with a wide field of view (e.g., 50 degrees diagonally) in front of the user. This unit is responsible for precisely overlaying the virtual images rendered by the GPU onto the physical book seen by the user using optical perspective, achieving a seamless integration of virtual information and the real world.

[0113] Audio Output Unit: A pair of miniature speakers or bone conduction transducers are integrated into the left and right temples of the device, near the user's ears. This unit is responsible for playing audio generated by the speech synthesis engine. Employing an open audio solution, it can transmit system voice to the user without interfering with the user's reception of ambient sounds, ensuring both safety and comfort.

[0114] The data storage and communication module provides the system with high-speed data reading and writing capabilities and external connectivity.

[0115] Onboard storage unit: Contains two types of storage media: one is high-speed, low-power LPDDR5 memory (RAM), used to temporarily store program code, model weights, and real-time data in the central synchronization buffer queue during system operation; the other is high-capacity UFS3.1 flash memory, used for persistent storage of the operating system, all applications, offline loaded digital twin models of books, and the initial knowledge graph.

[0116] Wireless communication unit: Integrates a wireless chip supporting Wi-Fi 6E and Bluetooth 5.2. Wi-Fi is mainly used for high-speed downloading of new book content, updating AI models, or non-real-time data synchronization with the cloud. Bluetooth can be used to connect to external controllers or for low-power communication between devices.

[0117] like Figure 1 and Figure 2 As shown, the method of the present invention can be performed by this electronic device. In a preferred embodiment, the electronic device is an integrated augmented reality (AR) glasses design, but the invention is not limited thereto. The electronic device typically includes a processor, a memory, an input unit, an output unit, and a communication bus interconnecting these components.

[0118] The processor is the computing core that executes the method of this invention. In this embodiment, the processor can be a heterogeneous computing platform, specifically comprising: Central Processing Unit (CPU): Used to run the operating system and perform general logic control, such as managing and scheduling the various software modules and data flows in the method of this invention.

[0119] Graphics Processing Unit (GPU): Dedicated to performing graphics rendering tasks, particularly rendering virtual visual effects such as highlights and lines in real time and accurately registering them with the physical world, based on the decisions generated in the method of the present invention.

[0120] Neural Processing Unit (NPU): Dedicated to efficiently performing inference computations for neural network models, including but not limited to: models for speech recognition and natural language understanding, models for eye-tracking data analysis, models for implicit cognitive state inference, and policy network models for generating dialogue policies.

[0121] A memory used to store instructions and data. This memory may include: Non-volatile memory, such as flash memory, is used for persistent storage of the operating system, computer program instructions that execute the methods of the present invention, and offline constructed digital twin models and initial knowledge graphs of books.

[0122] Volatile memory, such as random access memory (RAM), is used to temporarily store instructions and intermediate data, such as real-time acquired audio and eye-tracking data packets, while the processor executes program instructions.

[0123] The input unit is used to obtain external information required for the execution of the method of the present invention, and may specifically include: Microphone or microphone array: Used to capture the user's voice commands and provide raw audio data for speech recognition.

[0124] Eye-tracking sensors: typically consisting of an infrared light source and an infrared camera, used to capture the user's eye movements in real time, thereby obtaining data such as gaze coordinates and saccade paths, providing a basis for referential resolution and cognitive state inference.

[0125] Ambient-aware cameras: used to capture images of the physical environment outside the device, providing spatial positioning anchors for the overlay of augmented reality visual effects through technologies such as Simultaneous Localization and Mapping (SLAM).

[0126] The output unit, used to present the response generated by the method of the present invention to the user, may specifically include: Speakers or bone conduction transducers: used to play audio responses generated by the speech synthesis engine.

[0127] Display Unit: In this embodiment, a semi-transparent display screen, such as a waveguide display, is preferred. This unit is used to overlay the virtual visual information rendered by the GPU with the physical reality seen by the user through the display screen, thereby achieving synchronous visual enhancement.

[0128] The communication bus is used to realize data interconnection and communication between the hardware components such as the processor, memory, input unit and output unit, so as to ensure that the multimodal data stream required by the method of the present invention can be efficiently transmitted and processed.

[0129] The present invention also provides an embodiment of a computer-readable storage medium.

[0130] The computer-readable storage medium is a non-transitory storage medium that stores computer program instructions. Specific forms of this medium may include, but are not limited to: read-only memory (ROM), random access memory (RAM), flash memory, solid-state drive (SSD), or optical disc.

[0131] When the computer program instructions stored on the computer-readable storage medium are executed by one or more processors (such as the processor in the electronic device in Part Three), the electronic device is enabled to implement the deep semantic parsing voice question-answering method as described in Part One. Specifically, executing the instructions will cause the device to perform the following steps: Perform offline preprocessing of book content: This includes structuring the digital book content to establish information units, creating digital twin models containing spatial coordinates for each information unit, performing multimodal semantic parsing on the information units, and constructing an initial knowledge graph containing content nodes and semantic concept nodes based on the parsing results.

[0132] Perform real-time acquisition and synchronization of multimodal information: including controlling the input unit to collect the user's voice data stream and eye-tracking data stream, and adding a uniform timestamp to the data stream to achieve precise synchronization.

[0133] Perform cross-modal context fusion and dereference resolution: This includes parsing voice commands to obtain the user's explicit intent and mapping ambiguous pronouns (such as this one) in the voice to a specific information unit in the digital twin model by associating the voice command timestamp with synchronized eye-tracking data.

[0134] Perform implicit cognitive state inference and dynamic reconstruction of knowledge graph: This includes continuously analyzing eye-tracking data streams to identify specific difficulty patterns exhibited by users when understanding the relationship between two separate information units. When such a pattern is identified, a new edge representing the inferred association is dynamically added between the semantic concept nodes corresponding to the two information units in the knowledge graph, and a weight quantifying the degree of difficulty is assigned to this edge.

[0135] The generation of proactive teaching dialogue strategies includes constructing a comprehensive state representation that includes the user's explicit intent, implicit cognitive state, and current knowledge graph state, and selecting an optimal teaching dialogue action based on this comprehensive state representation through a pre-set strategy model.

[0136] The process of generating and outputting multimodal responses includes generating corresponding natural language text based on the selected dialogue action, synthesizing the text into speech for playback, and generating visual enhancement effects (such as highlighting or connecting lines) that are precisely synchronized with the speech content in time, which are then overlaid and displayed on the corresponding position in the physical book through the output unit.

[0137] In summary, the program instructions stored on the storage medium cover the complete process from knowledge construction, multimodal perception, cross-modal understanding, cognitive inference, intelligent decision-making to multimodal response, enabling electronic devices to realize the proactive teaching dialogue function proposed in this invention.

Claims

1. A deep semantic parsing voice question-answering method, characterized in that, Includes the following steps: S1. Perform offline preprocessing on the target book content to construct an initial knowledge graph containing physical space information and semantic information; S2. Acquire multimodal data streams in real time and synchronously through electronic devices, wherein the multimodal data streams include speech data streams and eye-tracking data streams; S3. Process the collected multimodal data stream to achieve accurate mapping of the user's fuzzy physical reference instruction to the specific information unit of the book; S4. Based on the continuous eye-tracking data stream analysis, infer the user's implicit cognitive state, and dynamically reconstruct the knowledge graph under the trigger of a specific cognitive state; S5. Based on the user's explicit questions, the inferred implicit cognitive state, and the dynamically reconstructed knowledge graph, generate an active teaching dialogue strategy; S6. Transform the dialogue actions generated by the active teaching dialogue strategy into a user-perceptible multimodal response and output it.

2. The deep semantic parsing speech Q&A method according to claim 1, characterized in that, Step S1 includes: The digital content of a book is divided into multiple information units, each of which records the content type, content data, and its bounding box coordinates in the physical space of the page. Semantic parsing is performed on the information unit to extract semantic concept nodes and symbol nodes; The information unit, the semantic concept node, and the symbol node are used as a node set, and the structural relationship, source relationship, semantic relationship, and symbol semantic relationship between nodes are used as an edge set to construct the initial knowledge graph.

3. The deep semantic parsing speech Q&A method according to claim 1, characterized in that, Step S3 includes: Using automatic speech recognition and natural language understanding technologies, demonstrative pronouns and their corresponding time points are identified from the speech data stream; Within the time window corresponding to the time point, one or more gaze events are retrieved from the eye-tracking data stream, and their gaze point coordinates are obtained; By calculating the spatial distance between the gaze point coordinates and the bounding box coordinates of the information unit, the target information unit with the smallest distance is determined, thus completing the referential resolution.

4. The deep semantic parsing speech Q&A method according to claim 1, characterized in that, The S4 step includes: Monitor whether the user's gaze repeatedly saccades between two non-adjacent information units; When repeated saccades are detected, a confusion score that integrates saccade frequency and fixation duration is calculated; When the confusion score exceeds a preset threshold, the user's implicit cognitive state is determined to be difficulty in establishing relationships. The preset threshold is a critical score that is optimized and determined based on the statistical distribution of confusion scores of multiple users exhibiting difficulty in establishing relationships, which is collected in advance.

5. The deep semantic parsing speech Q&A method according to claim 4, characterized in that, The S4 step also includes: When the implicit cognitive state is determined to be difficult to establish a relationship, a new temporary association edge is generated between the two nodes representing the two non-adjacent information units in the current knowledge graph. The temporary associated edge is added to the edge set of the current knowledge graph to obtain the updated knowledge graph.

6. The deep semantic parsing speech Q&A method according to claim 5, characterized in that, Step S5 includes: When the implicit cognitive state is that establishing relationships is difficult and the updated knowledge graph contains the temporary related edges, A strategy function is used to select and output an exploratory questioning dialogue action, which aims to explore the potential relationship between the two nodes connected by the temporary association edge.

7. The deep semantic parsing speech Q&A method according to claim 1, characterized in that, Step S6 includes: The dialogue actions are converted into speech signals using natural language generation and speech synthesis technologies. If the dialogue action is associated with a specific information unit, then retrieve the bounding box coordinates of the information unit; The augmented reality display device is controlled to highlight or draw virtual connecting lines at the physical locations of the physical book in the user's field of vision, so as to synchronize voice guidance with visual focus.

8. A deep semantic parsing voice Q&A system, characterized in that, include: A multimodal sensing input module is used to acquire multimodal data streams, including speech data streams and eye-tracking data streams, in real time and synchronously; The data storage and communication module is used to store the initial knowledge graph containing physical space information and semantic information; The central processing module is configured to process the acquired multimodal data streams to achieve precise mapping from the user's fuzzy physical reference instructions to specific information units in the book. Specifically, based on continuous eye-tracking data stream analysis, the implicit cognitive state of the user is inferred. Under the trigger of a specific cognitive state, the knowledge graph is dynamically reconstructed. Based on the user's explicit questions, the inferred implicit cognitive state, and the dynamically reconstructed knowledge graph, an active teaching dialogue strategy is generated. The multimodal response output module is used to convert the dialogue actions generated by the active teaching dialogue strategy into user-perceptible multimodal responses and output them.

9. An electronic device, comprising an input unit, an output unit, a memory, and a processor, characterized in that, When the processor executes a computer program, it implements the method as described in any one of claims 1-7.

10. A storage medium, characterized in that, It stores a computer program thereon, characterized in that the computer program, when executed by a processor, implements the method as described in any one of claims 1-7.

Images