A toy robot interaction method and system based on speech recognition

Abstract

The application discloses a toy robot interaction method and system based on voice recognition, and relates to the technical field of intelligent robot interaction, which comprises the following steps: collecting robot interaction data, preprocessing and generating standard interaction data; extracting voice evidence, semantic slot, action primitive and state checkpoint, and constructing an interaction heterogeneous graph; when there are multiple candidate intentions, determining semantic divergence items, selecting a non-main task executor to inquire and generating revised anchor data; inputting the interaction heterogeneous graph and the revised anchor data into an improved GTN model to generate responsibility migration results; dividing a no-responsibility action domain, a responsibility migration domain and a responsibility termination boundary; dividing responsibility actions into transient actions and commitment actions, establishing a two-state commitment chain and determining a revised boundary; determining actions to be revised, retaining commitment actions, restoring the robot state and executing a local replacement action sequence, and updating the interaction heterogeneous graph. The application realizes semantic inquiry and local error correction, and improves the accuracy and continuity of toy robot interaction.

Description

Technical Field

[0001] This invention relates to the field of intelligent robot interaction technology, and in particular to a toy robot interaction method and system based on speech recognition. Background Technology

[0002] With the development of speech recognition, intelligent robot control, and child interaction technologies, toy robots are gradually acquiring the capabilities of speech acquisition, speech recognition, action execution, and interactive feedback. Most existing toy robots recognize user speech, map the recognized text to preset intentions, and then invoke the chassis, servo motors, lights, or sound-producing actuators to complete the corresponding actions. Some solutions utilize historical dialogue records and robot status information to correlate and process the current interactive content.

[0003] Existing voice interaction methods often eliminate ambiguity in user speech regarding object, direction, degree of action, or execution order by re-questioning, repeating recognition, or directly executing candidate actions. They lack a verification mechanism that utilizes non-primary task actuators to express candidate semantic differences and obtain user feedback, which can easily change the robot's task state during the semantic confirmation stage.

[0004] When a user negates, corrects, or supplements an action that has already been performed, existing methods often re-identify the current speech and generate a new action sequence as a whole. They lack tracking of the responsibility relationship between the corrected speech, historical semantics, and the performed actions, making it difficult to determine the actual scope of actions affected by the erroneous semantics. They also lack state constraints that distinguish between confirmed actions and correctable actions, which can easily lead to problems such as correctly completed actions being repeatedly undone, local errors triggering overall replanning, and unstable connection between continuous interaction states.

[0005] Therefore, how to provide a toy robot interaction method and system based on speech recognition is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose a toy robot interaction method and system based on speech recognition. This invention fully utilizes speech recognition, heterogeneous graph reasoning, and robot execution control technologies to achieve semantic ambiguity verification, interaction responsibility localization, and local action correction. It has the advantages of high interaction accuracy, small impact range of erroneous actions, and high stability of continuous interaction.

[0007] According to an embodiment of the present invention, a toy robot interaction method based on speech recognition includes: Collect robot interaction data, perform preprocessing on toy robot data, and generate standard interaction data; Based on standard interactive data, voice evidence, semantic slots, action primitives, and state checkpoints are extracted, and an interaction heterogeneous graph is constructed according to semantics and execution relationships. When a user's voice corresponds to multiple candidate intents, the semantic divergence item is determined, a non-primary task executor is selected to generate a proxy confirmation action, feedback is collected, and corrected anchoring data is generated. The interaction heterogeneous graph and the modified anchoring data are input into the improved GTN model. The responsibility migration results are generated through the modified anchoring bidirectional graph transformation layer, the semantic responsibility confluence kernel layer and the action spectrum responsibility migration layer. Based on the liability migration results, determine the no-liability action domain, liability migration domain, and liability termination boundary, and generate the liability action range; The executed actions within the scope of responsibility actions are divided into transient actions and committed actions. Based on user confirmation and interaction cycle, a dual-state commitment chain of interactive actions is established and the correction boundary is determined. Based on the dual-state commitment chain of interactive actions and the correction boundary, the action to be corrected is determined, the committed actions are retained, the corresponding robot state is restored and a local replacement action sequence is generated, and after execution, the interaction heterogeneous graph is updated to generate the toy robot interaction result.

[0008] Optionally, the toy robot data includes user voice data, robot status data, and historical interaction data.

[0009] Optionally, the preprocessing of the toy robot data includes speech noise reduction, speech endpoint detection, speech segment recognition, time format unification, alignment of speech time with action execution time, alignment of robot state time, and association of historical interaction rounds.

[0010] Optionally, constructing the interactive heterogeneous graph includes: Extract speech recognition text, candidate recognition text, and speech moments from user speech data in standard interaction data, generate speech evidence and perform semantic annotation, and generate object slots, action slots, direction slots, degree slots, and order slots; Extract actuator number, action type, action start and end time, action control quantity and robot state from robot state data in standard interactive data, merge them according to actuator number, action type and continuous execution period to generate action primitives, and generate state checkpoints based on robot state at action start and end time. Extract the interaction round number and the association record of user voice and robot response action from the historical interaction data in the standard interaction data. Map voice evidence, semantic slots, action primitives and state checkpoints as nodes. Based on semantic annotation, association records and action sequence, establish semantic attribution edge, action trigger edge, execution order edge and state arrival edge to construct an interaction heterogeneous graph.

[0011] Optionally, the step of collecting feedback and generating corrected anchoring data includes: Read the current voice evidence node and associated semantic slot node from the interaction heterogeneous graph, combine candidate intents according to the candidate slot values ​​corresponding to the same voice evidence, compare object slots, action slots, direction slots, degree slots and order slots, and determine the slots with different values ​​as semantic divergence items; Extract the actuator number and available status from the robot state data, extract the action primitives and actuator call records from the historical interaction data, form the main task executor set by associating the executors with the action primitives of the candidate intents, and select the executors that can map semantic divergence items to different output states from the remaining available executors as proxy executors. The candidate slot values ​​of semantic divergence terms are mapped to different output states of the proxy executor, generating and executing proxy confirmation actions that do not invoke the main task executor set. Extract user feedback voice and feedback voice time after the proxy confirmation action is executed from user voice data and perform semantic annotation. Based on the confirmation feedback, retain the corresponding candidate intent, remove the confirmed candidate intent based on the negative feedback, extract the correction slot value based on the correction feedback and update the candidate slot value corresponding to the semantic divergence item, and determine the target intent and anchor semantic slot node. Write the user feedback voice ID, feedback voice time, feedback semantic type, anchor semantic slot node ID, correction slot value, target intent ID, proxy confirmation action ID, and interaction round ID into the anchoring record to generate corrected anchoring data.

[0012] Optionally, the generation of responsibility migration results includes: The relation adjacency matrix soft selection, meta-path adjacency matrix combination, and graph feature propagation structure of the GTN model are retained. The graph transformation layer of the GTN model is transformed into a modified anchored bidirectional graph transformation layer. A semantic responsibility confluence kernel layer and an action genealogy responsibility migration layer are set sequentially at the output of the modified anchored bidirectional graph transformation layer to construct an improved GTN model. Generate a corresponding relational adjacency matrix based on the semantic belonging edge, action trigger edge, execution order edge and state arrival edge in the interaction heterogeneous graph. Read the user feedback voice number, anchor semantic slot node number and interaction round number from the corrected anchoring data to generate user feedback voice nodes. Connect the user feedback voice nodes to the anchor semantic slot nodes to form anchoring relationships and add the anchoring relationships to the relational adjacency matrix set. The modified anchoring bidirectional graph transformation layer sets up a forward execution channel and a modified tracing channel. The forward execution channel performs soft selection and weighted combination on the relation adjacency matrix, combines the relation adjacency matrix obtained from the adjacency graph transformation layer by layer and propagates node features to generate a forward execution metapath graph from historical voice evidence nodes through semantic slot nodes to executed action primitive nodes. The modified tracing channel takes the user feedback voice node and anchoring relationship as the starting point, performs soft selection, layer-by-layer combination and node feature propagation on the relation adjacency matrix within the corresponding interaction round of the modified anchoring data, and generates a modified tracing metapath graph from anchoring semantic slot nodes to historical executed action primitive nodes. The semantic responsibility confluence core layer aligns the forward execution metapath graph and the correction and tracing metapath graph according to the interaction round number, semantic slot node number and action primitive node number. Metapaths that pass through the same historical semantic slot node, reach the same executed action primitive node and whose action execution time is earlier than the user feedback voice time are written into the responsibility confluence unit. Responsibility confluence units that are associated with the same historical semantic slot node and connected to consecutive actual executed actions are aggregated to generate the semantic responsibility confluence core. The action genealogy responsibility migration layer starts from the semantic responsibility confluence core and propagates responsibility markers only along the meta-paths that connect the actually executed action primitives in the forward execution meta-path graph. When the current action primitive has an independent supporting meta-path that starts from the semantic slot node outside the semantic responsibility confluence core, does not pass through the semantic responsibility confluence core, and reaches the current action primitive node, the corresponding responsibility propagation is terminated. The end state checkpoint of the last responsibility action primitive before termination is determined as the responsibility stop checkpoint. The action primitives, propagation order, and responsibility stop checkpoints passed through by the responsibility marker are recorded to generate the responsibility migration result.

[0013] Optionally, the scope of the generated responsibility actions includes: Extract the primitives of responsibility actions, the order of propagation, and the responsibility cessation checkpoints from the results of responsibility migration; Executed actions that have not been marked by the responsibility marker and executed actions whose responsibility propagation is blocked by the independent support element path are written into the non-responsibility action domain. Executed actions that have been continuously marked by the responsibility marker are written into the responsibility migration domain. The responsibility stop checkpoint is determined as the responsibility termination boundary. By associating the non-responsible action domain, the responsibility migration domain, and the responsibility termination boundary, the action responsibility division result is generated.

[0014] Optionally, establishing the interactive action two-state commitment chain and determining the correction boundary includes: Read the responsibility migration domain, action primitive number and status checkpoint number from the action responsibility division results, extract user feedback voice from user voice data, extract the association record of user feedback voice and executed action and interaction round number from historical interaction data, identify the user feedback voice indicating that the action result is correct as user confirmation voice, and generate interaction round switching record based on the change of interaction round number; Actions that are not associated with user confirmation voice and whose status checkpoints are not used as the starting state of a new interaction round are marked as transient actions. Actions that are associated with user confirmation voice and whose status checkpoints are used as the starting state of a new interaction round are marked as committed actions. Write the action primitive number, action status flag, status checkpoint number, interaction round number, and user confirmation voice number of the committed action, and connect them in the order of action execution to generate a bi-state commitment chain of interactive actions; Retrieve committed actions in reverse order from the end of the responsibility migration domain, and determine the end state checkpoint of the committed action closest to the end of the responsibility migration domain as the correction boundary. If there are no committed actions in the responsibility migration domain, determine the start state checkpoint of the starting action of the responsibility migration domain as the correction boundary.

[0015] Optionally, the generation of toy robot interaction results includes: Based on the dual-state commitment chain of interactive actions and the correction boundary, transient actions located after the correction boundary in the responsibility migration domain are identified as actions to be corrected, while actions without responsibility and commitment actions are retained. Extract the current robot motion state, posture state, and actuator state from the robot state data, compare them with the robot state in the state checkpoint corresponding to the correction boundary, generate the recovery control quantity, and restore the robot state. Read the target intent from the anchor data, match the replacement action primitives according to the semantic slots of the target intent and the action triggering relationship in the interaction heterogeneous graph, generate a local replacement action sequence according to the execution range and execution order of the action to be corrected, and execute it. The execution state of the local replacement action sequence is extracted from the robot state data, and the user feedback voice after execution is extracted from the user voice data. Based on the execution state and user feedback voice, new voice evidence, semantic slots, action primitives and state checkpoints are generated and connected edges are established. The interaction heterogeneous graph is written back to generate the toy robot interaction result.

[0016] According to an embodiment of the present invention, a toy robot interaction system based on speech recognition includes the following modules: The interactive data processing module is used to collect robot interaction data and perform preprocessing to generate standard interaction data. The interactive heterogeneous graph construction module is used to construct interactive heterogeneous graphs based on standard interactive data. The proxy verification module is used to generate candidate intents based on the interaction heterogeneous graph and current voice evidence, select a proxy executor to execute the proxy verification action, and generate corrected anchoring data based on user feedback. The responsibility inference module is used to input the interaction heterogeneous graph and the modified anchoring data into the improved GTN model to generate responsibility migration results. The action responsibility allocation module is used to determine the non-responsible action domain, responsibility migration domain, and responsibility termination boundary based on the responsibility migration results, and generate the action responsibility allocation results; The dual-state commitment construction module is used to read the responsibility migration domain from the action responsibility division results, divide transient actions and committed actions, establish a dual-state commitment chain of interactive actions, and determine the correction boundary. The local interaction correction module is used to determine the action to be corrected based on the dual-state commitment chain of the interaction action, the correction boundary, the action responsibility division result, and the correction anchoring data, update the interaction heterogeneous graph, and generate the interaction result of the toy robot.

[0017] The beneficial effects of this invention are: This invention generates standard interaction data by collecting user voice data, robot state data, and historical interaction data. It further extracts voice evidence, semantic slots, action primitives, and state checkpoints to construct an interaction heterogeneous graph. This enables user voice, semantic content, robot actions, and execution states to form a continuous association, improving the toy robot's complete expression of the interaction context and actual action process, and providing a stable data foundation for voice correction and action responsibility analysis.

[0018] When multiple candidate intentions coexist, this invention determines semantic divergence terms and the main task executor set, selects a proxy executor from the remaining available executors, converts candidate slot values ​​into proxy confirmation actions, and obtains user feedback and generates corrected anchoring data without calling the main task executor set. This enables the robot to confirm the user's intention using the entity output state before formally executing the target action, reducing the problem of semantic ambiguity directly being converted into incorrect task actions, and improving the efficiency of interactive confirmation and the accuracy of voice intention recognition.

[0019] This invention utilizes an improved GTN model comprising a modified anchoring bidirectional graph transformation layer, a semantic responsibility confluence kernel layer, and an action spectrum responsibility migration layer. It performs responsibility confluence and migration analysis on historical execution paths and corrective tracing paths, determining the non-responsible action domain, responsibility migration domain, and responsibility termination boundary. Furthermore, it defines the correction boundary by combining transient actions, committed actions, and interactive action bi-state commitment chains. Only actions requiring correction generate local replacement action sequences, ensuring that confirmed actions are preserved. This reduces the probability of local semantic errors causing complete action cancellation and repetitive execution, improving continuous interaction stability, action correction accuracy, and robot state recovery efficiency. Attached Figure Description

[0020] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They explain the invention together with the embodiments of the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is an overall flowchart of a toy robot interaction method based on speech recognition proposed in this invention; Figure 2 This is a schematic diagram of the improved GTN model structure of a toy robot interaction method based on speech recognition proposed in this invention. Figure 3 This is a functional diagram of a toy robot interaction system based on speech recognition proposed in this invention; Detailed Implementation

[0021] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0022] refer to Figure 1 and Figure 2 A toy robot interaction method based on speech recognition, comprising: Collect robot interaction data, perform preprocessing on toy robot data, and generate standard interaction data; Based on standard interactive data, voice evidence, semantic slots, action primitives, and state checkpoints are extracted, and an interaction heterogeneous graph is constructed according to semantics and execution relationships. When a user's voice corresponds to multiple candidate intents, the semantic divergence item is determined, a non-primary task executor is selected to generate a proxy confirmation action, feedback is collected, and corrected anchoring data is generated. The interaction heterogeneous graph and the modified anchoring data are input into the improved GTN model. The responsibility migration results are generated through the modified anchoring bidirectional graph transformation layer, the semantic responsibility confluence kernel layer and the action spectrum responsibility migration layer. Based on the liability migration results, determine the no-liability action domain, liability migration domain, and liability termination boundary, and generate the liability action range; The executed actions within the scope of responsibility actions are divided into transient actions and committed actions. Based on user confirmation and interaction cycle, a dual-state commitment chain of interactive actions is established and the correction boundary is determined. Based on the dual-state commitment chain of interactive actions and the correction boundary, the action to be corrected is determined, the committed actions are retained, the corresponding robot state is restored and a local replacement action sequence is generated, and after execution, the interaction heterogeneous graph is updated to generate the toy robot interaction result.

[0023] In this embodiment, the toy robot data includes user voice data, robot status data, and historical interaction data.

[0024] In this embodiment, the preprocessing of the toy robot data includes speech noise reduction, speech endpoint detection, speech segment recognition, time format unification, alignment of speech time with action execution time, alignment of robot state time, and association of historical interaction rounds.

[0025] In this embodiment, constructing the interactive heterogeneous graph includes: Extract speech recognition text, candidate recognition text, and speech moments from user speech data in standard interaction data, generate speech evidence and perform semantic annotation, and generate object slots, action slots, direction slots, degree slots, and order slots; Extract actuator number, action type, action start and end time, action control quantity and robot state from robot state data in standard interactive data, merge them according to actuator number, action type and continuous execution period to generate action primitives, and generate state checkpoints based on robot state at action start and end time. Extract the interaction round number and the association record of user voice and robot response action from the historical interaction data in the standard interaction data. Map voice evidence, semantic slots, action primitives and state checkpoints as nodes. Based on semantic annotation, association records and action sequence, establish semantic attribution edge, action trigger edge, execution order edge and state arrival edge to construct an interaction heterogeneous graph.

[0026] In this embodiment, the step of collecting feedback and generating corrected anchoring data includes: Read the current speech evidence node and associated semantic slot nodes from the interaction heterogeneous graph. Combine candidate intents according to the candidate slot values ​​corresponding to the same speech evidence. Compare object slots, action slots, direction slots, degree slots, and order slots. Determine the slots with different values ​​as semantic divergences, where: The current voice evidence node and associated semantic slot nodes are read from the interaction heterogeneous graph, specifically as follows: Retrieve the voice evidence node according to the current interaction round number, read the voice evidence node number with the largest voice moment in the current interaction round, retrieve the semantic attribution edge with the voice evidence node number as the source node index, read all semantic slot nodes connected by the semantic attribution edge, extract the slot type, normalize the slot value, candidate recognition text sequence number and node number in the semantic slot nodes respectively, and group the semantic slot nodes belonging to the same candidate recognition text sequence number into the same candidate slot group. Candidate intents are combined based on the candidate slot values ​​corresponding to the same voice evidence, specifically as follows: Candidate slot groups are read in ascending order of candidate recognition text number. For each candidate slot group, the slot values ​​are arranged in a fixed order of object slot, action slot, direction slot, degree slot, and order slot. The position where no slot value is extracted is written as NULL, forming a candidate intent record consisting of 5 slot values. The candidate intent records with 5 slot values ​​are deduplicated, and the records with smaller candidate recognition text numbers are retained. Candidate intent numbers are assigned in ascending order according to the candidate recognition text numbers of the retained records. Slots with different values ​​are identified as semantic divergence terms, specifically: For each of the object slot, action slot, direction slot, degree slot, and order slot, read all slot values ​​from the candidate intents. Perform deduplication on slot values ​​of the same slot type. When only one slot value is retained after deduplication, the corresponding slot is recorded as a consistent slot. When at least two slot values ​​are retained after deduplication, the corresponding slot is recorded as a semantic divergence item. Assign processing order to semantic divergence items according to the order of object slot, action slot, direction slot, degree slot, and order slot, and record the slot type, candidate intent number, and candidate slot value of the semantic divergence item. The actuator ID and available state are extracted from the robot state data. Action primitives and actuator call records are extracted from historical interaction data. The executors associated with the candidate intents and action primitives are grouped into a main task executor set. From the remaining available executors, those capable of mapping semantic divergences to different output states are selected as proxy executors. Where: The executors that associate candidate intents with action primitive calls are grouped into a main task executor set, specifically: Retrieve historical interaction data for records linking voice and robot response actions based on the object slot value, action slot value, direction slot value, degree slot value, and order slot value for each candidate intent. Read the slot combinations and candidate intents. Figure 1 The historical action primitive numbers are used to retrieve the call records of action primitives and executors according to the action primitive numbers. All called executor numbers are extracted, and the executor numbers extracted from different candidate intentions are combined and duplicate numbers are removed to generate the main task executor set. From the remaining available executors, select the executor capable of mapping semantic divergence terms to different output states as the proxy executor, specifically: Read all actuator numbers and available states from the robot state data. Form a set of remaining available actuators by executors whose available state field value is 1 and whose actuator number does not belong to the main task executor set. For each remaining available actuator, read the action primitive call record in the historical interaction data. Combine the action primitives that only call the current remaining available actuator into discrete output states according to action type and action control quantity. For each candidate slot value of the semantic divergence term, the number of times the candidate slot value and each discrete output state co-occurs in the historical interaction record is counted. For each candidate slot value, the discrete output state with the largest number of co-occurrences is selected. When all candidate slot values ​​can obtain discrete output states and the discrete output states obtained by different candidate slot values ​​are different from each other, the remaining available executors are included in the proxy candidate set. The minimum value of the number of co-occurrences of each candidate slot value corresponding to each proxy candidate executor is calculated as the number of mapping support times. Candidate executors are arranged in descending order of the number of times they are supported by mapping. If the number of times they are supported by mapping is the same, they are arranged in ascending order of the average execution time of similar actions in history. If they are still the same, they are arranged in ascending order of executor number. The executor at the top of the list is determined as the proxy executor. If the proxy candidate set is empty, no main task execution instruction is generated and the candidate intent is reserved for further verification. The candidate slot values ​​of semantic divergence terms are mapped to different output states of the proxy executor, generating and executing proxy confirmation actions that do not invoke the main task executor set, where: The candidate slot values ​​of semantic divergence terms are mapped to different output states of the proxy executor, specifically as follows: One semantic divergence item is processed at a time according to the semantic divergence item processing order. The candidate slot value of the current semantic divergence item and the mapping relationship between the candidate slot value and the discrete output state obtained during the agent executor screening process are read. A unique agent output state is recorded for each candidate slot value. The agent output state is jointly represented by the action type and the action control variable. The candidate intent number, semantic divergence item slot type, candidate slot value, agent executor number, and agent output state are written into the confirmation mapping record. When different candidate slot values ​​have the same agent output state, the current agent executor is abandoned and the next executor is selected according to the agent candidate set. Generate and execute proxy confirmation actions that do not invoke the main task executor set, specifically: Read the agent actuator number, action type, and action control quantity corresponding to the current candidate slot value being queried. Read the agent actuator's pre-execution state from the robot state data. Write the agent actuator number and action control quantity into the query control command. Sort the execution duration of actions with the same agent output state in the historical interaction data from smallest to largest and take the median as the current agent output state holding duration. Write a recovery command to restore the agent actuator's pre-execution state at the end of the query control command. After checking that there is no intersection between all the actuator numbers included in the query control command and the main task actuator set, issue the query control command. If there is an intersection, stop issuing the command and reselect the agent actuator. Generate agent query actions in sequence according to the semantic divergence item processing order and candidate intent number. Assign a unique agent query action number to each agent query action and record the start time and end time. Extract user feedback voice and feedback voice time after the proxy confirmation action is executed from user voice data and perform semantic annotation. Based on the confirmation feedback, retain the corresponding candidate intent; based on the negative feedback, remove the confirmed candidate intent; based on the correction feedback, extract the correction slot value and update the candidate slot value corresponding to the semantic divergence item; determine the target intent and anchor semantic slot node, where: Extract the user's feedback voice and the timing of the feedback voice after the proxy confirmation action is executed from the user's voice data, and perform semantic annotation. Specifically: Read the end time of the proxy confirmation action according to the proxy confirmation action number. Select voice segments from the user voice data whose voice start time is later than the end time of the proxy confirmation action and whose interaction round number is consistent with the proxy confirmation action. Select the first voice segment in ascending order of voice start time as the user feedback voice. Write the voice start time as the feedback voice time. Perform the same text normalization and slot recognition as the voice evidence on the user feedback voice. Set the confirmation word set as yes, yes, okay, right, this one. Set the negation word set as no, no, don't, wrong, change one. When the user feedback voice extracts a valid slot value of the current semantic divergence item and the valid slot value is different from the candidate slot value of the confirmation, it is marked as correction feedback. If the correction feedback condition is not met and the negation word set is hit, it is marked as negation feedback. If the correction feedback and negation feedback conditions are not met and the confirmation word set is hit, it is marked as confirmation feedback. User feedback voice that does not meet the three conditions is not written into the anchor record and enters the next proxy confirmation. Based on the confirmed feedback, the corresponding candidate intents will be retained, specifically: Read the candidate intent number recorded when the proxy confirmation action is generated, associate the confirmation feedback with the candidate intent number, retain the object slot value, action slot value, direction slot value, degree slot value and order slot value of the candidate intent, record the candidate intent semantic slot node number associated with the current semantic divergence item as the anchor semantic slot node number, and mark the candidate intent as a confirmation candidate intent; Based on negative feedback, the candidate intent to be verified is eliminated, specifically as follows: Read the candidate intent number recorded when the proxy confirmation action is generated, delete the candidate intent to be confirmed from the current candidate intent set, retain the semantic slot node number of the rejected candidate intent on the current semantic divergence item as the anchor semantic slot node number, when the number of candidate intents after deletion is 1, mark the remaining candidate intents as target candidates, when the number of candidate intents after deletion is greater than 1, select the next candidate intent in ascending order of candidate intent number to continue proxy confirmation, when the number of candidate intents after deletion is 0, terminate the current proxy confirmation process and retain the current user voice as the voice to be re-recognized; Based on the correction feedback, the correction slot values ​​are extracted and the candidate slot values ​​corresponding to the semantic divergence items are updated, specifically as follows: Read the slot type and normalized slot value obtained from the user feedback voice semantic annotation. Determine the normalized slot value whose slot type is consistent with the current semantic divergence item slot type as the corrected slot value. Read the original candidate slot value of the current semantic divergence item in the query candidate intent. Replace the original candidate slot value with the corrected slot value. The other 4 types of slot values ​​remain unchanged to form the corrected 5-slot candidate intent. Record the semantic slot node number where the replaced candidate slot value is located as the anchor semantic slot node number. Determine the target intent and anchor semantic slot nodes, specifically as follows: When a confirmation feedback occurs, the confirmed candidate intent is determined as the target intent. When a negative feedback reduces the number of candidate intents to 1, the remaining candidate intents are determined as the target intent. When a correction feedback occurs, the candidate intent after the slot value replacement is determined as the target intent. If there are multiple semantic divergences and a user feedback only resolves the current semantic divergence, the proxy validation continues to be executed according to the processing order of the semantic divergences until the 5 types of slot values ​​form a unique candidate intent, and then the target intent is determined. The anchor semantic slot node is the semantic slot node of the currently validated semantic divergence in the original candidate intent. Confirmation feedback is anchored to the confirmed slot node, negative feedback is anchored to the negative slot node, and correction feedback is anchored to the slot node before the correction. Write the user feedback voice ID, feedback voice time, feedback semantic type, anchor semantic slot node ID, correction slot value, target intent ID, proxy confirmation action ID, and interaction round ID into the anchoring record to generate corrected anchoring data, where: Write the user feedback voice ID, feedback voice time, feedback semantic type, anchor semantic slot node ID, correction slot value, target intent ID, proxy confirmation action ID, and interaction round ID into the anchoring record, specifically: Anchoring records are established according to a fixed field order: user feedback voice number, feedback voice time, feedback semantic type, anchor semantic slot node number, correction slot value, target intent number, proxy confirmation action number, and interaction round number. The feedback semantic type is written with one text value: confirmation, negation, or correction. If no correction slot value is generated for confirmation or negation feedback, the correction slot value field is written to NULL. When the target intent is not yet determined, the current anchoring record is temporarily stored. After the target intent is determined, the temporarily stored anchoring records of the same interaction round and belonging to the same current voice evidence node are uniformly written to the final target intent number. Generate corrected anchoring data, specifically: All anchor records are sorted in ascending order by interaction round number. Within the same interaction round, they are sorted in ascending order by feedback voice time. If the feedback voice times are the same, they are sorted in ascending order by proxy confirmation action number. The sorted anchor records are combined into corrected anchor data. Each anchor record is connected to the user voice data through the user feedback voice number, connected to the interaction heterogeneous graph through the anchor semantic slot node number, connected to the proxy confirmation action record through the proxy confirmation action number, and connected to the finally determined target intent through the target intent number, forming an anchor data structure that can be read by the corrected anchor bidirectional graph transformation layer.

[0027] In this embodiment, generating the responsibility migration result includes: Retaining the relational adjacency matrix soft selection, meta-path adjacency matrix combination, and graph feature propagation structure of the GTN model, the graph transformation layer of the GTN model is modified into a modified anchored bidirectional graph transformation layer. A semantic responsibility confluence kernel layer and an action lineage responsibility migration layer are sequentially set at the output of the modified anchored bidirectional graph transformation layer to construct an improved GTN model. Specifically, the construction of the improved GTN model is as follows: The improved GTN model uses the node features and relational connection structure of the interaction heterogeneous graph as its basic input. Assuming the interaction heterogeneous graph contains N nodes, the node type, interaction round, time attribute, and node business attribute of each node are encoded into a 64-dimensional node feature vector, arranged in N×64 node feature matrix according to node index order. Semantic attribution edges, action triggering edges, execution order edges, and state arrival edges are each organized into four N×N basic relation adjacency matrices. The anchoring relations formed by the corrected anchoring data are organized separately into one N×N anchoring relation adjacency matrix. The forward execution channel reads... Four basic relation adjacency matrices are taken, and the correction tracing channel reads four basic relation adjacency matrices and one anchor relation adjacency matrix. The model is divided into six processing layers according to the processing order. The first layer is the input representation layer, the second to fourth layers are three-level correction anchor bidirectional graph transformation layers, each of which contains a forward execution GT unit and a correction tracing GT unit. The fifth layer is the semantic responsibility sink kernel layer, and the sixth layer is the action genealogy responsibility transfer layer. Both graph transformation channels use 64-dimensional hidden features. The fifth and sixth layers use a deterministic graph structure for processing, and no neural network parameters are set for training. The trainable parameters of the improved GTN model consist of relation soft selection parameters and node feature propagation parameters. The three-level forward execution channel sets four relation selection parameters for each of the four basic relation classes, for a total of 12 relation selection parameters. The three-level correction and tracing channel sets five relation selection parameters for each of the five relation classes, for a total of 15 relation selection parameters. All relation selection parameters are initialized to 0. After Softmax normalization, the initial weights of each basic relation in the forward execution channel are 0.25, and the initial weights of each relation in the correction and tracing channel are 0.2. Equal weight initialization is used. To avoid artificial bias towards a particular relation type at the beginning of model training, after the two graph transformation channels complete the three-level relation combination, a 64×64 node feature mapping matrix and a 64-dimensional bias vector are set respectively. Feature propagation is performed on the node information on the meta-path structure. The node feature propagation parameters contain a total of 8320 parameters, and the relation selection parameters contain a total of 27 parameters. Under the 64-dimensional hidden feature configuration, the model contains a total of 8347 trainable parameters. The relation adjacency matrix, node index, responsibility flow record, and responsibility migration record are all runtime data and are not included in the model training parameter scale. The improved GTN model's training data is jointly constructed from user speech data, robot state data, and historical interaction data according to interaction rounds. From the user speech data, it reads speech recognition text, semantic slots, and user feedback speech; from the robot state data, it reads the actual action execution state and actuator state; and from the historical interaction data, it reads records of the association between speech and executed actions, interaction round numbers, and records of the association between user feedback and executed actions. Paths that receive user confirmation and whose semantic slots are consistent with the actual executed actions are marked as positive execution samples. Paths that are explicitly rejected by the user or whose slots conflict with the final confirmed target intent are marked as positive execution samples. Sudden semantic action paths are marked as positive execution negative samples. Paths that have negative or corrective feedback and can be located to anchor semantic slots and historically executed actions are marked as corrective tracing positive samples. Historical action paths that conflict with anchor semantic slot values ​​and are rejected by the user are marked as corrective tracing negative samples. Records of executor unavailability, user-terminated interaction, and no actual action are not directly used as negative samples. All samples are divided into training set (first 70%), validation set (middle 15%), and test set (last 15%) according to the interaction time order to avoid future interaction information from the same time series from entering earlier training samples. During training, the path score is calculated by multiplying the weights of the meta-paths generated by the forward execution channel and the corrected tracing channel. The forward execution path loss and the corrected tracing path loss are calculated using binary cross-entropy. The weights of the two losses are both set to 0.5 and then summed to form the total training loss. The two losses are given the same weight because the forward execution relationship learning and the corrected tracing relationship learning are responsible for historical execution modeling and corrected evidence modeling, respectively. The influence ratio of either channel is not increased in advance during the training phase. The Adam optimizer is used to update the relationship selection parameters and node feature propagation parameters. The learning rate is set to 0.001. 32 complete interaction rounds are read in each batch, and the maximum number of training rounds is set to 200 rounds. After each round of training, the total loss of the validation set is calculated. Training is stopped when the total loss of the validation set does not decrease for 10 consecutive rounds, and the model parameters saved in the round with the lowest total loss of the validation set are restored. The semantic responsibility sink kernel layer (layer 5) and the action genealogy responsibility migration layer (layer 6) perform deterministic processing based on the meta-path structure generated by the model and do not participate in gradient updates. Compared to the original GTN, which learns the task-related graph structure through a combination of single relation soft selection and metapath, the improved GTN model expands the original graph transformation process into two independent parameter channels: a forward execution channel and a correction tracing channel. This separates the learning of the relationship between historical semantics and actual executed actions from the learning of the tracing relationship between user correction feedback and historical semantics and actions, reducing the coverage of correction information on the historical true execution relationship representation. A semantic responsibility confluence kernel layer is added at the output of the bidirectional graph transformation structure, further transforming the task-related metapaths output by the original GTN into a responsibility structure that can represent the common attribution of semantic sources and actual actions. An action genealogy responsibility migration layer is added at the output of the semantic responsibility confluence kernel layer, continuing to map semantic layer responsibility to the actual executed action sequence. Layers 5 and 6 use explicit graph structure judgment rules instead of setting up black-box training layers again, enabling the responsibility localization results to be traced along semantic slots, metapaths, and action primitives, providing interpretable model outputs for retaining non-responsible actions, determining the scope of responsible actions, and correcting local actions. Based on the semantic attribution edges, action triggering edges, execution order edges, and state arrival edges in the interaction heterogeneous graph, a corresponding relational adjacency matrix is ​​generated. The user feedback voice ID, anchor semantic slot node ID, and interaction round ID are read from the corrected anchoring data to generate user feedback voice nodes. These user feedback voice nodes are then connected to the anchor semantic slot nodes to form anchoring relationships, and these anchoring relationships are added to the relational adjacency matrix set. Where: Generate a corresponding adjacency matrix based on the semantic attribution edges, action triggering edges, execution order edges, and state arrival edges in the interaction heterogeneous graph, specifically as follows: Read the interaction round number, node type, and node number of all nodes in the interaction heterogeneous graph. First, sort them in ascending order by interaction round number. Within the same interaction round, sort them in the order of voice evidence nodes, semantic slot nodes, action primitive nodes, and state checkpoint nodes. Nodes of the same type are sorted in ascending order by node number. Assign matrix row and column indices to the sorted nodes. For semantic belonging edges, action triggering edges, execution order edges, and state arrival edges, build matrices multiplied by the total number of nodes. Write 1 when there is a corresponding edge between the source node row index and the target node column index, and write 0 when there is no corresponding edge. Generate 4 relational adjacency matrices with the same row and column index rules. The user feedback voice node is generated as follows: Read the user feedback voice number and interaction round number from the corrected anchoring data. Extract the user feedback voice text and feedback voice time from the user voice data according to the user feedback voice number. Use the user feedback voice number as the node number and write the node type as user feedback voice. Write the interaction round number, feedback voice time, and user feedback voice text into the node attributes. Append the user feedback voice node to the end of the corresponding interaction round node sequence. Add 1 row and 1 column to the four generated relation adjacency matrices respectively and initialize the newly added position to 0. The user feedback voice node is connected to the anchor semantic slot node to form an anchoring relationship, specifically: Read the anchor semantic slot node number from the corrected anchoring data, locate the anchor semantic slot node in the interaction heterogeneous graph according to the node number index, verify that the interaction round number of the anchor semantic slot node is consistent with the interaction round number of the user feedback voice node, establish a directed connection in the direction from the user feedback voice node to the anchor semantic slot node, do not establish a connection when the interaction round numbers are inconsistent, and mark the anchoring record as a round inconsistency record. Add the anchoring relationship to the relationship adjacency matrix set, specifically: After establishing and expanding the four relationship adjacency matrices, an anchor relationship adjacency matrix with the same row and column dimensions is created. A 1 is written at the intersection of the row index of the user feedback voice node and the column index of the anchor semantic slot node, and 0 is written at the other positions. The anchor relationship adjacency matrix is ​​added as the fifth type of relationship matrix to the relationship adjacency matrix set of the correction and tracing channel. The forward execution channel still reads the original four types of relationship adjacency matrices. The modified anchoring bidirectional graph transformation layer sets up a forward execution channel and a modified tracing channel. The forward execution channel performs soft selection and weighted combination on the relation adjacency matrix, combining the relation adjacency matrix obtained from the adjacency graph transformation layer by layer and propagating node features to generate a forward execution metapath graph from historical voice evidence nodes through semantic slot nodes to executed action primitive nodes. The modified tracing channel starts with user feedback voice nodes and anchoring relationships, and performs soft selection, layer-by-layer combination, and node feature propagation on the relation adjacency matrix within the corresponding interaction rounds of the modified anchoring data to generate a modified tracing metapath graph from anchoring semantic slot nodes to historical executed action primitive nodes, where: The modified anchored bidirectional graph transformation layer specifically includes: Basic Relationship Reading End: Reads four types of basic relationship adjacency matrices, anchor relationship adjacency matrices, and node feature matrices, and provides them to the forward execution channel and the correction and tracing channel, respectively; Dual-channel soft selection unit: Calculates relation selection weights for the relation adjacency matrices in the forward execution channel and the correction tracing channel respectively, and performs weighted combination to generate two types of weighted relation adjacency matrices; Forward metapath combination unit: Combines the forward execution weighted relation adjacency matrix layer by layer, retains the connection from historical speech evidence nodes through semantic slot nodes to executed action primitive nodes, and generates forward execution metapath; Corrected traceability meta-path combination unit: Taking user feedback voice nodes and anchoring relationships as the traceability entry point, it combines and corrects the traceability weighted relationship adjacency matrix layer by layer to generate a corrected traceability meta-path that reaches the historically executed action primitive node through the anchored semantic slot node; Bidirectional feature propagation unit: Aggregates node features along two types of meta-paths and converts them into 64-dimensional path node representations; Bidirectional graph transformation output: It gathers meta-path connections and path node representations, and outputs a forward execution meta-path graph and a corrected tracing meta-path graph; The forward execution channel performs soft selection and weighted combination on the relation adjacency matrix, specifically as follows: The forward execution channel is set up with 3 levels of graph transformation units. Each level of graph transformation unit sets one trainable relation selection parameter for each of the four types of relation adjacency matrices. The initial values ​​of the four relation selection parameters are all set to 0. After Softmax normalization, the initial selection weights of the four types of relations are all 0.25. During training, records of voice and actual action associations confirmed by the user are extracted from historical interaction data. The relation selection parameters are updated according to the matching error between the confirmed association path and the model-generated path. Equal weight initialization is used to avoid manual bias on semantic belonging edges, action triggering edges, execution order edges or state arrival edges at the beginning of training. Each relation adjacency matrix is ​​multiplied by the corresponding selection weight and then added according to the matrix position to obtain the weighted relation adjacency matrix of the current graph transformation unit. The adjacency matrices obtained from the adjacent graph transformation layer are combined layer by layer and node features are propagated, specifically as follows: The weighted relation adjacency matrix generated by the first-level graph transformation unit is combined with the weighted relation adjacency matrix generated by the second-level graph transformation unit by matrix multiplication. The resulting 2-hop relation matrix is ​​then multiplied with the weighted relation adjacency matrix generated by the third-level graph transformation unit to obtain a 3-hop meta-path adjacency matrix. For each matrix multiplication, the non-zero connection record of the source node number, intermediate node number, target node number, relation type sequence, and selection weights at each level are multiplied to form a meta-path index record. The node type, interaction round, time field, and enumeration fields in the node attributes are one-hot encoded. The numerical fields are normalized to 0 to 1 according to the minimum and maximum values ​​of the fields in the historical interaction data. The original features of various nodes are converted into 64-dimensional node features through linear mapping. The 64-dimensional node features of the target node's adjacent nodes are aggregated according to the meta-path adjacency matrix and updated node features are generated through the ReLU function. Generate a forward execution metapath graph from historical speech evidence nodes through semantic slot nodes to executed action primitive nodes, specifically as follows: The metapath index records are filtered to identify those with source node type of voice evidence node, intermediate node containing semantic slot node, and target node of executed action primitive node. The actual action execution records in the historical interaction data are read to verify that the target action primitive has been executed. Records in which voice evidence node and action primitive node belong to the same interaction round are retained according to the interaction round number. The filtered nodes, metapath connections, relation type sequences, path selection weights and updated node features are combined to form a forward execution metapath graph. The correction and tracing channel starts with user feedback voice nodes and anchor relationships. Within the corresponding interaction rounds of the corrected anchor data, it performs soft selection, layer-by-layer combination, and node feature propagation on the relationship adjacency matrix, specifically as follows: Based on the interaction round number in the corrected anchoring data, a round node index is established. All rows and columns of the relation adjacency matrix that do not belong to the corresponding interaction round are set to 0. The corrected tracing channel is set with a 3-level graph transformation unit. Each level of graph transformation unit sets trainable relation selection parameters for the semantic belonging edge, action trigger edge, execution order edge, state arrival edge, and the 5 relation adjacency matrices corresponding to the anchoring relationship. The initial values ​​of the 5 relation selection parameters are all set to 0. After Softmax normalization, the initial selection weights are all 0.2. During training, the relation selection parameters are updated based on the confirmed user corrected speech and the corrected action association records in the historical interaction data. Equal weight initialization is used to avoid the anchoring relationship suppressing the historical execution relationship at the beginning of the training stage. Matrix multiplication is performed layer by layer on the weighted relation adjacency matrix at each level, and 64-dimensional node features are propagated according to the obtained meta-path adjacency matrix. Generate a modified tracing metapath graph that leads from anchored semantic slot nodes to historically executed action primitive nodes, specifically as follows: From the meta-path index records generated by the correction tracing channel, select records where the source node is the user feedback voice node, the first intermediate node is the anchor semantic slot node of the correction anchor data record, and the target node is the historical executed action primitive node. Verify that the execution time of the target action primitive is earlier than the feedback voice time, delete meta-paths whose execution time does not meet the time order, and combine the retained nodes, anchor relationships, meta-path connections, relationship type sequences, path selection weights, and updated node features to form the correction tracing meta-path graph. The semantic responsibility confluence core layer aligns the forward execution meta-path graph and the correction and tracing meta-path graph according to the interaction round number, semantic slot node number, and action primitive node number. Meta-path combinations that commonly pass through the same historical semantic slot node, commonly reach the same executed action primitive node, and whose action execution time is earlier than the user's feedback voice are written into the responsibility confluence unit. Responsibility confluence units that are associated with the same historical semantic slot node and connect consecutively executed actions are aggregated to generate the semantic responsibility confluence core, where: The semantic responsibility confluence kernel layer specifically includes: Bidirectional metapath reading end: Reads semantic slot nodes, action primitive nodes, and time information from the forward execution metapath graph and the correction and tracing metapath graph; Path alignment unit: Matches two types of metapaths according to interaction round number, semantic slot node number, and action primitive node number; Responsibility Convergence Determination Unit: The path combination that passes through the same historical semantic slot node, reaches the same executed action primitive node, and satisfies that the action occurs before the feedback is determined as the responsibility convergence unit; Continuous responsibility aggregation unit: Responsibility confluence units that connect and associate nodes with the same historical semantic slot according to the execution order, and aggregate the continuous connection parts into a semantic responsibility confluence core; The output of the responsibility merging core: outputs historical semantic slot nodes, responsibility action start and end nodes, and the association structure between responsibility merging units; Align the forward execution metapath graph and the corrected tracing metapath graph according to the interaction round number, semantic slot node number, and action primitive node number, specifically as follows: Read the metapath index records of the two types of metapath graphs respectively, extract the interaction round number, the semantic slot node number passed through, and the termination action primitive node number for each metapath. Concatenate the three numbers in the order of interaction round number, semantic slot node number, and action primitive node number to form a path alignment key. Build a hash index with the path alignment key. Search and correct the records with the same path alignment key in the metapath index one by one in the forward execution metapath index. Do not build a confluence relationship if no matching path alignment key is found. Metapath combinations that share the same historical semantic slot node, reach the same executed action primitive node, and whose action execution time is earlier than the user's voice feedback time are written into the responsibility confluence unit, specifically: For forward execution metapaths and correction trace metapaths with the same path alignment key, read the historical semantic slot node number, action primitive node number, action start time, and user feedback voice time again. When the historical semantic slot node number is consistent, the action primitive node number is consistent, and the action start time is less than the user feedback voice time, establish a responsibility confluence unit, assign a continuous number to the responsibility confluence unit, and write the forward execution metapath number, correction trace metapath number, interaction round number, historical semantic slot node number, and executed action primitive node number. Aggregate responsibility confluence units that are associated with the same historical semantic slot node and connect consecutively executed actions, specifically as follows: The responsibility confluence units are grouped according to the historical semantic slot node number. Within each group, the action primitive nodes are arranged in ascending order according to the start time of the action. The execution order edge in the interaction heterogeneous graph is read. When there is a direct execution order edge between the action primitive node corresponding to the current responsibility confluence unit and the action primitive node corresponding to the next responsibility confluence unit, they are grouped into the same continuous responsibility segment. When there is no direct execution order edge, the current continuous responsibility segment is ended and a new continuous responsibility segment is created. The responsibility confluence unit number, historical semantic slot node number, start action primitive node number and end action primitive node number in the same continuous responsibility segment are written into the semantic responsibility confluence core. When the same historical semantic slot corresponds to multiple discontinuous responsibility segments, multiple semantic responsibility confluence cores are generated respectively. The action genealogy responsibility migration layer starts from the semantic responsibility confluence kernel and propagates responsibility markers only along the meta-paths connecting actually executed action primitives in the forward execution meta-path graph. When an action primitive has an independent supporting meta-path originating from a semantic slot node outside the semantic responsibility confluence kernel and reaching the current action primitive node without passing through the semantic responsibility confluence kernel, the corresponding responsibility propagation terminates. The end-state checkpoint of the last responsibility action primitive before termination is determined as the responsibility stop checkpoint. The action primitives traversed by the responsibility marker, the propagation order, and the responsibility stop checkpoint are recorded, generating the responsibility migration result, where: The action spectrum responsibility migration layer specifically includes: Responsibility Start Point Reader: Reads the start and end nodes of responsibility actions in the semantic responsibility confluence kernel and initializes the action primitives in the semantic responsibility confluence kernel as responsibility actions; Responsibility propagation unit: Propagates responsibility markers backward from the end of the semantic responsibility confluence kernel along the actual execution order in the forward execution metapath graph; Independent support retrieval unit: Taking the current responsibility candidate action primitive as the endpoint, retrieve the meta-path that starts from the semantic slot node outside the semantic responsibility confluence core and does not pass through the semantic responsibility confluence core; Responsibility Termination Determination Unit: If an independent supporting element path exists, stop the corresponding responsibility propagation; if no independent supporting element path exists, write the current action primitive into the responsibility action sequence and continue propagation. Responsibility Stop Location Unit: Reads the end state checkpoint of the last responsibility action primitive before the responsibility propagation stops, and determines the end state checkpoint as the responsibility stop checkpoint; Responsibility migration output: Records the responsibility action primitives, propagation sequence, and responsibility stop checkpoint according to the action execution order, and generates the responsibility migration result; The responsibility marker for metapath propagation only follows the forward execution path graph, connecting the primitives of actually executed actions. Specifically: Read the termination action primitive node recorded in each semantic responsibility confluence core, initialize the action primitive node in the semantic responsibility confluence core to the responsibility mark state, retrieve the next action primitive node from the termination action primitive node along the execution order edge in the forward execution metapath graph, check whether the next action primitive has an actual execution record from the historical interaction data, write the responsibility candidate mark only for action primitives with actual execution records and continue the retrieval, and do not enter the responsibility propagation path for action primitives without actual execution records. When multiple actual execution branches exist, establish propagation branches respectively and record the branch numbers. The corresponding responsibility propagation terminates when the current action primitive has an independent support meta-path that originates from a semantic slot node outside the semantic responsibility confluence core and reaches the current action primitive node without passing through the semantic responsibility confluence core. Specifically: Using the current responsibility candidate action primitive node as the endpoint, retrieve all metapath records in the forward execution metapath graph, delete metapaths that pass through historical semantic slot nodes within the semantic responsibility confluence kernel, retain metapaths whose source semantic slot nodes are in the same interaction round and do not belong to the semantic responsibility confluence kernel, calculate the product of the selection weights of each level of relation in each retained metapath as the path support weight, calculate the median of the support weights of all forward execution metapaths reaching the current action primitive node, and determine the retained metapaths with path support weights not lower than the median as independent support metapaths. If at least one independent support metapath is detected, do not write the current responsibility candidate action into the responsibility action sequence and stop the current propagation branch; if no independent support metapath is detected, write the current responsibility candidate action into the responsibility action sequence and continue propagation. The end-state checkpoint of the last responsible action primitive before termination is determined as the responsibility stop checkpoint, specifically: Read the last responsibility action primitive number according to the execution order of responsibility actions in the propagation branch, retrieve the state checkpoint node connected to the end time of the responsibility action primitive from the state arrival edge in the interaction heterogeneous graph, write the state checkpoint node number as the responsibility stop checkpoint number of the current propagation branch, and when the responsibility propagation reaches the end of the executed action sequence and still no independent support element path is found, determine the end state checkpoint of the last executed responsibility action primitive as the responsibility stop checkpoint. Record the action primitives, propagation sequence, and responsibility stop checkpoints traversed by the responsibility marker, and generate the responsibility migration result, specifically: Responsibility action primitives are collected according to the semantic responsibility confluence core number and propagation branch number. Based on the execution time of the action, they are written into the propagation sequence from early to late. The primitive number, propagation sequence, start state checkpoint number, and end state checkpoint number of each responsibility action are recorded. The responsibility stop checkpoint number is written at the end of the branch. All propagation branches corresponding to the same semantic responsibility confluence core are combined into a responsibility migration record. The responsibility migration records are summarized to generate the responsibility migration result.

[0028] In this embodiment, the scope of generating responsible actions includes: Extract the primitives of responsibility actions, the order of propagation, and the responsibility cessation checkpoints from the results of responsibility migration; Executed actions not traversed by the responsibility marker and executed actions whose responsibility propagation is blocked by independent support element paths are written into the non-responsibility action domain. Executed actions continuously traversed by the responsibility marker are written into the responsibility migration domain. The responsibility stop checkpoint is determined as the responsibility termination boundary, wherein: Executed actions that have not been marked for responsibility and executed actions whose responsibility propagation is blocked by independent supporting element paths are written into the non-responsibility action domain, specifically: Based on the interaction round number in the responsibility migration reading record, all action primitive nodes in the same interaction round are read from the interaction heterogeneous graph. The action primitives are checked against the action execution records in the historical interaction data to see if they have actual start and end times. Only action primitives that have been executed are retained, and a set of executed action primitive numbers is established. The set of executed action primitive numbers is compared item by item with the set of responsibility action primitive numbers. Action primitives that do not exist in the set of responsibility action primitive numbers are first marked as executed actions that have not been marked by responsibility. Then, using the responsibility stop checkpoint time as the time index, executed action primitives with a start time no earlier than the responsibility stop checkpoint time are retrieved from the forward execution metapath graph. The first action primitive that has an execution order connection with the responsibility action primitive is read in ascending order according to the action start time. The forward execution metapath is retrieved with the read action primitive as the endpoint, starting from the semantic slot node outside the semantic responsibility confluence core. Metapaths that pass through the semantic slot node inside the semantic responsibility confluence core are deleted. When at least one metapath that does not pass through the semantic responsibility confluence core and reaches the read action primitive is retained, the read action primitive is marked as an executed action whose responsibility propagation is blocked by an independent supporting metapath. Arrange the two types of executed actions in ascending order according to the start time of the action and remove duplicates according to the action primitive number. Write them into the no-liability action domain and record the action primitive number, action start time, action end time and no-liability reason flag. The no-liability reason flag is written into "no responsibility flag" and "independent support blocking" respectively. Write the executed actions that the responsibility marker passes through consecutively into the responsibility migration domain, specifically: Read the responsible action primitives arranged in the propagation order, and compare the propagation order and execution order connection of two adjacent responsible action primitives in turn. When the propagation order of the current responsible action primitive differs from that of the next responsible action primitive by 1 and there is an execution order edge from the current responsible action primitive to the next responsible action primitive in the interaction heterogeneous graph, the two responsible action primitives are assigned to the same continuous responsibility segment. If any condition is not met, the current continuous responsibility segment is terminated and a new continuous responsibility segment is established from the next responsible action primitive. The action primitive numbers in each continuous responsibility segment are arranged in the propagation order. Read the start state checkpoint of the first action primitive and the end state checkpoint of the last action primitive. Record the continuous responsibility segment number, action primitive number sequence, start state checkpoint number, end state checkpoint number and propagation order range. All continuous responsibility segments are combined into a responsibility migration domain. The liability cessation checkpoint is defined as the liability termination boundary, specifically: Read the responsibility stop checkpoint number from the responsibility migration read record, and read the checkpoint time, robot motion state, posture state and actuator state of the responsibility stop checkpoint through the state checkpoint node index. Write the responsibility stop checkpoint number into the responsibility termination boundary record. If the responsibility migration result contains only one responsibility stop checkpoint, directly use the read responsibility stop checkpoint. If there are multiple responsibility stop checkpoints, arrange them in order of the checkpoint time from early to late and select the responsibility stop checkpoint with the largest checkpoint time as the responsibility termination boundary of the entire interaction round. The remaining responsibility stop checkpoints are retained in each responsibility migration record to identify the stopping position of different responsibility propagation paths. By associating the non-responsible action domain, the responsibility migration domain, and the responsibility termination boundary, the action responsibility division result is generated.

[0029] In this embodiment, establishing the interactive action two-state commitment chain and determining the correction boundary includes: The responsibility migration domain, action primitive number, and status checkpoint number are read from the action responsibility division results. User feedback voice is extracted from user voice data. The association record between user feedback voice and executed actions and the interaction round number are extracted from historical interaction data. User feedback voice indicating that the action result is correct is identified as user confirmation voice. An interaction round switching record is generated based on the change of interaction round number, wherein: Retrieve the responsibility migration domain, action primitive number, and state checkpoint number from the action responsibility allocation results, specifically: Retrieve the action responsibility division results by the current interaction round number, read all continuous responsibility segments in the responsibility migration domain, read the action primitive number sequence arranged according to the propagation order for each continuous responsibility segment, and index the action primitive node from the interaction heterogeneous graph according to the action primitive number to extract the action start state checkpoint number and action end state checkpoint number. Arrange the responsibility action reading table in order from early to late according to the action start time. The responsibility action reading table is saved in the following order: action primitive number, start state checkpoint number, end state checkpoint number, action start time, action end time, and interaction round number. Extracting user feedback voice from user voice data specifically involves: Read the action end time of each executed action in the responsibility action reading table. Filter the voice records from the user voice data whose voice start time is no earlier than the action end time and whose interaction round number matches the interaction round number of the executed action. Then, combine the voice and executed action association records in the historical interaction data to filter out voice records that are not associated with the current action. Write the voice number, voice recognition text, voice start time, and voice end time from the retained voice records into the user feedback voice table. When multiple feedback voices are associated with the same executed action, they are arranged from earliest to latest according to the voice start time. Extract the association records between user feedback voice and executed actions, along with the interaction round number, from historical interaction data. Specifically: Retrieve voice response association records in historical interaction data by user feedback voice ID, read the action primitive ID and interaction round ID from the association record, compare the action primitive ID with the action primitive ID in the responsible action reading table, and retain only the association records with the same ID. When the same user feedback voice is associated with multiple executed actions, establish association items separately and record the user feedback voice ID, action primitive ID, interaction round ID and association time to form a user feedback action association table. The user feedback voice indicating that the action result is correct will be designated as the user confirmation voice. Specifically: Read the speech recognition text corresponding to the user feedback speech, remove spaces and punctuation, and convert it to a unified text format. Set the confirmation word set to "Yes", "That's right", "Correct", "That's it", "Okay", and set the negative word set to "No", "Wrong", "Don't", "Cannot", "Re-". Detect whether the normalized speech recognition text contains words from the confirmation word set and whether it contains words from the negative word set. When at least one confirmation word is hit and no negative word is hit, mark the user feedback speech as user confirmation speech. When both confirmation and negative words are hit, process it as a negative mark and do not write it into the user confirmation speech set. Establish a confirmation association between the user confirmation speech number and the action primitive number in the user feedback action association table. An interaction round switching record is generated based on the change of the interaction round number, specifically as follows: The interaction records in the historical interaction data are uniformly sorted by time according to the start time of voice and the start time of action. The interaction round number of two adjacent interaction records is read. When the interaction round number of the current record is inconsistent with the interaction round number of the next record, it is determined that an interaction round switch has occurred. The original interaction round number, the new interaction round number, and the start time of the first user voice in the new interaction round are recorded. Then, the status checkpoint nodes are selected from the status checkpoint nodes. The checkpoint time is not later than the start time of the first user voice in the new interaction round and later than the start time of the last action in the original interaction round. The checkpoints are arranged from late to early and the first status checkpoint is selected as the starting status checkpoint of the new interaction round. The status checkpoint number is written into the interaction round switch record. Actions that are not associated with user confirmation voice and whose status checkpoints are not used as the starting state of a new interaction round are marked as transient actions. Actions that are associated with user confirmation voice and whose status checkpoints are used as the starting state of a new interaction round are marked as committed actions. Write the action primitive number, action state flag, state checkpoint number, interaction round number, and user confirmation voice number of the committed action, and connect them in the order of action execution to generate a two-state commitment chain of interactive actions, where: Write the action primitive number, action status flag, status checkpoint number, interaction round number, and user confirmation voice number for the committed action, specifically: For each executed action in the responsibility migration domain, establish one bistate action record. Write the record in the order of action primitive number, action status flag, start state checkpoint number, end state checkpoint number, interaction round number, and user confirmation voice number. Write NULL to the user confirmation voice number field for transient actions. Write NULL to the user confirmation voice number field for commitment actions formed only by the start state condition of a new interaction round. Read all associated user confirmation voice numbers for commitment actions formed by user confirmation voices and save them in ascending order of voice start time. The interactive action two-state commitment chain is generated by connecting the actions in the order of action execution, specifically as follows: For each continuous responsibility migration domain, all bistate action records are read and arranged from earliest to latest according to the start time of the action. If the start times of the actions are the same, they are arranged from smallest to largest according to the action primitive number. The first record in the sorting is set as the chain head node. The end state checkpoint number of the current bistate action record is sequentially connected with the start state checkpoint number of the next bistate action record. The next action primitive number is written into the current record and the previous action primitive number is written into the next record. The connection is made continuously according to the sorting order until the end of the responsibility migration domain, forming an interactive action bistate commitment chain that simultaneously saves the transient action and commitment action states. Retrieve committed actions in reverse order from the end of the responsibility migration domain, and determine the end state checkpoint of the committed action closest to the end of the responsibility migration domain as the correction boundary. If there are no committed actions in the responsibility migration domain, determine the start state checkpoint of the starting action of the responsibility migration domain as the correction boundary.

[0030] In this embodiment, generating the toy robot interaction result includes: Based on the dual-state commitment chain of interactive actions and the correction boundary, transient actions located after the correction boundary in the responsibility migration domain are identified as actions to be corrected, while actions without responsibility and actions with commitment are retained, wherein: Based on the dual-state commitment chain of interactive actions and the correction boundary, transient actions located after the correction boundary in the responsibility migration domain are identified as actions to be corrected, specifically: Read the responsibility migration domain and the bi-state commitment chain of the interaction action according to the interaction round number, read the state checkpoint number and checkpoint time in the correction boundary record, use the state checkpoint time as the time boundary value, traverse all action primitives in the responsibility migration domain in ascending order according to the action start time, filter the action primitives whose action start time is not earlier than the correction boundary checkpoint time and whose action state is marked as transient action, arrange the filtered action primitives according to the original execution order, record the action primitive number, execution order number, action start time, action end time, start state checkpoint number, end state checkpoint number, actuator number and action control quantity, and generate the action record to be corrected; Retaining actions without liability and actions with commitment, specifically: Read all action primitive numbers in the non-responsible action domain from the action responsibility division results. Read all action primitive numbers marked as committed actions from the interactive action dual-state commitment chain. Merge the two types of action primitive numbers and remove duplicates according to the action primitive numbers to form a set of retained actions. Read the executor number called by each retained action, as well as the state before and after the action execution from the action execution record in the historical interaction data. Write the robot state field whose values ​​change between the state before and after execution into the set of retained state components. Write the state components generated by retained actions that occur after boundary correction into the set of retained state components first. Do not perform backtracking control on the retained state components when restoring the robot state and generating local replacement action sequences. Extract the current robot motion state, attitude state, and actuator state from the robot state data, compare them with the robot state in the state checkpoints corresponding to the correction boundary, generate recovery control variables, and restore the robot state, where: Extract the current robot motion state, posture state, and actuator state from the robot state data, specifically as follows: Using the start time of local correction as the reading time, the robot state data is read from the robot state data with the timestamp no later than the start time of local correction and the smallest time difference. The chassis position x-coordinate, chassis position y-coordinate, linear velocity and angular velocity are read from the motion state field. The yaw angle, pitch angle and roll angle are read from the attitude state field. The actuator current position, current velocity and on / off status are read from the actuator state field in ascending order of actuator number. The position coordinates are uniformly converted to millimeters, the linear velocity is uniformly converted to millimeters per second, the attitude angle is uniformly converted to degrees, and the angular velocity is uniformly converted to degrees per second. The current robot state vector is formed by arranging the chassis position x-coordinate, chassis position y-coordinate, linear velocity, angular velocity, yaw angle, pitch angle, roll angle and the state fields corresponding to each actuator number in a fixed order. The robot state is compared with the state checkpoint corresponding to the corrected boundary, specifically: Retrieve the state checkpoint node in the interactive heterogeneous graph according to the state checkpoint number in the corrected boundary record, read the chassis position, motion speed, attitude angle and each actuator state saved by the state checkpoint, arrange them in the order of the same fields in the current robot state vector to form the boundary robot state vector, subtract the components with the same field name and actuator number in the two state vectors one by one, and obtain the state recovery difference by subtracting the current robot state component from the boundary robot state component. Normalize the difference of the angle component to the range of -180 degrees to 180 degrees. Read the position sensor resolution, attitude sensor angle resolution, and minimum control step size of each actuator from the robot state data. Set the position allowable error to twice the position sensor resolution, the attitude allowable error to twice the attitude sensor angle resolution, and the actuator allowable error to the corresponding minimum control step size. State components whose absolute difference does not exceed the corresponding allowable error are recorded as components that do not need to be recovered. State components that belong to the set of retained state components are recorded as components that cannot be recovered. The remaining state components are recorded as components that need to be recovered. Generate recovery control variables and restore the robot state, specifically as follows: Read the state recovery difference of the component to be recovered item by item, extract the minimum control step size and the maximum allowable control quantity of the corresponding actuator from the robot state data, divide the absolute value of the state recovery difference by the maximum allowable control quantity of the single operation and round up to obtain the required number of control operations, distribute the state recovery difference equally according to the required number of control operations to form continuous recovery control quantities, and use the remaining unrecovered difference for the last control quantity to eliminate the margin generated by the equal distribution. The chassis recovery command is generated in the following order: first adjust the heading angle, then adjust the horizontal position, and then correct the heading angle. Recovery commands are generated in ascending order of actuator number for servo positions and actuators whose target values ​​can be directly set. For switch-type actuators, the target switch state is written into the correction boundary record only when the corresponding state component belongs to the component to be recovered. The recovery command is issued one by one and the robot status is reread after each command is executed. The recovery process ends when all the components to be recovered enter the corresponding allowable error range. If any control quantity exceeds the actuator's allowable control range, the control quantity is limited to the maximum allowable control quantity in a single operation and the number of control operations is increased. When the change in the state component is less than the corresponding sensor resolution after three consecutive executions, the recovery of the current state component is stopped and a recovery anomaly flag is written. The target intent is read from the anchor data for correction. Replacement action primitives are matched based on the semantic slots of the target intent and the action triggering relationships in the interaction heterogeneity graph. A local replacement action sequence is generated and executed according to the execution range and order of the action to be corrected. The target intent is read from the corrected anchoring data, specifically: Retrieve and correct anchoring data according to the current interaction round number, read the target intent number, obtain the object slot value, action slot value, direction slot value, degree slot value and order slot value with the target intent number, arrange the 5 types of slots in a fixed order of object slot, action slot, direction slot, degree slot and order slot, write NULL to slots that do not contain content, and form the target intent slot set by combining the slot type and slot value of non-NULL slots; Based on the semantic slots of the target intent and the action triggering relationships in the interaction heterogeneity graph, the action primitives are matched and replaced, specifically: Read the slot type and slot value in the target intent slot set one by one. Search for historical semantic slot nodes with completely consistent slot type and slot value in the interaction heterogeneous graph. Read the connected action primitive nodes along the action trigger edge from the detected historical semantic slot nodes. According to the interaction round number, form candidate action primitive groups by the action primitive nodes belonging to the same historical interaction round. Count the number of slots that are completely consistent with the non-NULL slots of the target intent for each candidate action primitive group. Check whether there are semantic slot nodes with different values ​​for the same slot type. Delete the candidate action primitive groups with different values ​​and keep the candidate action primitive groups with the same number of slots equal to the number of non-NULL slots of the target intent. When multiple candidate action primitive groups are retained, the number of times the user confirmation voice is associated after the execution of each action group is counted from the historical interaction data. The groups are sorted from largest to smallest number of user confirmations. If the number of user confirmations is the same, they are sorted from smallest to largest total execution time of the actions within the group. If they are still the same, they are sorted from smallest to largest number of the first action primitive number within the group. The action primitive in the first candidate action primitive group is selected as the replacement action primitive. If no candidate action primitive group covering all non-NULL slots of the target intent is found, the local replacement execution is stopped and the target intent number is written to the record to be replanned. The execution state of the local replacement action sequence is extracted from the robot state data, and the user feedback voice after execution is extracted from the user voice data. Based on the execution state and user feedback voice, new voice evidence, semantic slots, action primitives and state checkpoints are generated and connected edges are established. The interaction heterogeneous graph is written back to generate the toy robot interaction result.

[0031] refer to Figure 3 A toy robot interaction system based on speech recognition includes the following modules: The interactive data processing module is used to collect robot interaction data and perform preprocessing to generate standard interaction data. The interactive heterogeneous graph construction module is used to construct interactive heterogeneous graphs based on standard interactive data. The proxy verification module is used to generate candidate intents based on the interaction heterogeneous graph and current voice evidence, select a proxy executor to execute the proxy verification action, and generate corrected anchoring data based on user feedback. The responsibility inference module is used to input the interaction heterogeneous graph and the modified anchoring data into the improved GTN model to generate responsibility migration results. The action responsibility allocation module is used to determine the non-responsible action domain, responsibility migration domain, and responsibility termination boundary based on the responsibility migration results, and generate the action responsibility allocation results; The dual-state commitment construction module is used to read the responsibility migration domain from the action responsibility division results, divide transient actions and committed actions, establish a dual-state commitment chain of interactive actions, and determine the correction boundary. The local interaction correction module is used to determine the action to be corrected based on the dual-state commitment chain of the interaction action, the correction boundary, the action responsibility division result, and the correction anchoring data, update the interaction heterogeneous graph, and generate the interaction result of the toy robot.

[0032] Example 1: In an indoor continuous interactive test scenario, a toy robot equipped with a wheeled chassis, head servo motor, dual-arm servo motor, indicator lights, and voice playback actuator was used for verification. The robot received voice commands to complete forward movement, turning, arm swinging, head nodding, and light feedback. A total of 120 continuous interactive rounds were set, including 42 single intention rounds, 51 double candidate intention rounds, and 27 triple candidate intention rounds. 286 voice segments were collected.

[0033] The robot reads user voice data, robot state data, and historical interaction data, performs noise reduction, endpoint detection, and text recognition on the voice, unifies the voice time, action start and end time, and robot state sampling time to a millisecond-level time axis, and constructs an interaction heterogeneous graph based on semantic attribution, action triggering, execution order, and state arrival relationship.

[0034] In a multi-candidate interaction, the user's voice generated two candidate intentions: "move a little to the left" and "move a little to the right." The object slot for both was "robot," the action slot for both was "move," the degree slot for both was "a little," and the direction slots for both were "left" and "right." The system identified the direction slots as the semantic divergence item. Both candidate intentions invoked the left and right wheel motors. The system combined the two wheeled drive actuators into a main task actuator set and selected a proxy actuator from the remaining actuators. In the historical records, the head servo's 20-degree leftward deflection and the direction slot "left" received 46 confirmations, and the 20-degree rightward deflection and the direction slot "right" received 44 confirmations. The system selected the head servo as the proxy actuator.

[0035] The system controls the head servo to deflect 20 degrees to the left and hold for 0.6 seconds before returning to the 0-degree position. The left and right wheel motors do not receive motion control commands. The target displacement of the chassis is 0 mm. The encoder records the actual position change as 0.8 mm, which is lower than the 2 mm threshold for judging the main task status change. The feedback voice recognition is "No, to the right". The system marks it as correction feedback, extracts the direction slot value "right", determines the target intention as "move a little to the right", and generates corrected anchoring data.

[0036] The GTN model is improved by interactive heterogeneous graphs and modified anchoring data inputs. In this round, 18 forward execution meta-paths and 7 modified tracing meta-paths are obtained, forming 4 responsibility confluence units and aggregating them into 1 semantic responsibility confluence kernel. At the 5th action primitive, 1 independent supporting meta-path is detected, resulting in 4 responsibility action primitives and 1 responsibility stop checkpoint.

[0037] After the second responsibility action in the responsibility migration domain is completed, the associated user confirmation voice is used. The second action is marked as a commitment action, and the third and fourth actions are marked as transient actions. The system retrieves the second commitment action in reverse order, determines the end state checkpoint as the correction boundary, and determines the third and fourth actions as actions to be corrected.

[0038] During the recovery phase, the system detected a lateral position deviation of 34 mm, a heading angle deviation of 6.2 degrees, and a head servo position deviation of 0.8 degrees. The head servo angle feedback resolution was 1 degree, and the recovery allowable error was set to 2 degrees. The head servo did not require recovery. The system only generated recovery control quantities for the chassis state affected by the correction actions. After recovery, the position deviation was 3 mm and the heading angle deviation was 0.7 degrees. Then, based on the target intent, two local replacement actions were generated: a right turn of 8 degrees and a forward movement of 180 mm. Committed actions and no-fault actions were not executed repeatedly.

[0039] The model training used 4800 interaction rounds, generating a total of 12380 supervised path records, specifically including 6840 positive execution samples, 2380 negative execution samples, 1820 corrective tracing positive samples, and 1340 corrective tracing negative samples. Positive execution samples recorded the direction slot "left", the action primitive "turn left and move forward", and the user confirmation flag. Negative execution samples recorded the direction slot "right", the action primitive "turn left and move forward", and the user negation flag. Corrective tracing positive samples recorded the original direction slot "left", the feedback slot "right", the corrected action primitive, and the anchor semantic slot node number. The training set, validation set, and test set contained 3360, 720, and 720 interaction rounds, respectively. The Adam optimizer was used with a learning rate of 0.001, a batch size of 32, and a maximum training round count of 200 rounds. Training stopped when the validation set loss did not decrease for 10 consecutive rounds.

[0040] After the model training is completed, another 120 consecutive interaction rounds that did not participate in training, verification and testing are collected for closed-loop verification. The method of this invention uses the same 120 initial user voices and the same manually labeled target intents as the traditional method of directly executing the highest confidence intent and replanning after receiving correction feedback. After the robot generates different responses, it inputs confirmation, denial or correction feedback voices that match the response state.

[0041] In 78 rounds with multiple candidate intents, the traditional method resulted in 19 errors in main task execution, with an error rate of 24.4%, while the present invention resulted in 4 errors, with an error rate of 5.1%. In 120 verification rounds, the traditional method ultimately completed 109 rounds correctly, with a final intent execution accuracy rate of 90.8%, while the present invention completed 115 rounds correctly, with a final intent execution accuracy rate of 95.8%. Based on the statistics of 116 windows formed by five consecutive rounds of sliding windows, the traditional method had 83 windows that were all correct, with a completion rate of 71.6%, while the present invention had 105 windows that were all correct, with a completion rate of 90.5%.

[0042] In 36 correction rounds, the traditional method rolled back an average of 3.8 action primitives, while the present invention rolled back an average of 1.4 action primitives. The traditional method regenerated an average of 5.2 action primitives, while the present invention generated an average of 2.1 local replacement action primitives. Of the 96 confirmed actions, the traditional method repeated 18 (18.8%), while the present invention repeated 3 (3.1%). Using the corrected target state checkpoint record value as the error benchmark, the traditional method had an average position error of 21.6 mm and an average attitude angle error of 4.8 degrees, while the present invention had an average position error of 6.4 mm and an average attitude angle error of 1.1 degrees. The average correction time decreased from 2.84 seconds to 1.67 seconds. The 36 correction rounds involved a total of 268 executed action primitives, and 170 non-responsible or committed actions were retained, accounting for 63.4%.

[0043] Test results show that the present invention can complete semantic verification using a non-master task executor, locate semantic responsibility and the scope of affected actions by improving the GTN model, determine the correction boundary by using the interactive action bi-state commitment chain, and only restore and replace the actions that need to be corrected. Compared with the traditional complete replanning method, it reduces the execution of erroneous master tasks, action rollback, repetition of confirmed actions and correction time, improves the final intention execution accuracy and continuous interaction completion rate, and verifies the feasibility and effectiveness of the present invention.

[0044] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A toy robot interaction method based on speech recognition, characterized in that, include: Collect robot interaction data, perform preprocessing on toy robot data, and generate standard interaction data; Based on standard interactive data, voice evidence, semantic slots, action primitives, and state checkpoints are extracted, and an interaction heterogeneous graph is constructed according to semantics and execution relationships. When a user's voice corresponds to multiple candidate intents, the semantic divergence item is determined, a non-primary task executor is selected to generate a proxy confirmation action, feedback is collected, and corrected anchoring data is generated. The interaction heterogeneous graph and the modified anchoring data are input into the improved GTN model. The responsibility migration results are generated through the modified anchoring bidirectional graph transformation layer, the semantic responsibility confluence kernel layer and the action spectrum responsibility migration layer. Based on the liability migration results, determine the no-liability action domain, liability migration domain, and liability termination boundary, and generate the liability action range; The executed actions within the scope of responsibility actions are divided into transient actions and committed actions. Based on user confirmation and interaction cycle, a dual-state commitment chain of interactive actions is established and the correction boundary is determined. Based on the dual-state commitment chain of interactive actions and the correction boundary, the action to be corrected is determined, the committed actions are retained, the corresponding robot state is restored and a local replacement action sequence is generated, and after execution, the interaction heterogeneous graph is updated to generate the toy robot interaction result.

2. The toy robot interaction method based on speech recognition according to claim 1, characterized in that, The toy robot data includes user voice data, robot status data, and historical interaction data.

3. The toy robot interaction method based on speech recognition according to claim 1, characterized in that, The preprocessing of the toy robot data includes speech denoising, speech endpoint detection, speech segment recognition, time format unification, alignment of speech time with action execution time, alignment of robot state time, and association of historical interaction rounds.

4. The toy robot interaction method based on speech recognition according to claim 1, characterized in that, The construction of the interactive heterogeneous graph includes: Extract speech recognition text, candidate recognition text, and speech moments from user speech data in standard interaction data, generate speech evidence and perform semantic annotation, and generate object slots, action slots, direction slots, degree slots, and order slots; Extract actuator number, action type, action start and end time, action control quantity and robot state from robot state data in standard interactive data, merge them according to actuator number, action type and continuous execution period to generate action primitives, and generate state checkpoints based on robot state at action start and end time. Extract the interaction round number and the association record of user voice and robot response action from the historical interaction data in the standard interaction data. Map voice evidence, semantic slots, action primitives and state checkpoints as nodes. Based on semantic annotation, association records and action sequence, establish semantic attribution edge, action trigger edge, execution order edge and state arrival edge to construct an interaction heterogeneous graph.

5. The toy robot interaction method based on speech recognition according to claim 1, characterized in that, The process of collecting feedback and generating corrected anchoring data includes: Read the current voice evidence node and associated semantic slot node from the interaction heterogeneous graph, combine candidate intents according to the candidate slot values ​​corresponding to the same voice evidence, compare object slots, action slots, direction slots, degree slots and order slots, and determine the slots with different values ​​as semantic divergence items; Extract the actuator number and available status from the robot state data, extract the action primitives and actuator call records from the historical interaction data, form the main task executor set by associating the executors with the action primitives of the candidate intents, and select the executors that can map semantic divergence items to different output states from the remaining available executors as proxy executors. The candidate slot values ​​of semantic divergence terms are mapped to different output states of the proxy executor, generating and executing proxy confirmation actions that do not invoke the main task executor set. Extract user feedback voice and feedback voice time after the proxy confirmation action is executed from user voice data and perform semantic annotation. Based on the confirmation feedback, retain the corresponding candidate intent, remove the confirmed candidate intent based on the negative feedback, extract the correction slot value based on the correction feedback and update the candidate slot value corresponding to the semantic divergence item, and determine the target intent and anchor semantic slot node. Write the user feedback voice ID, feedback voice time, feedback semantic type, anchor semantic slot node ID, correction slot value, target intent ID, proxy confirmation action ID, and interaction round ID into the anchoring record to generate corrected anchoring data.

6. The toy robot interaction method based on speech recognition according to claim 1, characterized in that, The generated responsibility migration results include: The relation adjacency matrix soft selection, meta-path adjacency matrix combination, and graph feature propagation structure of the GTN model are retained. The graph transformation layer of the GTN model is transformed into a modified anchored bidirectional graph transformation layer. A semantic responsibility confluence kernel layer and an action genealogy responsibility migration layer are set sequentially at the output of the modified anchored bidirectional graph transformation layer to construct an improved GTN model. Generate a corresponding relational adjacency matrix based on the semantic belonging edge, action trigger edge, execution order edge and state arrival edge in the interaction heterogeneous graph. Read the user feedback voice number, anchor semantic slot node number and interaction round number from the corrected anchoring data to generate user feedback voice nodes. Connect the user feedback voice nodes to the anchor semantic slot nodes to form anchoring relationships and add the anchoring relationships to the relational adjacency matrix set. The modified anchoring bidirectional graph transformation layer sets up a forward execution channel and a modified tracing channel. The forward execution channel performs soft selection and weighted combination on the relation adjacency matrix, combines the relation adjacency matrix obtained from the adjacency graph transformation layer by layer and propagates node features to generate a forward execution metapath graph from historical voice evidence nodes through semantic slot nodes to executed action primitive nodes. The modified tracing channel takes the user feedback voice node and anchoring relationship as the starting point, performs soft selection, layer-by-layer combination and node feature propagation on the relation adjacency matrix within the corresponding interaction round of the modified anchoring data, and generates a modified tracing metapath graph from anchoring semantic slot nodes to historical executed action primitive nodes. The semantic responsibility confluence core layer aligns the forward execution metapath graph and the correction and tracing metapath graph according to the interaction round number, semantic slot node number and action primitive node number. Metapaths that pass through the same historical semantic slot node, reach the same executed action primitive node and whose action execution time is earlier than the user feedback voice time are written into the responsibility confluence unit. Responsibility confluence units that are associated with the same historical semantic slot node and connected to consecutive actual executed actions are aggregated to generate the semantic responsibility confluence core. The action genealogy responsibility migration layer starts from the semantic responsibility confluence core and propagates responsibility markers only along the meta-paths that connect the actually executed action primitives in the forward execution meta-path graph. When the current action primitive has an independent supporting meta-path that starts from the semantic slot node outside the semantic responsibility confluence core, does not pass through the semantic responsibility confluence core, and reaches the current action primitive node, the corresponding responsibility propagation is terminated. The end state checkpoint of the last responsibility action primitive before termination is determined as the responsibility stop checkpoint. The action primitives, propagation order, and responsibility stop checkpoints passed through by the responsibility marker are recorded to generate the responsibility migration result.

7. The toy robot interaction method based on speech recognition according to claim 1, characterized in that, The scope of the generated responsibility actions includes: Extract the primitives of responsibility actions, the order of propagation, and the responsibility cessation checkpoints from the results of responsibility migration; Executed actions that have not been marked by the responsibility marker and executed actions whose responsibility propagation is blocked by the independent support element path are written into the non-responsibility action domain. Executed actions that have been continuously marked by the responsibility marker are written into the responsibility migration domain. The responsibility stop checkpoint is determined as the responsibility termination boundary. By associating the non-responsible action domain, the responsibility migration domain, and the responsibility termination boundary, the action responsibility division result is generated.

8. The toy robot interaction method based on speech recognition according to claim 1, characterized in that, The establishment of the interactive action two-state commitment chain and the determination of the correction boundary include: Read the responsibility migration domain, action primitive number and status checkpoint number from the action responsibility division results, extract user feedback voice from user voice data, extract the association record of user feedback voice and executed action and interaction round number from historical interaction data, identify the user feedback voice indicating that the action result is correct as user confirmation voice, and generate interaction round switching record based on the change of interaction round number; Actions that are not associated with user confirmation voice and whose status checkpoints are not used as the starting state of a new interaction round are marked as transient actions. Actions that are associated with user confirmation voice and whose status checkpoints are used as the starting state of a new interaction round are marked as committed actions. Write the action primitive number, action status flag, status checkpoint number, interaction round number, and user confirmation voice number of the committed action, and connect them in the order of action execution to generate a bi-state commitment chain of interactive actions; Retrieve committed actions in reverse order from the end of the responsibility migration domain, and determine the end state checkpoint of the committed action closest to the end of the responsibility migration domain as the correction boundary. If there are no committed actions in the responsibility migration domain, determine the start state checkpoint of the starting action of the responsibility migration domain as the correction boundary.

9. The toy robot interaction method based on speech recognition according to claim 1, characterized in that, The generated toy robot interaction results include: Based on the dual-state commitment chain of interactive actions and the correction boundary, transient actions located after the correction boundary in the responsibility migration domain are identified as actions to be corrected, while actions without responsibility and commitment actions are retained. Extract the current robot motion state, posture state, and actuator state from the robot state data, compare them with the robot state in the state checkpoint corresponding to the correction boundary, generate the recovery control quantity, and restore the robot state. Read the target intent from the anchor data, match the replacement action primitives according to the semantic slots of the target intent and the action triggering relationship in the interaction heterogeneous graph, generate a local replacement action sequence according to the execution range and execution order of the action to be corrected, and execute it. The execution state of the local replacement action sequence is extracted from the robot state data, and the user feedback voice after execution is extracted from the user voice data. Based on the execution state and user feedback voice, new voice evidence, semantic slots, action primitives and state checkpoints are generated and connected edges are established. The interaction heterogeneous graph is written back to generate the toy robot interaction result.

10. A toy robot interaction system based on speech recognition, comprising executing the toy robot interaction method based on speech recognition as described in any one of claims 1 to 9, characterized in that, Includes the following modules: The interactive data processing module is used to collect robot interaction data and perform preprocessing to generate standard interaction data. The interactive heterogeneous graph construction module is used to construct interactive heterogeneous graphs based on standard interactive data. The proxy verification module is used to generate candidate intents based on the interaction heterogeneous graph and current voice evidence, select a proxy executor to execute the proxy verification action, and generate corrected anchoring data based on user feedback. The responsibility inference module is used to input the interaction heterogeneous graph and the modified anchoring data into the improved GTN model to generate responsibility migration results. The action responsibility allocation module is used to determine the non-responsible action domain, responsibility migration domain, and responsibility termination boundary based on the responsibility migration results, and generate the action responsibility allocation results; The dual-state commitment construction module is used to read the responsibility migration domain from the action responsibility division results, divide transient actions and committed actions, establish a dual-state commitment chain of interactive actions, and determine the correction boundary. The local interaction correction module is used to determine the action to be corrected based on the dual-state commitment chain of the interaction action, the correction boundary, the action responsibility division result, and the correction anchoring data, update the interaction heterogeneous graph, and generate the interaction result of the toy robot.

Images