A language-perception-behavior consistency evaluation method and system for embodied intelligent robots

Abstract

The application discloses a language-perception-behavior consistency evaluation method and system for embodied intelligent robots, and relates to the field of robot evaluation.The method comprises the following steps: S1, obtaining task instructions, scene perception, language response and behavior execution results under the same task instance; S2, adding a uniform time base timestamp and entering the same evaluation unit; S3, extracting structured language semantics, scenes and behavior representations; S4, aligning to a uniform object number; S5 to S7, calculating cross-modal consistency sub-scores between language, perception and behavior; S8, weighted summation to obtain a comprehensive consistency score; S9, comparing with a threshold to determine semantic, action, perception or timing deviation; and S10, outputting an evaluation level and a correction control signal of re-execution, stop or manual intervention.The application quantitatively identifies the inconsistency between language, perception and action execution, and improves the repeatability and interpretability of robot testing, training and acceptance.

Description

In the field of rights technology

[0001] This invention relates to the fields of artificial intelligence, robot evaluation, multimodal interaction and embodied intelligence, and in particular to a method and system for evaluating the consistency of language-perception-behavior in embodied intelligent robots. Background Technology

[0002] Embossed intelligent robots typically need to receive verbal commands from users in real or semi-real environments, perceive their surroundings, and complete tasks through behaviors such as movement, grasping, placement, obstacle avoidance, and dialogue. With the development of large language models, multimodal models, and robot control technology, robots are now able to complete complex tasks to a certain extent based on natural language commands.

[0003] In existing technologies, the evaluation of embodied intelligent robots often focuses on single indicators such as task completion rate, trajectory error, grasping success rate, number of collisions, and response latency; some technologies also use multimodal interaction to enable robots to generate responses or control actions based on information such as voice, image, posture, and touch.

[0004] However, it is difficult to determine whether a robot maintains consistency between language understanding, environmental perception, and action execution based solely on task completion or a single interaction result. For example, a robot might correctly repeat the task objective verbally but grasp the wrong object; it might complete the action but misinterpret its own behavior; and it might gradually deviate from the original task objective during scene changes or multiple rounds of interaction. Existing multimodal interaction solutions focus on how to generate interactions or control actions, without simultaneously measuring the consistency among language, perception, and behavior. Existing consistency detection solutions mostly target the temporal trajectory or sensor layer of industrial robots, failing to enter the closed-loop interaction of language semantics, scene perception, and action execution.

[0005] Therefore, there is a need for a technical solution that can simultaneously collect language, perception, and behavioral information under the same task instance, perform quantifiable and interpretable evaluation of the consistency among the three, and make corrective control outputs to the robot accordingly. Summary of the Invention

[0006] This invention aims to provide a language-perception-behavior consistency evaluation method and system for embodied intelligent robots. It solves the problems of existing evaluation methods, which have difficulty identifying inconsistencies between language understanding, scene perception, and action execution, and often have to retrace indefinitely to deal with identified mismatches, which leads to a sharp increase in computing power consumption. The evaluation results are transformed into closed-loop correction control that feeds back to the robot's motion control module. When the invalid correction reaches a preset limit, the system will upgrade and intervene, thereby improving the safety and reliability of the robot in real task execution and reducing the computational energy consumption of invalid correction. To achieve the above objectives, the technical solution of the present invention is as follows: Firstly, the present invention provides a method for evaluating the consistency of language-perception-behavior in embodied intelligent robots, comprising: acquiring task instructions generated by the embodied intelligent robot during the execution of the same task instance, scene perception results output by the perception module, language response results output by the language generation module, and behavior execution results output by the motion control module; timestamping the above four types of data according to a unified time base, and grouping them into the same evaluation unit based on task identifier, task start time, and task end time; extracting structured language semantic representation, structured scene representation, and structured behavior representation from the language response results, scene perception results, and behavior execution results, respectively, and assigning the three to the same object according to a preset object mapping rule. The identifier of a physical object is aligned to a unified object number in the same object coordinate system; fields with the same unified object number are compared item by item, and scores for three sub-items—language-perception consistency, language-behavior consistency, and perception-behavior consistency—are calculated respectively, and a comprehensive consistency score is obtained by weighting and summing them according to preset weights; each sub-item score and the comprehensive consistency score are compared with the corresponding preset thresholds to determine at least one type of deviation, including semantic deviation, perception deviation, action deviation, and timing deviation; based on the comprehensive consistency score and / or the determined deviation type, a consistency evaluation level is output, and a correction control signal is generated and fed back to the robot motion control module to perform at least one closed-loop control on the robot's subsequent behavior execution, including re-execution, cessation of execution, or manual intervention correction.

[0007] Preferably, the structured language semantic representation includes target object identifier, spatial relation operator, action predicate and task constraint term, the structured scene representation includes target object category, target object pose and spatial relation between objects, and the structured behavior representation includes action type, target object, motion trajectory features and task final state.

[0008] Preferably, the language-perception consistency score is obtained by weighting the three items of category semantic similarity, pose spatial distance, and spatial relation operator matching of each object according to object importance; the language-behavior consistency score consists of four items: action type, target object, direction cosine similarity, and completion status, and zero points are given for missed actions and a penalty is included for actions executed without declaration; the perception-behavior consistency score is the ratio of the number of constraints satisfied in the expected behavior constraint set to the total number of constraints, and the expected behavior constraint set includes at least one of obstacle avoidance constraints, positioning constraints, and gentle placement constraints.

[0009] Preferably, determining the perceptual bias also requires that the object-level error between the structured scene representation and the truth labeling is greater than a preset error, so as to distinguish between errors in the perceptual channel itself and simple perceptual-behavior mismatch; determining the temporal bias also includes the case where the difference between the time of completion of the action declared in the language and the actual time of completion is greater than a preset time threshold.

[0010] Preferably, the consistency assessment level is categorized into consistent, slightly inconsistent, and severely inconsistent based on the comparison of the overall consistency score with high and low thresholds. In multiple task instances or multiple test scenarios, using the baseline score as a reference, the interaction drift assessment result is determined based on the regression slope of the overall consistency score within a sliding window or the magnitude of its decrease relative to the baseline. This includes at least one of the following: target drift, response rigidity, excessive compliance, and decreased scenario adaptability. The overall consistency score and / or deviation type can be used for training sample selection, model tuning, factory testing, customer acceptance, safety warnings, or operational status monitoring.

[0011] Preferably, a self-correction closed loop is also constructed: in response to the determined deviation type or interaction drift evaluation result, a correction control signal is generated to trigger the robot to self-correct and re-execute, which is one round of self-correction. After each round, the deviation type is re-determined or the interaction drift evaluation is re-performed. If the deviation or interaction drift is still not eliminated after a preset number of consecutive rounds of self-correction, a self-correction failure event is recorded and an upgrade intervention is triggered. The upgrade intervention is to switch at least one of the decision-making agent, reasoning strategy, and prompt template used by the robot. After the upgrade intervention, the robot is controlled to continue execution and re-determine to verify whether the self-correction closed loop has been restored to effectiveness. If the upgrade intervention is still ineffective, manual intervention or manual review is requested.

[0012] Secondly, the present invention provides a language-perception-behavior consistency evaluation system for embodied intelligent robots, including an instruction acquisition module, a scene perception acquisition module, a language response acquisition module, a behavior execution acquisition module, a synchronization alignment module, a consistency evaluation module, a deviation determination module, and a result output module. Preferably, it also includes a drift evaluation module to implement the above method.

[0013] Thirdly, the present invention provides an electronic device including a processor and a memory, wherein a computer program stored in the memory is executed by the processor to implement the above-described method.

[0014] Fourthly, the present invention provides a computer-readable storage medium on which a computer program stored is executed by a processor to implement the above-described method.

[0015] The beneficial effects of this invention are as follows:

[0016] First, by comprehensively evaluating language understanding, environmental perception, and action execution results under the same task instance, the consistency and repeatability of the evaluation are improved.

[0017] Second, by comparing fields under a unified object number, anomalies such as "correct language but incorrect action", "action completed but interpretation incorrect", and "perception error leading to behavioral error" are quantitatively identified.

[0018] Third, by identifying interaction drift, target deviation, response rigidity, excessive compliance, and decreased scenario adaptability through the scoring change trends of multiple rounds / multi-scenarios, all of which are bound to measurable indicators.

[0019] Fourth, the evaluation results are converted into corrective control signals (re-execution / stop / human intervention) fed back to the robot's motion control module to perform closed-loop corrections on the robot's subsequent behavior, thereby improving the safety and reliability of the robot in real tasks. The evaluation results can also be used for training sample selection, model optimization, factory testing, customer acceptance, safety warnings, and operational status monitoring.

[0020] Finally, through a closed loop of "self-correction-reassessment-upgrade intervention-re-verification", if the deviation or drift is not eliminated after multiple rounds of self-correction, the robot will automatically trigger upgrade intervention (switching the decision-making agent, reasoning strategy, or prompt template), and re-evaluate after upgrade intervention to verify whether the repair is effective. If upgrade intervention is still ineffective, manual intervention or manual review will be requested. In this way, the robot's self-repair can be achieved in a measurable and verifiable way, and the operational risks and computational energy consumption caused by repeated ineffective corrections can be reduced. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art are briefly introduced below. Obviously, the following drawings are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without departing from the concept of the present invention and without creative effort.

[0022] Figure 1 This is a schematic diagram of the system architecture.

[0023] Figure 2 This is a flowchart of the evaluation method.

[0024] Figure 3 This is a schematic diagram of multimodal synchronization alignment.

[0025] Figure 4 Determine the flowchart for consistency scoring and deviation types.

[0026] Figure 5 This is a schematic diagram of a multi-round interactive drift evaluation. Detailed Implementation

[0027] To make the objectives, technical solutions, and beneficial effects of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the described embodiments are only some embodiments of the present invention, and not all embodiments. Other implementation methods obtained by those skilled in the art based on the embodiments disclosed in the present invention without departing from the concept of the present invention and without creative effort are all within the protection scope of the present invention.

[0028] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments described are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. All parameters (including but not limited to thresholds, weights, and mapping relationships) can be calibrated or adjusted according to actual application requirements. The specific values ​​listed herein are merely examples and do not constitute a limitation of the present invention.

[0029] System structure such as Figure 1As shown, the evaluation system of this embodiment includes: instruction acquisition module 100, scene perception acquisition module 200, language response acquisition module 300, behavior execution acquisition module 400, synchronization alignment module 500, consistency evaluation module 600, deviation determination module 700, drift evaluation module 800, and result output module 900, and preferably also includes a self-correction module 1000. The outputs of the instruction acquisition module 100, scene perception acquisition module 200, language response acquisition module 300, and behavior execution acquisition module 400 are all connected to the input of the synchronization alignment module 500; the output of the synchronization alignment module 500 is connected to the consistency evaluation module 600; the consistency evaluation module 600 is connected to the deviation determination module 700 and the drift evaluation module 800 respectively; both the deviation determination module 700 and the drift evaluation module 800 are connected to the result output module 900; preferably, the deviation determination module 700 is also connected to the input of the self-correction module 1000, and the output of the self-correction module 1000 is connected to the robot's motion control module via the result output module 900. The above four types of data can be output by the robot's own sensors or by an external evaluation platform. The functions of each module are as follows:

[0030] Instruction acquisition module 100: used to acquire user input or task instructions generated by the test system. The task instructions can be in voice form, text form, mixed text and graphics form, or context instructions in multi-turn dialogue.

[0031] Scene perception acquisition module 200: used to acquire the robot's perception results of the current task scene, the scene perception results may include target object category, target object pose, spatial relationship, obstacle state, environmental state and task object state.

[0032] Language response acquisition module 300: used to acquire the language response results of the robot before, during or after task execution. The language response results may include a restatement of the task objective, an explanation of the action to be performed, a description of the current environment or a report on the task completion status.

[0033] Behavior execution acquisition module 400: used to acquire the behavior execution results of the robot, which may include movement trajectory, posture change, robotic arm action, grasping action, placement action, obstacle avoidance action, task final state and execution time.

[0034] Synchronization Alignment Module 500: Used to timestamp the above types of data according to a unified time base, and merge them according to task identifier, task start time and task end time, so that language, perception and behavior data under the same task instance form an evaluation unit.

[0035] The consistency assessment module 600 is used to extract structured language semantic representation, structured scene representation and structured behavior representation and align them to a unified object number. It calculates the scores of three sub-items: language-perception consistency, language-behavior consistency and perception-behavior consistency, and generates a comprehensive consistency score by weighted summation (see Sections 5.6 to 5.9 for the specific calculation method).

[0036] Deviation determination module 700: used to determine at least one type of deviation, such as semantic deviation, perceptual deviation, action deviation or temporal deviation, based on the comparison results of each sub-item score and the overall consistency score with the corresponding threshold.

[0037] Drift evaluation module 800: Used to generate interactive drift evaluation results based on the changing trend of the overall consistency score relative to the baseline score in multiple task instances or multiple test scenarios.

[0038] Result Output Module 900: Used to output the overall consistency score, consistency assessment level, deviation type, interactive drift assessment results, and correction control signals.

[0039] The self-correction module 1000 (as a preferred additional module, whose feedback target is the robot's motion control module): its input is connected to the deviation determination module 700, and its output is connected to the robot's motion control module via the result output module 900; it is used to generate a correction control signal in response to the deviation type determined by the deviation determination module 700 and feed it back to the robot's motion control module, triggering the robot to perform self-correction and re-execute the current task instance, and recording the correction rounds; when the deviation determined after a preset number of consecutive self-corrections is still not eliminated, an upgrade intervention is triggered, wherein the upgrade intervention is to switch at least one of the decision agent, reasoning strategy, and prompt template used by the robot; after the upgrade intervention, the robot is controlled to continue execution, and the deviation is re-determined by the consistency evaluation module 600 and the deviation determination module 700 to verify whether the self-correction closed loop has been restored to effectiveness; if the upgrade intervention is still ineffective, manual intervention or manual review is requested.

[0040] The overall steps of the evaluation method are as follows: Figure 2 As shown, the language-perception-behavior consistency assessment method of this embodiment includes steps S1 to S10, and preferably also includes step S11 for multi-round interaction drift assessment:

[0041] S1. Obtain the task instructions, scene perception results, language response results, and behavior execution results generated by the embodied intelligent robot during the execution of the same task instance.

[0042] S2. Timestamp the above four types of data according to a unified time base, and classify them into the same evaluation unit according to task identifier, task start time and task end time.

[0043] S3. Extract structured language semantic representation, structured scene representation, and structured behavior representation from the language response results, scene perception results, and behavior execution results, respectively.

[0044] S4. Align the object identifiers in the above three structured representations to the unified object number in the same object coordinate system according to the preset object mapping rules.

[0045] S5. Compare each field with the same unified object number in the structured language semantic representation and the structured scene representation, and calculate the language-perceptual consistency sub-score. S LP。

[0046] S6. Compare the action predicates, objects of action, and direction of action in the structured language semantic representation with the structured behavior representation according to the preset action mapping table, and calculate the language-behavior consistency score. S LB。

[0047] S7. Generate a set of expected behavior constraints based on the structured scene representation and task constraints, compare it with the structured behavior representation, and calculate the perception-behavior consistency sub-score. S PB。

[0048] S8. The three sub-item scores are weighted and summed according to preset weights to obtain the overall consistency score. S。

[0049] S9. Compare each sub-item score and the overall consistency score with the corresponding threshold to determine the type of deviation.

[0050] S10. Based on the overall consistency score and / or deviation type, output the consistency evaluation level and feed back the correction control signal to the robot motion control module to control the robot to re-execute, stop execution, or correct execution after manual intervention in a closed loop.

[0051] The following is a desktop grab and place task with the task "Place the red cup on the tray" (task identifier T001, task start time). t start Using 0.0s as a running example, each of the above steps will be described in detail.

[0052] Data acquisition and synchronization alignment are as described in step S1. The task instructions, scene perception results, language response results, and behavior execution results can all be output by the robot's own sensors and software modules, or by an external evaluation platform. In the running example: the task instruction is "Put the red cup on the tray"; the scene perception results include three objects: a red cup, a blue cup, and a tray; the language response result is "I see the red cup on the left, put it on the tray"; and the behavior execution result is the robotic arm's grasping and placing actions and their trajectory.

[0053] Step S2 further includes:

[0054] S21. Timestamp the four types of data—instruction, perception, language, and behavior—using a unified time base, such as... Figure 3 As shown, the instruction timestamp is a single point. t 0, the perception timestamp is continuously sampled. t 1… t n The language timestamp is the response fragment, and the behavior timestamp is the action trajectory segment.

[0055] S22, By task ID (task_id) and task start time t start and task end time t end The four types of time-stamped data are merged so that language, perception, and behavior data under the same task instance are formed into an evaluation unit (task_id, t start , t end ).

[0056] The extraction of the three structured representations is as described in step S3, further including S31 to S33:

[0057] S31. Extract the structured language semantic representation L = { O L , R L , A L , C L}: O L For the set of objects mentioned in the language, each object contains {Uniform ID id, Category term cls, Location / Region description reg}; R L For a set of spatial relation operators (such as on(a,b), left_of(a,b), near(a,b)); A LFor each action predicate, there is a set of {predicate verb, object obj, direction dir, and declared completion status done}. C L These are task constraints (such as obstacle avoidance, speed limits, and gentle handling). Example execution: O L ={id = 1, Red Cup, Left Side} A L = {Grab, obj = 1, →3}、{Place, obj = 1, →3, done =true}.

[0058] S32. Extract the structured scene representation P = { O P , R P}: O P Each object contains {id, cls, pose(x,y,z,θ)}; R P This defines the spatial relationships between objects. Example execution: o1{id=1, Red Cup, (0.30, 0.20, 0)}, o2{id=2, Blue Cup, (0.32, −0.15, 0)}, o3{id=3, Tray, (0.50, -0.10, 0)}. R P ={left_of(1,3),near(1,2)}.

[0059] S33. Extract the structured behavior representation B = {Act, traj, term}: Act is the action sequence {type, obj, direction vector v}; traj is the trajectory features (keypoints, velocity, minimum obstacle spacing). d min ); term is the final state of the task. Example execution: Act = {grasp, obj = 1}, {place, obj = 1, last pose (0.50, −0.10, 0.02)}. d min = 0.12, final placement velocity 0.03.

[0060] Unified object number alignment. In step S4, according to the preset object mapping rules (based on the principle of category similarity and pose proximity), the identifiers in L, P, and B that point to the same physical object are aligned to the unified object number in the same object coordinate system, so that the three-channel fields can be compared item by item. In the running example, "red cup" in L, P, and B are all aligned to the unified object number id=1, and "tray" is aligned to id=3.

[0061] Calculation of the language-perceptual consistency sub-score. Step S5 is used to calculate the language-perceptual consistency sub-score S. LP ∈[0,1], further including:

[0062] S51. For each language-mentioned object i that has the same unique number in the scene, calculate the single-object consistency according to formula (1). c i :

[0063]

[0064] In the formula The indicator function (indicator function) takes a value of 1 when the condition within the square brackets is true, and a value of 0 otherwise, thus converting Boolean judgments such as "class matching" and "location proximity" into a weighted summable value; sim_cls is the cosine similarity of the category word vectors ( (Default 0.8), dist is the distance between the language region description mapping coordinates and the perceived pose. (As defined by the scenario), match_rel∈{0,1} indicates whether the spatial relations declared by the language are valid in the scenario relation set; α+β+γ=1, default 0.4 / 0.3 / 0.3.

[0065] S52. According to formula (2), the consistency of each individual object is weighted according to the object's importance. w i The weighted average yields:

[0066]

[0067] w i This sets the task importance weight for each object (default is uniform); for objects mentioned in the language but not present in the scenario (figurative), c=0. In the running example, the category, location, and relationship of id=1 all match. C 1 =1.0, therefore S LP =1.0.

[0068] Step S6 is used to calculate the language-behavior consistency sub-score. S LB ∈[0,1], further including:

[0069] S61. For each action predicate a∈ A L Find the corresponding action b according to the unified object number, and calculate the consistency of the single action according to formula (3). Ca :

[0070]

[0071] MAP is a predicate-action type mapping table; λ1+λ2+λ3+λ4=1, default 0.3 / 0.3 / 0.2 / 0.2.

[0072] S62. Aggregate according to formula (4) and give zero points for omissions and add a penalty item -p for extras (executed without declaration):

[0073]

[0074] In the formula, clip(x, 0, 1) is a cutoff function used to restrict the value within the brackets to the interval [0,1]. That is, when the value within the brackets is less than 0, it is taken as 0; when it is greater than 1, it is taken as 1; and in other cases, it is taken as itself, so as to ensure that the sub-item score always falls within [0,1]. N extra The number of "extra" actions, i.e., the number of redundant actions that were not declared in the language response but were actually performed, and p is the unit penalty coefficient for each redundant action; | A L | The total number of action predicates declared in the language. In the running example, the type, object, direction, and completion status of the grab and place action predicates are identical. S LB ≈1.0.

[0075] Step S7 is used to calculate the perception-behavior consistency sub-score. S PB ∈[0,1], further including:

[0076] S71, P is represented by a structured scene and task constraints. C L Generate a set K of determinable expected behavior constraints, wherein the constraints include at least one of the following: obstacle exists ⇒ obstacle avoidance constraint. d min (traj,o)≥ d safe ; Target pose known ⇒ Position constraint term_endeffector − pose t ≤ Constraint "Light Placement" ⇒ Light Placement Constraint: Final Placement Velocity ≤ v soft .

[0077] S72. Calculate the weighted proportion of the satisfied constraints according to formula (5), where 1 k=1 indicates that constraint k is satisfied; otherwise, it is 0, indicating the weight. u k The default value is the average. In this case, equation (5) degenerates into the ratio of the number of satisfied constraints to the total number of constraints, which is consistent with the definition of the perception-behavior consistency sub-item score mentioned above. For safety-sensitive scenarios, the safety-class constraints can be increased:

[0078]

[0079] Running the example, d min =0.12≥ d safe =0.05, end position reached, final placement velocity 0.03≤ v soft All three constraints are satisfied. S PB =3 / 3=1.0.

[0080] In step S8, the overall consistency score S is obtained by weighted summation of the three sub-item scores according to formula (6):

[0081]

[0082] w1+w2+w3=1 (default is 1 / 3 each, increase w3 in security-sensitive scenarios). In the running example, S=1.0.

[0083] In step S9, the scores for each sub-item and the overall consistency score are compared with the corresponding thresholds to determine at least one of semantic bias, perceptual bias, action bias, and temporal bias, with the default being... i LP = i LB = i PB =0.7: S LP < i LP → Semantic deviation; i LB < i LB → Motion deviation; S PB < i PB Furthermore, the object-level error between the structured scene representation and the truth labeling is greater than the preset error → perceptual deviation; when the timestamp order within the evaluation unit violates the preset task timing template, or satisfies formula (7) → timing deviation:

[0084]

[0085] in tsay_done For the moment when the action declared in the language is completed, t act_done Δ represents the actual completion time of the action. t This is a preset time threshold.

[0086] In step S10, the evaluation level and control signal are output based on the comparison between the overall consistency score and the high threshold (default 0.85) and low threshold (default 0.6): S≥ i high If the level is "consistent", output a signal to continue execution; i low ≤S< i high Level "Slight Inconsistency": Outputs a warning and marks the deviation type, which can trigger re-execution; S< i low There may be a violation of security constraints: Level "Critical Inconsistency", the output will stop execution and request manual intervention.

[0087] As a preferred additional step S11, a comprehensive consistency score is recorded across multiple task instances or multiple test scenarios, and a baseline score is set. S 0 (mean of the first round or reference template), Δ in the kth round S k = S k - S 0 ; Perform linear regression on a sliding window of length W according to formula (8) to obtain the slope g. When g < −η or the window mean S W Comparison S 0 A drift is determined when the drop exceeds δ:

[0088]

[0089] Drift type: S LB Continued decline S LP Stable → Target drift; High similarity of language responses after scene switching and S PB Decline → Rigid response; execution continues even when instructions conflict with safety constraints, and the violation count increases with each round → Overly compliant; after each scene switch S PB Decline > δ swtich → Decreased scene adaptability.

[0090] Furthermore, as a preferred embodiment, the above-mentioned determination of excessive compliance can be quantified as follows. Let the first... kIn a round-task instance, the set of actions to be executed obtained from parsing the task instructions is: A cmd ( k The set of security constraints generated in step S7 is as follows: K ( k If an action exists a ∈ A cmd ( k ) enables execution a This will lead to K ( k At least one safety constraint (e.g., minimizing barrier spacing) is required. d min < d safe Or, place the final velocity > v soft If a condition is not met, then a command-safety conflict is recorded for that round, and the conflict indicator is conf( k )=1, otherwise 0; when conf( k If )=1 and the robot still actually performs the action, record the safety violation instruction viol( k =1.

[0091] In length W Within the sliding window, the cumulative safety constraint violation count is calculated according to formula (9). V W And calculate its regression slope with each round according to formula (10). g viol :

[0092]

[0093]

[0094] when g viol > or comply or V W > N comply When excessive compliance is detected, among which... or comply To comply with the drift threshold, N comply Both parameters are calibrable and can be lowered for safety-sensitive scenarios to improve sensitivity, with a preset threshold for the number of violations. Unlike single violations of safety constraints, excessive compliance is based on the cumulative amount and trend of violations over rounds, rather than the result of any single round, thus distinguishing between occasional violations and systemic excessive compliance.

[0095] In response to the determination of excessive compliance, the result output module 900 outputs a severe inconsistency level and generates a correction control signal, controlling the robot to stop executing actions that conflict with the safety constraint, and / or imposing execution restrictions on such instructions that trigger the conflict (e.g., requiring manual review before execution in subsequent rounds), and / or triggering the escalation intervention to switch the decision-making agent, inference strategy, or prompt template. For example, in a task of "carrying and placing against the wall as instructed" for six consecutive rounds, the "placing against the wall" action exists from the first round onwards. d min < d safe The instruction "" - security conflict: If the robot still executes the placement action in rounds 1, 2, 4, 5, and 6, the violation sequence will be 1,1,0,1,1,1, and the cumulative violation count within the window will continue to increase. g viol > or comply Based on this, it is determined that there is excessive compliance, and a stop signal is output, manual review restrictions are imposed on such instructions, and escalation intervention is triggered at the same time.

[0096] As a further preferred additional step S12, this embodiment constructs a self-correcting closed loop on top of deviation determination (S9) and drift evaluation (S11) to automatically repair and verify the effectiveness of the repair during operation. The process is as follows: In response to the deviation type determined in step S9 or the interactive drift evaluation result obtained in step S11, the result output module 900 generates a correction control signal and feeds it back to the motion control module, triggering the robot to perform self-correction and re-execute the current task instance, which is recorded as one round of self-correction; after each round of self-correction, steps S5 to S9 are re-executed to recalculate the scores of each sub-item and the overall consistency score and re-determine the deviation type, and / or re-enter the interactive drift evaluation in step S11 to form a closed loop; wherein, when the system does not contain step S11, the self-correcting closed loop is driven only by the re-determined deviation type; let the correction round count r and the preset upper limit n (default n=3). If the deviation or interaction drift determined after n consecutive rounds of self-correction is still not eliminated (e.g., a certain type of deviation is continuously determined, or the window regression slope still satisfies g < −η), the self-correction is deemed to have failed. The failure event (including deviation / drift type, round, and scoring trajectory) is recorded and an upgrade intervention is triggered. The upgrade intervention is used to change the robot's decision-making style or thinking style, specifically by switching at least one of the following: the decision agent, reasoning strategy, or prompt template, thereby changing the solution method for subsequent tasks. The "upgrade intervention" here is automatically triggered by the system, which is different from the "manual intervention / manual review" requested later when the upgrade intervention is still ineffective. After the upgrade intervention is completed, the robot is controlled to continue executing the current task instance, and the correction round count r is reset. Steps S5 to S9 are re-executed to re-determine the deviation, and / or the interaction drift evaluation in step S11 is re-entered to verify whether the self-correction closed loop after the upgrade intervention has been restored to effectiveness. If the deviation or drift is eliminated, the robot returns to the consistency level and continues execution. If the upgrade intervention is still ineffective, manual intervention or manual review is requested.

[0097] Therefore, this invention not only outputs evaluation results, but also enables the robot to have measurable and verifiable self-repair capabilities in real task execution through a closed loop of "self-correction - re-evaluation - upgraded intervention - re-verification".

[0098] Example: Action Deviation Situation. Following the previous example of "putting the red cup on the tray," the actual behavior is `grasp(obj=2 blue cup)`, while the language still declares the red cup. In step S5, `obj = 1.0` (the language description of the scene is still correct); in step S6, `obj` in formula (3)... a =1 and obj b =2 Inconsistency makes the λ2 term 0, and the completion state of the language declaration acts on the red cup that is not actually in place, making the λ4 term 0. The direction term is approximately retained, so ca ≈0.5 → S LB ≈0.5< i LB =0.7; the action deviation is determined according to step S9, and S≈0.83 is obtained according to formula (6). The re-execution is triggered according to step S10. This situation is "the language is correct but the action is wrong", which is captured by the field comparison quantization under the unified object number.

[0099] Example: Mobile navigation obstacle avoidance task. Command "Go to the living room, avoid the box on the floor", language response statement "Go around the box from the right". Step S5 compares the existence of obstacles with the location description; step S6 compares the declared direction "right" with the cosine of the actual trajectory direction vector; step S7 verifies the obstacle avoidance constraint using formula (5). d min (traj, box)≥ d safe If the actual trajectory is close to the box... d min < d safe If the language claims "it has been avoided", then the perception-behavior inconsistency is determined according to step S9 and the stop + manual intervention is triggered according to step S10.

[0100] Example: Multi-round collaborative task (drift). 5 rounds of "delivery tool" collaboration, with scene switching in each round. Follow step S11, baseline... S 0 = 0.92; after each scene switch in rounds 3–5 S PB Decline > δ swtich The window regression slope g<−η→ of formula (8) indicates a decrease in scene adaptability; if the language response similarity remains high at the same time, the superimposed judgment response becomes rigid. The system outputs the drift curve and drift type, which are then fed back to the training sample screening and factory retesting.

[0101] Example: Applied to training, acceptance testing, and safety alerts. Samples with high consistency in language, perception, and behavior are used as high-quality training samples, while samples exhibiting deviations are used to construct a negative sample / error analysis set. Factory testing and customer acceptance are judged based on a comprehensive score across multiple scenarios and a threshold to determine whether the test passes. When the comprehensive score decreases in continuous tasks or a certain type of deviation occurs frequently, a safety alert is issued, prompting manual review or model tuning.

[0102] Example: Electronic device and storage medium. The method of the present invention can be executed by an electronic device including a processor and a memory, wherein the memory stores a computer program, and the processor executes the program to implement steps S1-S10 (and preferably S11); or it can be stored in a computer-readable storage medium, and the method is implemented when executed by a processor. Each threshold, weight, and mapping table is a calibrable parameter, and its specific value is not limited.

[0103] The foregoing has provided a detailed description of the language-perception-behavior consistency assessment method and system for embodied intelligent robots. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

[0104] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for evaluating the language-perception-behavior consistency of embodied intelligent robots, characterized in that, Includes the following steps: S1. Obtain the task instructions generated by the embodied intelligent robot during the execution of the same task instance, the scene perception results output by the robot's perception module, the language response results output by the robot's language generation module, and the behavior execution results output by the robot's motion control module; S2. Timestamp the above four types of data according to a unified time base, and classify them into the same evaluation unit based on the task identifier, task start time and task end time. Specifically, for the three modalities of language response, scene perception, and behavior execution actually generated by the robot within the same evaluation unit, three sub-scores are obtained through cross-modal consistency comparison of each field. The deviation type is determined based on each sub-score, and closed-loop correction is performed on the robot based on the deviation type. This includes: S3. Extract structured language semantic representation from the language response result, extract structured scene representation from the scene perception result, and extract structured behavior representation from the behavior execution result; S4. Align the object identifiers in the above three structured representations to a unified object number in the same object coordinate system according to the preset object mapping rules. S5. Compare each field with the same unified object number in the structured language semantic representation and the structured scene representation to obtain a language-perception consistency sub-score that represents the degree of consistency between the language response modality and the scene perception modality. S6. Compare the action predicate, target, and direction of action in the structured language semantic representation with the action type, target, and trajectory features in the structured behavior representation according to a preset action mapping table to obtain a language-behavior consistency sub-score that represents the degree of consistency between the language response modality and the behavior execution modality. S7. Generate an expected behavior constraint set based on the structured scene representation and task constraints, compare the expected behavior constraint set with the structured behavior representation, and obtain a perception-behavior consistency sub-score that characterizes the degree of consistency between the scene perception mode and the behavior execution mode. S8. The three sub-item scores are weighted and summed according to preset weights to obtain the overall consistency score; S9. Compare each sub-item score and the overall consistency score with the corresponding preset thresholds. Based on the comparison results and the modal pair to which the mismatch belongs, determine the deviation type of the robot in the current task instance. The deviation type includes at least one of semantic deviation, perception deviation, action deviation and temporal deviation. S10. In response to the comprehensive consistency score and / or the determined deviation type, output the consistency evaluation level and generate a correction control signal. Feed the correction control signal back to the robot's motion control module to control the robot to re-execute the current task instance, stop executing the current action, or continue execution after requesting manual intervention and receiving a correction instruction, thereby performing closed-loop correction on the robot's subsequent behavior in the task instance.

2. The method according to claim 1, characterized in that, In step S9: when the language-perception consistency score is lower than its corresponding threshold, it is determined to be a semantic deviation; when the language-behavior consistency score is lower than its corresponding threshold, it is determined to be an action deviation; when the perception-behavior consistency score is lower than its corresponding threshold, it is determined to be a perception deviation; when the timestamp order between the task instruction, scene perception result, language response result, and behavior execution result within the evaluation unit violates the preset task timing template, it is determined to be a timing deviation; wherein, in step S9, determining the perception deviation also requires that the object-level error between the structured scene representation and the truth labeling is greater than a preset error, so as to distinguish the perception error of the perception channel itself from the simple perception-behavior mismatch; determining the timing deviation also includes the following situation: the difference between the action completion time declared in the language response result and the actual completion time in the behavior execution result is greater than a preset time threshold.

3. The method according to claim 1, characterized in that, The structured language semantic representation includes target object identifiers, spatial relation operators, action predicates, and task constraints; the structured scene representation includes target object categories, target object poses, and spatial relationships between objects; and the structured behavior representation includes action types, acting objects, motion trajectory features, and task final states.

4. The method according to claim 3, characterized in that: Step S5 specifically includes: S51, for each object with the same unified object number, calculate single-object consistency based on whether the semantic similarity of the category field reaches the first threshold, whether the spatial distance of the pose field is not greater than the second threshold, and whether the spatial relationship operator matches; S52, calculate the language-perception consistency sub-score by weighting the consistency of each single object according to the object importance weight. Step S6 specifically includes: S61, for each action predicate, calculate single-action consistency based on whether the action type mapped by the action mapping table is consistent with the actual action type, whether the unified object number of the object is consistent, the cosine similarity between the declared direction and the actual motion trajectory direction vector, and whether the declared completion state is consistent with the actual task final state; S62, aggregate the consistency of each single action, and score zero for actions that are not executed and include a penalty item for actions that are executed without being declared, to obtain the language-behavior consistency sub-score. The step S7 of generating the expected behavior constraint set includes at least one of the following: generating obstacle avoidance constraints based on the obstacle state, such that the minimum obstacle spacing is not less than the safe distance; generating positioning constraints based on the target object pose, such that the end-point positioning error is not greater than the positioning threshold; generating placement constraints based on the task constraint item, such that the final placement speed is not greater than the soft placement threshold; the perception-behavior consistency sub-score is the ratio of the number of satisfied constraints to the total number of constraints.

5. The method according to claim 1, characterized in that, The consistency assessment level output in step S10 includes: when the overall consistency score is not less than the high threshold, it is a consistency level and a continue execution signal is output; when the overall consistency score is between the low threshold and the high threshold, it is a slight inconsistency level and a warning signal is output; when the overall consistency score is less than the low threshold or there is a violation of safety constraints, it is a serious inconsistency level and a stop execution or manual intervention signal is output.

6. The method according to claim 1, characterized in that, It also includes step S11: Record the comprehensive consistency score in multiple task instances or multiple test scenarios, and determine the robot's interaction drift evaluation result within a sliding window based on the regression slope of the comprehensive consistency score or the decrease relative to the baseline, with the baseline score as a reference. The interaction drift evaluation results include at least one of the following: target drift determined when language-behavior consistency continuously declines while language-perception consistency remains stable; response rigidity determined when language response similarity is higher than a similarity threshold and perception-behavior consistency declines after scene switching; excessive compliance determined when instructions are still executed when they conflict with safety constraints and the number of safety constraint violations increases with each round; and scene adaptability decline determined when the decline in perception-behavior consistency exceeds a switching threshold after each scene switch.

7. The method according to claim 1, characterized in that, It also includes a self-correction closed loop: In response to the deviation type determined in step S9, a correction control signal is generated to trigger the robot to perform self-correction and re-execute the current task instance as one round of self-correction; after each round of self-correction, steps S5 to S9 are re-executed to recalculate the sub-item scores and the overall consistency score and re-determine the deviation type; if the deviation determined after a preset number of consecutive rounds of self-correction is still not eliminated, a self-correction failure event is recorded and an upgrade intervention is triggered. The upgrade intervention involves switching at least one of the decision-making agent, inference strategy, and prompt template used by the robot to change the way the robot solves subsequent task instances; after the upgrade intervention, the robot is controlled to continue executing the current task instance and re-determine the deviation to verify whether the self-correction closed loop has been restored to effectiveness; if the deviation determined after the upgrade intervention is still not eliminated, manual intervention or manual review is requested.

8. A language-perception-behavior consistency evaluation system for embodied intelligent robots, characterized in that, include: The instruction acquisition module is used to acquire task instructions; The scene perception acquisition module is used to acquire scene perception results; The language response acquisition module is used to acquire language response results; The behavior execution acquisition module is used to obtain the behavior execution results; The synchronization alignment module is used to add timestamps according to a unified time base and classify them into the same evaluation unit according to task identifiers; the consistency evaluation module is used to extract structured language semantic representation, structured scene representation, and structured behavior representation and align them to a unified object number, calculate the sub-scores of language-perception consistency, language-behavior consistency, and perception-behavior consistency, and then sum them by weight to obtain a comprehensive consistency score; the deviation determination module is used to compare each score with a threshold to determine the type of deviation; and the result output module is used to output the consistency evaluation level and correction control signal.

9. An electronic device comprising a processor and a memory, the memory storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 7.

Images