Manipulation identification method and system based on human-computer interaction

Abstract

The invention relates to the technical field of operation error recognition, and discloses an operation error recognition method and system based on human-computer interaction, and the method comprises the steps: collecting man-made interaction behaviors through a front end; carrying out behavior analysis on the interaction behavior, and carrying out primary recognition on errors; transmitting the operation instruction of the interaction behavior which is not judged to be wrong in the primary identification to an execution part, and carrying out secondary error identification based on mechanical constraint in the execution part; analyzing the execution part behaviors to finish the final recognition of errors; after identification is completed, execution partition is performed on each result which is identified as an error, and an execution part rejects an interactive behavior of execution errors, so that operation errors are prevented. Safety and accuracy in complex interaction tasks are greatly improved, and the limitation that in the prior art, operation errors are difficult to detect and process in real time is overcome.

Description

Technical Field

[0001] The present invention relates to the technical field of operation error recognition, and specifically provides an operation error recognition method and system based on human - machine interaction. Background Art

[0002] With the rapid development of intelligent manufacturing and industrial automation, the operation applications of dual - arm robots in complex environments are becoming more and more extensive. Dual - arm robots can interact with operators through various sensors and control systems to achieve efficient industrial production. However, with the increase in task complexity, operation errors have become a problem that cannot be ignored. Operation errors may lead to equipment damage, reduced production efficiency, and even safety accidents.

[0003] Traditional operation error detection systems often rely on a single monitoring method and cannot effectively capture potential errors in complex interaction environments. In addition, the operation behaviors of operators usually have diversity and uncertainty, especially when dual - arm robots perform complex tasks. Traditional error detection methods are difficult to accurately determine whether the operation meets the expectations. Therefore, how to real - time identify operation errors and take corresponding measures during the interaction between dual - arm robots and humans has become an urgent problem to be solved in the field of intelligent control. Summary of the Invention

[0004] In view of the above - mentioned problems, the present invention is proposed.

[0005] To solve the above - mentioned technical problems, the present invention provides the following technical solutions: An operation error recognition method based on human - machine interaction, including:

[0006] Collecting human interaction behaviors through the front - end;

[0007] Performing behavior analysis on the interaction behaviors for the initial recognition of errors;

[0008] Transmitting the operation instructions of the interaction behaviors that are not initially recognized as errors to the execution part, and performing a secondary recognition of errors based on mechanical constraints in the execution part;

[0009] Completing the final recognition of errors through the analysis of the behaviors of the execution part;

[0010] After the recognition is completed, perform execution isolation on each result recognized as an error. The execution part refuses to execute the interaction behaviors with errors to prevent operation errors.

[0011] As a preferred solution of the operation error recognition method based on human - machine interaction according to the present invention, wherein: the interaction behaviors include operation instructions issued to the dual - arm robot and interaction through the actions feedback by the robot.

[0012] As a preferred solution of the operation error recognition method based on human-computer interaction according to the present invention, wherein: the initial recognition includes training a random forest model based on the destructive behavior patterns marked in historical data, and using the random forest model to predict whether the current behavior performs a destructive behavior;

[0013] When it is recognized as a destructive behavior, the destruction target of the destructive behavior is further recognized. If the permission of the destruction target is open, it is determined that the result of the initial recognition is normal; if the permission of the destruction target is not open, it is determined that the result of the initial recognition is a mistake;

[0014] The destruction target is the target object corresponding to the current destructive behavior; using the position of the arm, the visual feedback result, and the operation instruction as inputs, the trained LSTM model processes the time series data to measure the current destructive behavior object, calculates the confidence score for each possible target object, and selects the one with the highest confidence as the destruction target;

[0015] Wherein, when the system registers that the target needs to be destroyed, the permission of the target will be opened.

[0016] As a preferred solution of the operation error recognition method based on human-computer interaction according to the present invention, wherein: the mechanical constraints include: joint angle constraint: ensuring that the joint angles of each robotic arm do not exceed the designed range; speed and acceleration constraint: restricting the movement speed and acceleration of the robotic arm within a safe range; working space constraint: keeping the operating range of the robotic arm within a predetermined working space and not colliding with surrounding objects; force and torque constraint: during the grasping or operation process, the applied force and torque conform to the physical capabilities of the device; time constraint: the execution time of the operation cannot be too long or too short to ensure the task completion time;

[0017] θ 1 ∈ [ α min ,α max ]

[0018] v 2 ≤β max

[0019] a 3 ≤γ max

[0020] P 4 ∈ [ δ min ,δ max ]

[0021] d 5 ( O 6 ) ≥∈safe

[0022] F 7 ≤ ζ max

[0023] T 8 ≤ η max

[0024] t 9 ∈ [ θ min , θ max ]

[0025] T 10 ∈ [ ι min , ι max ]

[0026] where θ 1 represents the current joint angle of the robotic arm 1, α min represents the minimum limit of the joint angle α max represents the maximum limit of the joint angle, v 2 represents the movement speed of the robotic arm 2, β max represents the maximum allowable speed, a 3 represents the acceleration of the robotic arm 3, γ max represents the maximum allowable acceleration, P 4 represents the end position of the robotic arm 4, δ min represents the minimum position δ of the workspace max represents the maximum position of the workspace, d 5 (O 6 ) represents the distance between the robotic arm 5 and the obstacle 6, ∈ safe represents the safety distance, F 7 represents the grasping force of the robotic arm 7, ζ max represents the maximum allowable grasping force, T 8 represents the torque applied by the robotic arm 8, η max represents the maximum allowable torque, t 9 represents the time taken for the robotic arm 9 to complete the operation, θ min represents the minimum allowable operation time θ max represents the maximum allowable operation time, T 10 represents the temperature of the environment, ι min represents the minimum allowable environment temperature, ι max represents the maximum allowable environment temperature.

[0027] As a preferred solution of the operation error recognition method based on human-computer interaction according to the present invention, wherein: the secondary recognition includes that the system monitors the joint angles of the robotic arm in real time to ensure that it operates within the allowable physical range; if it is detected that the range exceeds the mechanical constraint, it is determined as a misbehavior and the operation is immediately stopped.

[0028] As a preferred solution of the operation error recognition method based on human-computer interaction according to the present invention, wherein: the final recognition includes constructing intention tags for the intention concepts of each step, and constructing an intention tag network according to the causal relationship between each intention; in the network, the operation intentions with a sequential operation relationship are represented by directed lines, and the direction of the line is from the intention tag of the previous step to the intention tag of the subsequent step; the operation intentions without a sequential operation relationship are represented by undirected lines in the network;

[0029] The input is the continuous action information of the dual-arm robot, and the LSTM model trained for the recognition of the executed behavior is used to predict the intention; the prediction result of the current behavior is output;

[0030] Wherein, the prediction result of the current behavior is an intention tag;

[0031] Use the depth-first search algorithm to traverse in the network to obtain all possible paths. Among them, the paths are generated according to the direction of the lines in the network. The traversal starting point is the initial state of the robot, and the traversal end point is the final intention of the operation;

[0032] Arrange the intention tags predicted according to the current behavior and the other behaviors that have occurred in the order of occurrence to obtain the intention tag sequence XL; match XL with all possible paths. When the sequence of XL is a part of the intention tag sequence of the path L, it is determined that XL matches L;

[0033] If there is no matching path, the recognition result is determined as a mistake; if there is a matching path, it is determined as no mistake;

[0034] When the recognition result of the final recognition is a mistake, the intention tag sequence XL is sent to the management staff for permission verification, and whether to execute is selected through the feedback of the management staff; if the feedback result is to execute, the recognition result is corrected to no mistake; if the feedback result is not to execute, the original recognition result is maintained.

[0035] As a preferred solution of the operation error recognition method based on human-computer interaction according to the present invention, wherein: the execution isolation for each recognized mistake result includes that when the system recognizes the result of the operation mistake, the system immediately pauses the current operation, stops the movement of the robotic arm, but does not cut off the interaction control signal, and restores the dual-arm robot to the safe state without mistake at the previous moment;

[0036] Continuously receive control instructions for interaction through the front end, continuously identify operation errors in the control instructions. If the control signal is restored to a non-error state, continue to execute the operation.

[0037] An operation error recognition system based on human-computer interaction using the method described in the present invention, characterized in that:

[0038] An acquisition unit that acquires human interaction behaviors through the front end;

[0039] An identification unit that performs behavior analysis on the interaction behaviors to perform a primary identification of errors; transmits the operation instructions of the interaction behaviors that are not determined to be errors in the primary identification to the execution part, and performs a secondary identification of errors based on mechanical constraints in the execution part; completes the final identification of errors through the analysis of the behaviors of the execution part;

[0040] A partition unit that, after completion of the identification, performs execution partitioning on each result identified as an error respectively. The execution part refuses to execute the interaction behaviors with errors to prevent operation errors.

[0041] A computer device, comprising: a memory and a processor; the memory stores a computer program, wherein: when the processor executes the computer program, the steps of the method described in any one of the present invention are implemented.

[0042] A computer-readable storage medium, on which a computer program is stored, wherein: when the computer program is executed by a processor, the steps of the method described in any one of the present invention are implemented.

[0043] Advantages of the present invention: The operation error recognition method based on human-computer interaction provided by the present invention, through an intelligent and multi-level error recognition method, combined with an intention marking network and an execution partitioning mechanism, provides an innovative and effective solution for the operation of a two-armed robot, greatly improving the safety and accuracy in complex interaction tasks, and overcoming the limitations in the prior art that it is difficult to detect and process operation errors in real time. Description of the Drawings

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for description in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0045] Figure 1 It is the overall flowchart of an operation error recognition method based on human-computer interaction provided by the first embodiment of the present invention. Detailed Embodiments

[0046] To make the above objects, features, and advantages of the present invention more apparent and understandable, the following provides a detailed description of the specific embodiments of the present invention in conjunction with the accompanying drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present invention.

[0047] Example 1, referring to Figure 1 , which is an embodiment of the present invention, provides an operation error recognition method based on human-computer interaction, including:

[0048] S1: Collect human interaction behaviors through the front end.

[0049] The interaction behaviors include operation instructions issued to the dual-arm robot and interaction through the actions feedback by the robot.

[0050] Among them, "interaction behavior" can specifically refer to the behavior of humans operating the dual-arm robot. It covers the operation instructions issued by the operator to the dual-arm robot through various means (such as gestures, buttons, touch screens, joysticks, etc.) and interaction through the actions feedback by the robot. Interaction behaviors can include the following specific operations:

[0051] Manual input commands: Instructions issued to the robot through a physical interface (such as a control panel, remote control, keyboard, etc.), such as actions like moving, grasping, rotating, etc.

[0052] Gesture control: The operator controls the dual-arm robot to perform specific tasks through preset gestures or postures. For example, gesture recognition is used to control grasping or placing objects.

[0053] Touch and slide operations: Use a touch screen or a sliding panel to operate the robot for fine operations such as adjusting the angle and speed of the robotic arm.

[0054] Force feedback interaction: During the execution process, the resistance or feedback encountered by the dual-arm robot when performing actions is sensed through a force feedback device to help the operator judge whether the action is correct and adjust the operation.

[0055] Visual or voice-assisted interaction: Operate the robot through a camera or voice commands. The operator can identify the task object through the visual system and issue precise commands to the robot.

[0056] S2: Conduct behavior analysis on the interaction behaviors for the initial recognition of errors.

[0057] The initial recognition includes training a random forest model based on the destructive behavior patterns marked in historical data, and using the random forest model to predict whether the current behavior is a destructive behavior (such a behavior is generally large-scale and has no specific behavior area limit. Using a random forest can only effectively output a judgment result. For the specific location, further analysis of the content display in the interaction interface is required). When it is recognized as a destructive behavior, the destruction target of the destructive behavior is further identified. If the permission of the destruction target is open, the initial recognition result is determined to be normal; if the permission of the destruction target is not open, the initial recognition result is determined to be a mistake.

[0058] It should be noted that the random forest is a mature machine learning algorithm, which is good at processing multi-dimensional feature spaces and improves the prediction accuracy by integrating multiple decision trees. This algorithm can effectively process the features of complex interaction behaviors and classify large-scale destructive behaviors. Since the random forest can process historical behavior data and generate a judgment model, it is very suitable for judging whether an operation belongs to a destructive behavior.

[0059] The destruction target is the target object corresponding to the current destructive behavior; using the position of the arm, the visual feedback result, and the operation instruction as inputs, the trained LSTM model processes time series data to measure the current destructive behavior object, calculates the confidence score for each possible target object, and selects the one with the highest confidence as the destruction target. Among them, when the system registers that a target needs to be destroyed, the permission of the target will be opened.

[0060] LSTM is a neural network particularly suitable for processing time series data, which can capture the temporal dependence relationships in complex operations. In the present invention, the LSTM model can utilize continuous input data such as the position of the robotic arm, the visual feedback result, and the operation instruction to gradually analyze the target object corresponding to the current behavior. By calculating the confidence for each target object, the system can determine whether the current operation is directed at a potential destruction target.

[0061] It should also be noted that, for example, in a manufacturing workshop, an operator and a two-armed robot work together. The operator needs to command the robot to perform complex operation tasks, such as grasping an object in a specific area. However, the operator may accidentally fall or collide with the robot, resulting in incorrect actions of the robot.

[0062] For example:

[0063] Limb collision: The operator's arm or body accidentally touches the operating robotic arm, causing the robotic arm to deviate from its trajectory, perform incorrect grasping actions, and may even collide with other equipment, causing damage or danger.

[0064] Falls: If the operator accidentally falls while interacting with the robot, it may cause the robot to receive incorrect instructions, such as rapidly moving the robotic arm or grasping incorrectly, which may damage the equipment or the surrounding environment.

[0065] By the system identifying these behaviors, it can timely judge whether these abnormal behaviors will lead to operation failures or equipment damage, and perform control according to the permissions of the damaged targets. The identification of such behaviors generally depends on when interacting. Then, by identifying the interaction operation force, frequency, length of the interaction behavior (if it is an interface operation, this length is the sliding length of the interaction behavior in the interface) or duration, etc. Of course, this prediction result can only identify relatively obvious situations, and more detailed situations do not need to be identified, so as to ensure the freedom of the operator during interaction.

[0066] When designing the recognition of operation errors in the interaction system, overly precise recognition may lead to misjudgment, thus having a negative impact on the naturalness of the operation and the user experience. When identifying abnormal interaction behaviors, if the system precisely analyzes and judges every tiny detail, it may misjudge normal operations as error behaviors. The following are several key reasons why it is not easy to perform overly precise recognition, mainly to prevent misjudgment and maintain the freedom of interaction:

[0067] Prevent misjudgment and avoid interrupting the normal operation process

[0068] Overly precise recognition means that the system will strictly monitor all detailed operations, which may lead to normal interaction behaviors being misjudged as destructive behaviors. For example, during the interaction process, the operator may cause some slight changes, such as slight speed fluctuations or changes in the operation angle, due to inertia or minor adjustment actions. If the system strictly judges this, it may misjudge these normal operations as errors, resulting in unnecessary operation interruptions.

[0069] In the interface of the sliding operation, the user may quickly slide their finger to achieve a certain interaction purpose. If the system overly precisely monitors the sliding force or speed, it may misjudge this rapid operation as an out-of-control operation, thus triggering incorrect cut-offs or warnings and interrupting the user's normal operation process.

[0070] Natural fluctuations of interaction behaviors

[0071] In human-machine interaction, the operator's behaviors usually have natural fluctuations and flexibility. The operator cannot precisely maintain the stability of force, speed, or angle every time, and these minor changes usually do not affect the operation result. Therefore, if the system overly precisely identifies the force, frequency, and duration of interaction behaviors, it may interfere with these natural fluctuations and misjudge them as abnormal.

[0072] In actual operation, the operator may cause slight changes in the force or sliding length during the interaction due to fine-tuning of gestures or body postures. Overly precise recognition will identify these insignificant fine-tunings as potential misoperations, interfering with the normal workflow.

[0073] Prevent the system from restricting the freedom of operation

[0074] Overly strict recognition criteria will reduce the freedom of operation. The interaction system needs to find a balance between providing accurate misoperation detection and maintaining the flexibility of user operations. If the system's recognition mechanism is too precise, it may limit the user's operation method, forcing the user to complete operations at a specific speed, force, and path, which conflicts with the diversity of actual operations.

[0075] For the control of the robotic arm, if the system overly precisely requires the operator to always maintain a certain fixed operation force or angle, the user may feel restricted, resulting in reduced operation efficiency or increased psychological burden. The operator may need to quickly adjust actions during actual operation, and an overly precise system will interfere with this free adjustment.

[0076] Ignoring details can reduce the system's computational burden

[0077] The system must process a large amount of sensor data during the recognition process, including the force, sliding length, frequency, and duration of interaction operations, etc. Overly precise analysis will increase the system's computational burden. Especially in a real-time interaction environment, this burden will affect the overall performance and response time of the system. If the system only focuses on obvious abnormal behaviors and adopts a loose tolerance for smaller details, it can reduce the amount of computation and improve the system efficiency.

[0078] For a complex production task, the system needs to monitor the interactions between multiple operators and robots in real time. If every subtle operation detail has to be precisely monitored and analyzed, the system will consume a large amount of computational resources, which may cause the system to react sluggishly and be unable to make decisions in a timely manner. By reducing the detail analysis, the system can more effectively identify obvious abnormal behaviors and reduce resource consumption.

[0079] Avoid causing psychological burden to the operator

[0080] If the system overly precisely recognizes operation mistakes, the operator may feel restricted during the interaction and be afraid that every subtle operation will be misjudged as a mistake. This psychological pressure will affect the operator's work efficiency and may even cause the operator to be overly cautious when performing tasks, affecting the overall production efficiency.

[0081] When an operator controls a dual-arm robot to perform an assembly task, if the system makes strict requirements for every subtle part of the operation, the operator may be overly worried about misoperations, thus slowing down the operation speed and losing the fluency and efficiency of the operation.

[0082] Allowing a certain degree of operating freedom helps to adapt to various operating styles.

[0083] Each operator will have different operating habits and styles when interacting with the system. Some people like to quickly slide and operate, while some are used to slower and more precise operations. If the system's recognition is too precise to adapt to these differences in operating styles, it may reduce the user experience. Maintaining a certain degree of operating freedom and allowing the system to adapt to different styles of user behavior can enhance the flexibility of the system.

[0084] On a touch interface, some users may be used to quickly sliding for navigation, while other users may tend to slow and precise operations. An overly precise recognition mechanism may make inconsistent judgments for different operating styles, resulting in differences in user experience and reducing the adaptability of the operation.

[0085] S3: Transmit the operation instructions of the interaction behavior that is not initially recognized as a mistake to the execution part, and perform a secondary recognition of mistakes based on mechanical constraints in the execution part.

[0086] The mechanical constraints include: joint angle constraint: ensuring that the joint angles of each robotic arm do not exceed the designed range; speed and acceleration constraint: restricting the movement speed and acceleration of the robotic arm within a safe range; workspace constraint: keeping the operating range of the robotic arm within a predetermined workspace and not colliding with surrounding objects; force and torque constraint: during grasping or operation, the applied force and torque conform to the physical capabilities of the device; time constraint: the execution time of the operation cannot be too long or too short to ensure the task completion time.

[0087] θ 1 ∈ [ α min ,α max ]

[0088] v 2 ≤β max

[0089] a 3 ≤γ max

[0090] P 4 ∈ [ δ min ,δ max ]

[0091] d5 ( O 6 ) ≥∈ safe

[0092] F 7 ≤ζ max

[0093] T 8 ≤η max

[0094] t 9 ∈ [ θ min ,θ max ]

[0095] T 10 ∈ [ ι min ,ι max ]

[0096] Among them, θ 1 represents the current joint angle of the robotic arm 1, α min represents the minimum limit of the joint angle α max represents the maximum limit of the joint angle, v 2 represents the movement speed of the robotic arm 2, β max represents the maximum allowable speed, a 3 represents the acceleration of the robotic arm 3, γ max represents the maximum allowable acceleration, P 4 represents the end position of the robotic arm 4, δ min represents the minimum position of the workspace δ max represents the maximum position of the workspace, d 5( O 6) represents the distance between the robotic arm 5 and the obstacle 6, ∈ safe represents the safety distance, F 7 represents the grasping force of the robotic arm 7, ζ max represents the maximum allowable grasping force, T 8 represents the torque applied by the robotic arm 8, η max represents the maximum allowable torque, t 9 represents the time taken for the robotic arm 9 to complete the operation, θ min represents the minimum allowable operation time θ max represents the maximum allowable operation time, T 10 represents the temperature of the environment, ι min represents the minimum allowable environmental temperature, ι max represents the maximum allowable environmental temperature.

[0097] The system monitors the joint angles of the robotic arm in real time to ensure its operation within the allowable physical range; if it detects that the range exceeds the mechanical constraints, it is determined as a malfunction and the operation is immediately stopped.

[0098] S4: Through the analysis of the behavior of the execution part, the final identification of the malfunction is completed.

[0099] The final identification includes constructing intention tags for the intention concepts of each step and constructing an intention tag network based on the causal relationship between each intention; in the network, the operation intentions with a sequential operation relationship are represented by directed connections, and the connection direction is from the intention tag of the previous step to the intention tag of the subsequent step; the operation intentions without a sequential operation relationship are represented by undirected connections in the network.

[0100] It should be noted that by using directed connections in the network, the system can clarify the sequence of each operation. This sequence is crucial, especially in complex tasks where certain steps must be continued after the previous operation is completed. The directed connections ensure that the system can track the progress of the operation and avoid malfunctions or failures caused by incorrect step sequences. In some cases, there is no strict sequence between operation steps and they can be carried out in parallel. By representing these parallel operation intentions with undirected connections in the network, the system can flexibly handle the situation of multiple operations being carried out in parallel. For example, in a robot assembly task, certain steps can be executed simultaneously. This design allows the system to identify which operations can be processed in parallel to optimize task execution. Through the intention tag network, the system can generate an execution path from the initial state. Based on the operation intentions and connection relationships in the network, the system can track all possible execution paths from the starting point to the final goal of the operation to ensure the sequence and integrity of the operation. If the system detects a behavior deviating from the path during the execution process, it can be immediately identified as a malfunction. Through the pre-designed intention tag network of the system, it can effectively track all intention tags in the execution path until the final operation intention is realized. Each operation intention has a clear directionality, which helps to prevent path interruptions or incorrect executions in the execution of complex tasks.

[0101] Among them, the intention marking network is a pre-designed network architecture, aiming to plan all possible operation steps and their logical sequences according to the requirements of the task. Through predefined intention markings and connection relationships, the system can effectively identify and execute each step in the task. This design establishes a clear framework for the system to execute tasks, allowing the system to execute operations orderly according to task requirements, rather than randomly selecting or skipping steps. During the development stage, the operator or system designer manually defines the intention markings for each operation step according to the specific requirements of the task, and sets the direction of the connection according to the logical sequence and operation requirements of the task. This pre-designed network by humans can be customized for different tasks to ensure that the system can follow the correct path when executing tasks.

[0102] An intention marking refers to the logical expression of each specific operation step, which represents the operation goal of the operator or system when executing a specific step. For example, the operator may wish to "grab an object", "move the robotic arm to a specified position", or "rotate the robotic arm". Each operation intention is assigned a marking for identifying and tracking these operations in the network.

[0103] Specifically: The role of intention markings: Intention markings are the abstract representations of each step in task execution. The system uses these markings to track the progress of operations and ensure that operations are executed according to the expected goals. Each marking represents the goal or intention of a task or operation.

[0104] The input is the continuous action information of a dual-arm robot. An LSTM model trained for behavior recognition of the executed actions is used to predict the intention; the prediction result of the current behavior is output. Among them, the prediction result of the current behavior is an intention marking.

[0105] Use the depth-first search algorithm to traverse the network to obtain all possible paths. Among them, the paths are generated according to the connection direction in the network. The traversal starting point is the initial state of the robot, and the traversal end point is the final intention of the operation.

[0106] Arrange the intention marking predicted according to the current behavior and other behaviors that have occurred in the order of occurrence to obtain the intention marking sequence XL; match XL with all possible paths. When the sequence of XL is a part of the intention marking sequence of path L, it is determined that XL matches L. If there is no matching path, the recognition result is determined to be a mistake; if there is a matching path, it is determined to be without mistake.

[0107] When the recognition result of the final recognition is a mistake, send the intention marking sequence XL to the management for permission verification, and select whether to execute according to the feedback of the management; if the feedback result is to execute, correct the recognition result to be without mistake; if the feedback result is not to execute, maintain the original recognition result.

[0108] That is to say, by tracking the real-time sequence of intention tags, the system can dynamically adjust the operation process according to the actual operations. The system arranges the predicted intention tags of the current behavior with the sequence of behavior tags that occurred before to obtain the current operation progress (intention tag sequence XL). This provides a mechanism for comparison at any time. The system can match XL with all the paths traversed by DFS to check whether the current operation conforms to the predetermined plan. This enables the system to dynamically track the current operation process and promptly detect whether the operation is consistent with the predetermined path. If it is found that the current operation deviates from the expected path, the system can immediately identify it and take corresponding measures. This design not only improves the real-time response ability of the system but also ensures that the operation will not fail due to path deviation. Through path matching, the system can determine whether the operation is within the expected path or has deviated. However, if no matching part of the operation sequence XL can be found among all possible paths, the system identifies it as a mistake. To avoid interrupting the operation or over-intervening due to misjudgment, the system designs a flexible feedback mechanism. That is, when a mistake is identified, the system will send the intention tag sequence XL to the management staff for permission verification.

[0109] The management staff can judge the operation according to the actual situation. Especially for some specific scenarios, there may be temporary adjustments to the operation steps or non-standard operations. Through the feedback of the management staff, the system can choose whether to continue the operation according to the actual situation. This design not only ensures the safety of the operation but also avoids the problem of misjudgment caused by the system being too strict.

[0110] S5: After the identification is completed, perform execution isolation on each result identified as a mistake respectively. The execution part refuses to execute the interactive behavior of the mistake to prevent operation mistakes.

[0111] When the system identifies the result of an operation mistake, the system immediately pauses the current operation, stops the movement of the robotic arm, but does not cut off the control signal of the interaction, and restores the dual-arm robot to the safe state without mistakes at the previous moment.

[0112] By continuously receiving the control instructions of the interaction at the front end and continuously identifying operation mistakes in the control instructions, if the control signal is restored to a mistake-free state, the operation continues.

[0113] The instant reaction mechanism ensures that at the moment when a mistake is detected, the system can promptly abort the current task to avoid more serious consequences caused by misoperation or operation mistakes. For example, if the robotic arm is performing a fine grasping task and the operator inputs a wrong movement instruction, the system can immediately stop the movement of the robotic arm after identifying the instruction error, preventing the robotic arm from failing to grasp or damaging the target object.

[0114] By restoring the dual-arm robot to the safe state at the last moment without error, the system can ensure that the robot arm returns to a verified, non-dangerous state. This design ensures that after an error occurs, the device can evacuate from the dangerous action and enter a known safe operating state to prevent the accident from continuing to expand.

[0115] Restoring to a safe state means restoring the system to the last verified, correct operating state, which not only protects the operator and equipment, but also provides a stable starting point for subsequent task execution. For example, after a collision or misoperation is detected, the robot arm will be adjusted back to its safe initial position to prevent the robot arm from remaining in a dangerous area.

[0116] After the operation error is identified and suspended, the system will continue to identify the control signals received in real time and determine whether the error continues. Through this continuous monitoring, the system can dynamically determine when to resume operation, thereby avoiding the system being suspended for a long time.

[0117] This design provides flexibility for the operation process: when the system detects that the operation has returned to normal, it can automatically continue the suspended operation without completely suspending the task process. This ensures the efficiency and continuity of the system and avoids frequent interruptions. For example, when the robot arm is paused due to an error, the operator can re-enter the correct operation instructions, and the system will recognize and judge the correctness of these instructions in real time. When it is confirmed that there is no error, the robot arm will automatically resume and continue to perform the task.

[0118] On the other hand, this embodiment also provides an operation error identification system based on human-computer interaction, which includes:

[0119] The collection unit collects human interaction behaviors through the front end.

[0120] The recognition unit performs behavioral analysis on the interactive behavior and performs initial recognition of errors; transmits the operation instructions of the interactive behavior that are not determined to be errors in the initial recognition to the execution part, and performs secondary recognition of errors based on mechanical constraints in the execution part; and completes the final recognition of errors by analyzing the behavior of the execution part.

[0121] After the identification is completed, the isolation unit performs isolation on each result identified as an error, and the execution part refuses to execute the interaction behavior of the error to prevent operation errors.

[0122] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs.

[0123] The logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch instructions from the instruction execution system, apparatus, or device and execute the instructions), or used in combination with these instruction execution systems, apparatuses, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0124] More specific examples (non-exhaustive list) of computer-readable media include the following: electrical connection parts with one or more wirings (electronic devices), portable computer disk cartridges (magnetic devices), random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memories), fiber optic devices, and portable compact disc read-only memories (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, then editing, interpreting, or otherwise processing it as appropriate, and then storing it in a computer memory.

[0125] It should be understood that each part of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits with logic gate circuits for implementing logical functions on data signals, application specific integrated circuits with suitable combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0126] Embodiment 2, an embodiment of the present invention, provides a method for identifying operation errors based on human-computer interaction. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0127] The experiment set up three different technical solutions for comparison: Prior Art 1, Prior Art 2, and the method of the present invention, and tested their performances in different environments.

[0128] Prior Art 1: Based on a simple rule engine, use preset parameter thresholds to judge operation behaviors, and the system identifies operation errors based on limited sensor data such as position and force feedback. When the operation exceeds the threshold, the system stops the operation, but there is no further automatic recovery mechanism.

[0129] Prior Art 2: Adopt an operation recognition system based on a traditional machine learning model, and use a support vector machine (SVM) model to identify operation behaviors. This system relies on a large training set, but lacks continuous action time series analysis and does not deeply process complex behavior patterns.

[0130] The experimental scenario simulated an automatic assembly task in an industrial production environment, where an operator cooperated with a two-armed robot to complete a series of complex grasping and assembly tasks. The test environments included:

[0131] Normal operation environment: Without emergencies, the operation proceeds as planned.

[0132] Sudden error environment: The operator has an accident, such as falling, or the arm colliding with the robot, resulting in the interruption of the operation.

[0133] Uncertain operation environment: The operator temporarily changes the operation process, and the instructions are inconsistent with the plan.

[0134] The present invention and the two prior arts were respectively run in the three environments, and the recorded data are shown in Table 1.

[0135] Table 1 Experimental data table

[0136]

[0137] As can be seen from the tabular data, the method of the present invention has shown excellent performance in different test environments, especially having obvious advantages in key indicators such as error recognition rate, isolation reaction time, recovery time, and error rate.

[0138] In a normal operating environment, the operation completion rate of the present invention is as high as 98.7%, significantly higher than 90.2% of the prior art 1 and 93.5% of the prior art 2. This indicates that the three-level recognition mechanism of the present invention can efficiently detect and process complex interaction operations to ensure that tasks are completed as expected. In contrast, due to the relatively low error recognition accuracy of the prior art 1 and 2, the operation completion rate is relatively low.

[0139] In terms of the error recognition rate, the recognition rate of the method of the present invention reaches 96.2%, far higher than 75.3% of the prior art 1 and 81.2% of the prior art 2. This improvement benefits from the combined use of the random forest model and the LSTM model in the present invention, enabling the system to process complex time series data and accurately judge operation behaviors. The prior art 1 and 2 rely on rule engines and traditional SVM models and fail to comprehensively capture error behaviors in complex interactions, resulting in a relatively low error recognition rate.

[0140] In terms of the isolation reaction time, the isolation reaction time of the method of the present invention is only 50 milliseconds, significantly better than 150 milliseconds of the prior art 1 and 120 milliseconds of the prior art 2. This shows that the present invention can detect errors and trigger operation isolation in a shorter time, effectively avoiding potential damage to equipment or operators caused by operation errors. The isolation reaction time of the prior art is relatively long and may not be able to prevent the further expansion of operation errors in a timely manner.

[0141] In a sudden error environment, the error recognition rate of the present invention reaches 90.7%, far higher than 65.1% of the prior art 1 and 72.6% of the prior art 2. This indicates that the error recognition mechanism of the present invention can effectively respond to unexpected events such as an operator falling or colliding. Due to the lack of deep time series analysis capabilities in the prior art 1 and 2, they perform poorly in dealing with sudden errors, resulting in a low error recognition rate.

[0142] In terms of operation recovery efficiency, the method of the present invention performs excellently, and its recovery time remains within 2 seconds in all test environments, while the recovery times of the prior art 1 and 2 are 5.0 seconds and 3.9 seconds respectively. This shows that the present invention can recover to a safe state and continue operations more quickly, effectively improving the task execution efficiency of the system.

[0143] In terms of the error rate, the error rate of the method of the present invention is controlled between 1.2% and 2.0% under different test environments, far lower than 8.7% to 14.8% of the prior art 1 and 6.2% to 10.9% of the prior art 2. This indicates that the multi-level error detection and isolation mechanism of the present invention can significantly reduce operation errors and ensure the high reliability of the system.

[0144] In summary, the method of the present invention is superior to the prior art in terms of various indicators such as operation completion rate, error recognition rate, isolation reaction time, and error rate. Since the prior art 1 uses a rule engine and fails to handle complex operation behaviors, resulting in poor recognition accuracy, slow reaction, and long recovery time. Although the prior art 2 uses an SVM model, it lacks in-depth analysis of complex temporal behaviors, resulting in deficiencies in complex interaction environments. The present invention combines a random forest and an LSTM model to ensure the efficient and safe operation of the system in different environments, demonstrating strong innovation and advantages.

[0145] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered by the scope of the claims of the present invention.

Claims

1. A method for identifying operation errors based on human-computer interaction, characterized in that: include: Collect human interaction behaviors through the front end; Conducting behavioral analysis on the interactive behavior to perform initial identification of errors; The operation instructions of the interactive behaviors that are not judged as errors in the initial recognition are transmitted to the execution part, and the errors are secondary recognized based on the mechanical constraints in the execution part; By analyzing the behaviors of the execution part, the final identification of the error is completed; After the recognition is completed, each result that is recognized as an error is isolated from execution, and the execution part refuses to execute the interaction behavior with error to prevent operation errors.

2. The method for identifying operation errors based on human-computer interaction according to claim 1, characterized in that: The interactive behavior includes issuing operating instructions to the dual-arm robot and interacting through the actions fed back by the robot.

3. The method for identifying operation errors based on human-computer interaction according to claim 2, characterized in that: The initial identification includes training a random forest model based on destructive behavior patterns marked in historical data, and using the random forest model to predict whether the current behavior performs a destructive behavior; When a sabotage behavior is identified, the target of the sabotage behavior is further identified. If the permission for the sabotage target is enabled, the initial identification result is determined to be normal. If the permission to destroy the target is not enabled, the initial recognition result is considered to be a mistake; The sabotage target is the target object corresponding to the current sabotage behavior; Using the arm position, visual feedback results and operation instructions as input, the trained LSTM model processes the time series data to measure the current destructive behavior target, calculates the confidence score for each possible target object, and selects the one with the highest confidence as the destructive target; Among them, when the system registration requires destruction of the target, the target's permissions will be opened.

4. The method for identifying operation errors based on human-computer interaction according to claim 3, characterized in that: The mechanical constraints include: joint angle constraints: ensuring that the joint angle of each robot arm does not exceed the design range; speed and acceleration constraints: limiting the movement speed and acceleration of the robot arm within a safe range; workspace constraints: the operating range of the robot arm remains within the predetermined workspace and does not collide with surrounding objects; force and torque constraints: during the grasping or operation process, the applied force and torque are consistent with the physical capabilities of the device; time constraints: the execution time of the operation cannot be too long or too short to ensure the task completion time; θ1∈ [ a min ,a max v2≤β max a3≤γ max P4∈ [ d min ,d max d5 ( O6 ) ≥ε safe F7≤ζ max T8≤η max t9∈ [ i min ,i max T 10 ∈ [ ι min ,ι max Among them, θ1 represents the current joint angle of robot arm 1, α min Represents the minimum limit of the joint angle α max represents the maximum limit of the joint angle, v2 represents the movement speed of robot 2, β max represents the maximum speed allowed, a3 represents the acceleration of robot arm 3, γ max represents the maximum acceleration allowed, P4 represents the end position of the robot arm 4, δ min represents the minimum position of the workspace δ max represents the maximum position of the workspace, d 5( O 6) represents the distance between the robot arm 5 and the obstacle 6, ∈ safe represents the safety distance, F7 represents the grasping force of the robot arm 7, ζ max represents the maximum allowable grasping force, T8 represents the torque applied by the robot arm 8, η max represents the maximum torque allowed, t9 represents the time it takes for the robot arm 9 to complete the operation, θ min The minimum allowed operation time θ max Indicates the maximum allowed operation time, T 10 represents the temperature of the environment, ι min Indicates the minimum allowable ambient temperature, ι max Indicates the maximum allowable ambient temperature.

5. The method for identifying operation errors based on human-computer interaction as claimed in claim 4, characterized in that: The secondary identification includes that the system monitors the joint angles of the robotic arm in real time to ensure that it operates within the permitted physical range; if it is detected that it exceeds the range of the mechanical constraints, it is determined to be an erroneous behavior and the operation is stopped immediately.

6. The method for identifying operation errors based on human-computer interaction according to claim 5, characterized in that: The final identification includes constructing an intention tag for the intention concept of each step, and constructing an intention tag network according to the causal relationship between each intention; In the network, the operation intention with a sequential operation relationship is represented by a line with a direction, and the direction of the line is from the intention mark of the preceding step to the intention mark of the succeeding step; in the network, the operation intention without a sequential operation relationship is represented by a line without a direction; The input is the continuous action information of the dual-arm robot, and the LSTM model trained for the execution behavior recognition is used to predict the intention; Output the predicted result of the current behavior; Among them, the predicted result of the current behavior is an intention marker; Use the depth-first search algorithm to traverse the network and obtain all possible paths, where the paths are generated according to the connection direction in the network, the traversal starting point is the initial state of the robot, and the traversal end point is the final intention of the operation; Arrange the intention tags predicted based on the current behavior and other behaviors that have occurred in the order of occurrence to obtain the intention tag sequence XL; match XL with all possible paths, and when the sequence of XL is part of the intention tag sequence of path L, it is determined that XL matches L; If there is no matching path, the recognition result is judged to be an error; if there is a matching path, it is judged to be no error; When the final recognition result is an error, the intention mark sequence XL is sent to the administrator for authority check, and the administrator chooses whether to execute it based on the feedback; if the feedback result is to execute it, the recognition result is corrected to be error-free; if the feedback result is not to execute it, the original recognition result is maintained.

7. The method for identifying operation errors based on human-computer interaction according to claim 6, characterized in that: The said performing isolation for each result identified as an error respectively includes that when the system identifies the result of an operation error, the system immediately pauses the current operation, stops the movement of the robot arm, but does not cut off the interactive control signal, and restores the dual-arm robot to a safe state without an error at the last moment; The front end continuously receives interactive control instructions, continuously identifies operation errors in the control instructions, and continues to execute the operation if the control signal recovers to a non-error state.

8. An operation error identification system based on human-computer interaction using the method according to any one of claims 1 to 7, characterized in that: The collection unit collects human interaction behaviors through the front end; An identification unit, which performs behavior analysis on the interaction behavior and performs initial identification of errors; The operation instructions of the interactive behaviors that are not determined as errors in the initial recognition are transmitted to the execution part, and the errors are secondary recognized based on mechanical constraints in the execution part; and the final recognition of the errors is completed by analyzing the behaviors of the execution part; After the identification is completed, the isolation unit performs isolation on each result identified as an error, and the execution part refuses to execute the interaction behavior of the error to prevent operation errors.

9. A computer device comprising: Memory and processor; The memory stores a computer program, characterized in that: when the processor executes the computer program, the steps of the operation error identification method based on human-computer interaction are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the operation error identification method based on human-computer interaction are implemented.

Images