Robot failure analysis method, device, and storage medium
By acquiring image and sensor data during robot tasks and combining them with a large language model, the accuracy of environmental and parameter changes is solved, enabling faster and more accurate fault identification and task optimization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- PING AN TECH (SHENZHEN) CO LTD
- Filing Date
- 2025-06-10
- Publication Date
- 2026-08-04
AI Technical Summary
In existing technologies, the accuracy of robot fault analysis is poor, and the lack of in-depth understanding of robot behavior leads to inaccurate fault analysis.
By acquiring image and sensor data from the robot during task execution, analyzing environmental and parameter changes, and combining this with a large language model, the cause of the failure can be determined.
It improves the accuracy of robot fault analysis, enabling faster and more accurate identification of fault causes, and enhancing the reliability and efficiency of robot task execution.
Smart Images
Figure CN120395886B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a robot fault analysis method, device and storage medium. Background Technology
[0002] With the rapid development of robotics technology, robots are being applied more and more widely in various industries and fields. For example, in the healthcare sector, robots can be used for drug dispensing and surgical assistance, effectively improving hospital operational efficiency and the quality of medical services. In the financial sector, robots can be used for customer reception, intelligent guidance, and business processing assistance in bank lobbies, effectively improving the level of automation in financial services. The efficient operation of a robot system depends on the accuracy and stability of the robot's task execution, but robots often face various malfunctions during task execution, such as hardware damage, software crashes, and sensor failures.
[0003] To address the aforementioned issues, large language models can be used to perform fault analysis on the robot's task execution information. However, the fault analysis returned by large language models is limited to the semantic level, lacking a deep understanding of the robot's behavior, and is largely based on fault pattern recognition from previously trained data, resulting in poor accuracy in fault analysis.
[0004] Therefore, improving the accuracy of robot fault analysis is an urgent problem to be solved. Summary of the Invention
[0005] The main objective of this application is to provide a robot fault analysis method, device, and storage medium, which aims to improve the accuracy of robot fault analysis.
[0006] In a first aspect, this application provides a robot fault analysis method, including:
[0007] The task data collected by the robot when it performs multiple sub-tasks in the task to be analyzed includes multiple image data and multiple sensor data.
[0008] Based on multiple image data corresponding to each of the subtasks, the target subtask that failed to execute is determined from the multiple subtasks;
[0009] Based on multiple image data corresponding to each of the target sub-tasks, determine the environmental change data when the robot performs each of the target sub-tasks;
[0010] Based on multiple sensor data corresponding to each of the target sub-tasks, determine the parameter change data when the robot performs each of the target sub-tasks;
[0011] Based on the environmental change data and parameter change data when the robot performs each of the target sub-tasks, the cause of failure for each target sub-task is determined.
[0012] Secondly, this application also provides a robot fault analysis method, including:
[0013] Acquire task data corresponding to each of the multiple sub-tasks in the task to be analyzed performed by the robot; wherein, the task data includes multiple image data and multiple sensor data;
[0014] Based on multiple image data corresponding to each sub-task, determine the environmental change data corresponding to each sub-task;
[0015] Based on multiple sensor data corresponding to each of the target sub-tasks, determine the parameter change data corresponding to each of the sub-tasks;
[0016] The environmental change data and parameter change data corresponding to the multiple sub-tasks are input into the large language model so that the large language model can determine the target sub-task that failed and the cause of the failure of the target sub-task based on the environmental change data and parameter change data of each sub-task.
[0017] Thirdly, this application also provides a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the robot fault analysis method as described above.
[0018] Fourthly, this application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the robot fault analysis method described above.
[0019] This application provides a robot fault analysis method, device, and storage medium. The method involves acquiring task data collected by the robot while it executes multiple sub-tasks within a task to be analyzed. This task data includes multiple image data and multiple sensor data. Based on the multiple image data corresponding to each sub-task, the method identifies the target sub-task that failed. Based on the multiple image data corresponding to each target sub-task, the method determines the environmental change data during the robot's execution of each target sub-task. Based on the multiple sensor data corresponding to each target sub-task, the method determines the parameter change data during the robot's execution of each target sub-task. Based on the environmental change data and parameter change data during the robot's execution of each target sub-task, the method determines the cause of failure for each target sub-task. By retrospectively analyzing the image data characterizing the environmental conditions and environmental changes during the robot's execution of sub-tasks, as well as the sensor data characterizing the parameter changes, the method comprehensively considers the impact of environmental factors and robot-specific factors on task execution. This allows for faster and more accurate determination of robot fault causes, significantly improving the accuracy of robot fault analysis. Attached Figure Description
[0020] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0021] Figure 1 This application provides a flowchart illustrating the steps of a robot fault analysis method.
[0022] Figure 2 for Figure 1 A flowchart illustrating the sub-steps of the robot fault analysis method in the diagram;
[0023] Figure 3 A schematic diagram of a scenario for the robot fault analysis method provided in the embodiments of this application;
[0024] Figure 4 A flowchart illustrating the steps of a robot fault analysis method provided in another embodiment of this application;
[0025] Figure 5 A schematic block diagram of a robot fault analysis device provided in this application embodiment;
[0026] Figure 6 This is a schematic block diagram of the structure of a computer device provided in an embodiment of this application.
[0027] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0029] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.
[0030] This application provides a robot fault analysis method, device, and storage medium. The robot fault analysis method can be applied to a terminal device or a server. The terminal device can be an electronic device such as a mobile phone, tablet computer, laptop computer, desktop computer, personal digital assistant, or wearable device. The server can be a single server or a server cluster composed of multiple servers.
[0031] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.
[0032] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of a robot fault analysis method provided in an embodiment of this application.
[0033] like Figure 1 As shown, the robot fault analysis method includes steps S101 to S105.
[0034] Step S101: Obtain task data collected by the robot when it performs multiple sub-tasks in the task to be analyzed. The task data includes multiple image data and multiple sensor data.
[0035] The task to be analyzed can be either successfully executed or unsuccessfully executed. A task to be analyzed is an operational process with a clear objective, such as picking up an item from a table. Subtasks are the basic operational units that make up the task to be analyzed, responsible for completing a specific step or stage in the operational process of the task. Examples include identifying and locating a target object on the table, planning the movement path of a robotic arm, controlling the robotic arm to move to the target position, performing a grasping operation, and moving the object to a designated area. It is understood that the successful execution of the task to be analyzed does not equate to the successful execution of all its subtasks, and the failure of the task to be analyzed does not equate to the failure of any of its subtasks. Specifically, in some cases, one or more subtasks within the task to be analyzed may fail; in other cases, all subtasks within the task to be analyzed may execute successfully.
[0036] Image data in the task data can be obtained from images taken by the robot during the execution of each sub-task. Images can be acquired through imaging devices mounted on the robot itself or external imaging devices deployed in the task environment surrounding the robot. Images can include environmental images, target object images, and sub-task execution status images. Environmental images record the working environment during task execution and can include images of surrounding objects, obstacles, and the operating area. Target object images represent objects or scenes related to the task objectives of each sub-task, helping the robot with identification and localization. Task execution status images show the robot's state while performing a specific sub-task, such as the position of the robotic arm or the state of an object being grasped.
[0037] The sensor data in the task data can be various sensory data collected by the robot during the execution of each sub-task. Sensor data can come from different sensors mounted on the robot, providing detailed information about the environment, objects, and the robot's state. Specifically, sensor data can include position and attitude sensor data providing information such as the robot's position, speed, and angle changes; force and torque sensor data providing information about the contact strength between the robot and objects; and tactile sensor data detecting contact between the robot and its environment, and so on.
[0038] It should be noted that, in order to further ensure the privacy and security of the aforementioned target watermark and related information, the aforementioned target watermark and related information can also be stored in a node of a blockchain. The technical solution of this application can also be applied to adding other data files stored on the blockchain. The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.
[0039] In one embodiment, before acquiring task data collected when the robot executes multiple sub-tasks in the task to be analyzed, including multiple image data and multiple sensor data, the method further includes: acquiring multiple image data and multiple sensor data corresponding to the successful execution of multiple sub-tasks; and constructing a task database based on the multiple image data and multiple sensor data corresponding to the successful execution of multiple sub-tasks.
[0040] Understandably, building a task database can provide a reliable data foundation for subsequent fault backtracking and optimization of correction strategies.
[0041] Step S102: Based on the multiple image data corresponding to each subtask, determine the target subtask that failed to execute from the multiple subtasks.
[0042] Each image data point can be used to characterize the environmental conditions at a specific point in time during the execution of each subtask. Under normal circumstances, without external environmental factors (such as temporary obstacle occlusion), the image data corresponding to each successful execution of a subtask should be consistent. For example, in the multiple image data points corresponding to each successful execution, features such as the positions of objects in the environment, the angle of the robot arm, and surrounding obstacles should exhibit consistent patterns of change. However, if these consistent characteristics deviate significantly during a particular execution of a subtask, the image data will also show abnormal changes. These abnormal changes usually indicate a problem during execution, potentially leading to the failure of the subtask. Therefore, by comparing and analyzing one or more of the multiple image data points corresponding to each subtask, the target subtask that failed can be effectively identified.
[0043] In one embodiment, determining the target subtask that failed to execute based on multiple image data corresponding to each subtask includes: obtaining first image data from the task data of each subtask to characterize the environmental situation at the end of the subtask execution, and obtaining second image data from the end of the subtask execution to characterize the environmental situation at the end of the subtask execution to characterize the successful execution; matching the first image data and the second image data corresponding to each subtask to obtain the matching degree corresponding to each subtask; and determining the subtask with a matching degree less than a preset matching degree threshold as the target subtask that failed to execute.
[0044] The second image data can utilize a pre-defined task database, while both the first and second image data can be based on one or more sources. It should be noted that, to ensure the accuracy and effectiveness of the analysis, the first and second image data should be equal in quantity. This ensures that the environmental conditions of each subtask can be compared one-to-one with the corresponding successfully executed environmental conditions, avoiding misjudgments caused by data inconsistencies.
[0045] Specifically, the first image data corresponding to each subtask is matched with the second image data. This can be done by using image matching algorithms (such as image feature extraction, similarity calculation, etc.) to calculate the matching degree for each subtask. The matching degree reflects the similarity between the actual execution result and the expected successful execution result of the subtask. For all subtasks, if their matching degree is less than a preset matching degree threshold, then the subtask is identified as a target subtask that failed to execute. The matching degree threshold can be set according to actual needs; a matching degree below this threshold indicates that an anomaly occurred during the execution of the subtask that could cause the subtask to fail.
[0046] In one embodiment, acquiring second image data representing the successful execution of each subtask at the end of its execution includes: acquiring multiple environmental image data representing the successful execution of multiple subtasks; constructing a task database based on the multiple environmental image data corresponding to the multiple subtasks; and acquiring the second image data corresponding to each subtask from the task database.
[0047] The multiple environmental image data used to characterize successful execution can be obtained by filtering all image data acquired during the successful execution of the subtask.
[0048] Step S103: Based on multiple image data corresponding to each target sub-task, determine the environmental change data when the robot performs each target sub-task.
[0049] Each image data point reflects the environmental state at a specific point in time when the robot performs each sub-task. Conversely, multiple image data points that are sequential in time reflect the environmental changes throughout the entire process of the robot performing the sub-task. Therefore, by comparing and analyzing multiple image data points corresponding to each sub-task, changes in relevant entities in the environment during the robot's execution of the sub-task can be identified and determined. These changes can include: the presence and disappearance of objects, changes in object positions, changes in the robot's position, and whether or not the target object is identified and grasped. By recording these changes in detail, environmental change data corresponding to each sub-task can be obtained. Environmental change data can include the changing objects, the type of change, the location of the change, and the time of the change.
[0050] Step S104: Based on multiple sensor data corresponding to each target sub-task, determine the parameter change data when the robot performs each target sub-task.
[0051] Sensor data can be raw data collected from various sensors installed on the robot. It's understandable that, given the diverse types of sensors used in the robot, the collected data is not limited to reflecting changes in the robot's own state, but can also reflect changes in the surrounding environment. Specifically, sensors can be position sensors, force / torque sensors, temperature sensors, tactile sensors, and acoustic sensors, etc. Parameter change data can include changes in position parameters (such as the robot's current position, orientation, and speed), force / torque parameters (such as the force or torque applied when the robot performs grasping, pushing, or other interactive operations), state parameters (such as the robot's working state during execution), and so on. It's understandable that by collecting and analyzing this sensor data, and identifying key parameter changes when the robot performs each target sub-task, a strong basis can be provided for subsequent decision-making and task execution.
[0052] Step S105: Based on the environmental change data and parameter change data when the robot performs each target sub-task, determine the cause of failure for each target sub-task.
[0053] When a robot performs a task, the success or failure of the task is closely related to the environmental state and the robot's own parameter state. Environmental change data, associated with the robot's environmental state, reflects the dynamic changes in the surrounding environment during task execution, such as obstacle movement and target object position shifts. Meanwhile, parameter change data, associated with the robot's own parameter state, reflects the robot's own state and the process of its actions, such as position, velocity, and torque. Therefore, when a target sub-task fails, by retrospectively analyzing data from both the environmental and parameter change dimensions, it is possible to identify which specific state change and / or abnormal parameter fluctuation caused the sub-task's failure. This allows for precise determination of the cause of the sub-task failure, effectively improving the accuracy of robot fault analysis.
[0054] Taking the application of robots in the healthcare field as an example, let's assume the robot's task is to distribute medicine, and the target sub-task is "grabbing medicine." The specific analysis process of the environmental and parameter change data for this target sub-task is as follows: Analysis of the parameter change data for this target sub-task reveals that the image data shows the target medicine's position is offset and not in the expected area. Further analysis of the parameter change data for this target sub-task reveals that the force sensor data is normal (gradually increasing) before grasping, but the force value suddenly fluctuates abnormally during the grasping process, indicating abnormal contact of the robot's end effector.
[0055] Therefore, it is inferred that the failure of the target sub-task to grasp the target may be caused by both the movement of the object's position and the mechanical grasping deviation, thus determining the cause of the failure of the target sub-task.
[0056] In one embodiment, such as Figure 2 As shown, step S105 includes sub-steps S1051 to S1054.
[0057] Sub-step S1051: Obtain target environment change data and target parameter change data when the robot successfully executes the target sub-task.
[0058] The target environment change data and target parameter change data can be obtained through a pre-built task database. The construction of this task database can be referred to the construction of the task database in the above embodiments. It should be noted that the task database in this application can be one or multiple, as long as it can meet the needs of data comparison analysis and fault backtracking, which will not be elaborated here.
[0059] Sub-step S1052: Compare the environmental change data when the robot performs each target sub-task with the target environment change data, and determine the first sub-fault cause of the target sub-task execution failure based on the comparison results.
[0060] The environmental change data represents the environmental changes when the target sub-task fails, while the target environmental change data represents the environmental changes when the target sub-task succeeds. The differences between the two are identified by comparing the environmental change data and the target environmental change data. Specifically, the comparison between the environmental change data and the target environmental change data can be achieved using traditional image processing algorithms, such as image difference comparison methods and feature point matching comparison methods; deep learning models, such as convolutional neural networks and generative adversarial networks; or machine learning methods, such as support vector machines and decision trees.
[0061] In some cases, discrepancies exist between environmental change data and target environment change data, indicating that changes in the external environment caused the target subtask to fail. Further analysis of these data differences can pinpoint the primary cause of the subtask's failure. For example, a new obstacle might have appeared during the subtask's execution, preventing it from following its intended steps and ultimately leading to failure. Conversely, in some cases, there are no discrepancies between environmental change data and target environment change data, indicating that the subtask's failure is unrelated to the external environment.
[0062] In one embodiment, the environmental change data includes multiple first image data, and the target environmental change data includes multiple second image data. Comparing the environmental change data during the robot's execution of each target sub-task with the target environmental change data, and determining the first sub-fault cause for the failure of the target sub-task based on the comparison results, includes: comparing the multiple first image data corresponding to each target sub-task with the multiple second image data to obtain multiple image comparison data corresponding to each target sub-task; and analyzing the first fault cause corresponding to each target sub-task based on the multiple image comparison data corresponding to each target sub-task.
[0063] For example, suppose the environmental change data includes first image data A1-E1, the target environmental change data includes second image data A2-E2, and there is a one-to-one correspondence between the acquisition time of the images corresponding to the first image data A1-E1 and the images corresponding to the second image data A2-E2.
[0064] By comparing these two sets of image data, it was determined that there is a difference between the first image data D1 and the second image data D2. Further comparison revealed that an object P, which is not present in the second image data D2, appeared in the first image data D1. Therefore, it can be determined that the first cause of failure in the target sub-task is the appearance of an obstacle.
[0065] Sub-step S1053: Compare the parameter change data when the robot executes each target sub-task with the target parameter change data, and determine the second sub-fault cause of the target sub-task execution failure based on the comparison results.
[0066] Among them, parameter change data is used to characterize the changes in robot parameters when the target sub-task fails, while target parameter change data is used to characterize the changes in robot parameters when the target sub-task succeeds. The specific process and explanation of determining the cause of the second sub-fault by comparing the parameter change data and the target parameter change data can be referred to the specific process of determining the cause of the first sub-fault based on the comparison results of environmental change data and target environmental change data, which will not be repeated here.
[0067] In one embodiment, the parameter change data includes multiple first parameter change curves, and the target parameter change data includes multiple second parameter change curves. Comparing the parameter change data during robot execution of each target sub-task with the target parameter change data, and determining the second sub-fault cause for the failure of the target sub-task based on the comparison results, includes: comparing the multiple first parameter change curves corresponding to each target sub-task with the multiple second parameter change curves to obtain comparison data of the multiple parameter change curves corresponding to each target sub-task; and analyzing the second fault cause corresponding to each target sub-task based on the comparison data of the multiple parameter change curves corresponding to each target sub-task.
[0068] For example, the first parameter variation curve includes the actual gripping force curve and the actual position trajectory curve; the second parameter variation curve includes the reference gripping force and the reference position trajectory curve.
[0069] The actual gripping force curve was compared with the reference gripping force curve, as follows: the actual gripping force curve rapidly rose to 20N within 0.3s and fluctuated; the reference gripping force curve slowly rose to 10N within 1s; therefore, it was initially inferred that the robot's gripping force control was abnormal.
[0070] A comparison of the actual position trajectory curve and the reference position trajectory curve reveals the following: the actual position trajectory curve exhibits slight displacement fluctuations, sometimes described as "pauses" or "jitters"; the reference position trajectory curve shows smooth movement and a simple path. Therefore, it is initially inferred that the robot actuator may be experiencing motion jitter due to unstable load or abnormal control commands. Ultimately, based on these preliminary inferences—namely, abnormal robot gripping force control and the possibility of motion jitter caused by unstable load or abnormal control commands—the second cause of the target subtask is determined to be abnormal robot gripping force control and an unstable execution path.
[0071] Sub-step S1054: Based on the first sub-fault cause and the second sub-fault cause, determine the fault cause of the robot's failure to execute each target sub-task.
[0072] In some cases, the failure of a target sub-task is caused by external environmental factors, in which case the first sub-cause of failure can determine the cause of failure for each target sub-task. In other cases, the failure is caused by factors within the robot itself, in which case the second sub-cause of failure can determine the cause of failure for each target sub-task. In still other cases, the failure of a target sub-task is caused by a combination of external environmental factors and factors within the robot itself; in this case, it is necessary to combine the first and second sub-cause of failure to determine the final cause of failure for the target sub-task. This approach allows for more accurate fault diagnosis and helps to take targeted repair and optimization measures based on different fault causes, thereby improving the robot's reliability and task execution efficiency.
[0073] In one embodiment, the robot fault analysis method further includes: determining first pose data corresponding to the target sub-task based on sensor data of the target sub-task, the first pose data being used to characterize the robot's position and attitude when the target sub-task ends; determining second pose data corresponding to the next sub-task of the target sub-task, the second pose data being used to characterize the robot's position and attitude when the next sub-task begins execution; determining pose change data of the robot based on the first pose data corresponding to the target sub-task and the second pose data corresponding to the next sub-task; and adjusting the robot's position and attitude based on the pose change data before the robot executes the next sub-task of the target sub-task.
[0074] Specifically, when a failed target subtask ends, its current pose (i.e., the first sub-data) differs from the pose at the end of a successfully executed target subtask. This can lead to the inability to correctly complete the next subtask and may also cause subsequent subtasks to deviate or fail. Therefore, it is necessary to correct the robot's pose to ensure the successful execution of subsequent subtasks.
[0075] like Figure 3 As shown, Figure 3 This is a schematic diagram of a scenario for the robot fault analysis method provided in an embodiment of this application.
[0076] For example, such as Figure 3 As shown, the task includes subtask 1, subtask 2, and subtask 3, where task 2 is the target subtask that failed to execute. An error occurred in task 2, and the breakpoint D is set as the node where task 2 ends. Based on the first pose data corresponding to breakpoint D (i.e., the robot's position and orientation at the end of task 2) and the robot's position and orientation at the start of task 3, the system calculates how to correct the trajectory from breakpoint D to task 3. Based on the calculation results, a corresponding motion command (e.g., move left) is generated, ensuring that the robot can correct its movement using this command when task 2 fails, thus enabling it to correctly execute the subsequent task 3.
[0077] It should be noted that by planning for corrections from the failed target sub-tasks, the robot's correction execution time is effectively reduced, greatly improving the robot's correction efficiency as well as the overall execution efficiency and success rate of the task.
[0078] In one embodiment, determining the robot's pose change data based on the first pose data corresponding to the target sub-task and the second pose data corresponding to the next sub-task includes: obtaining target image data from the image data of the target sub-task, wherein the target image data is used to characterize the environmental situation at the end of the execution of the target sub-task; planning a movement route for the robot from the target sub-task to the next sub-task based on the target image data; and determining the robot's pose change data based on the movement route, the first pose data corresponding to the target sub-task, and the second pose data corresponding to the next sub-task.
[0079] In some cases, the failure of the target subtask is caused by the external environment, such as the sudden appearance of obstacles. Therefore, when making corrective plans, it is necessary to incorporate the environmental conditions at the end of the target subtask into the path planning and pose adjustment calculation process to ensure the robot's obstacle avoidance safety, accurate posture, and improve the success rate of task execution.
[0080] The robot fault analysis method provided in the above embodiments, by retrospectively analyzing image data that characterizes the environmental conditions and environmental changes when the robot performs sub-tasks, as well as sensor data that characterizes parameter changes when the robot performs sub-tasks, more comprehensively considers the impact of environmental factors and robot-specific factors on task execution. This enables the robot to determine the cause of faults more quickly and accurately, and helps to take targeted repair and optimization measures based on different fault causes, thereby improving the reliability and efficiency of robot task execution.
[0081] Please refer to Figure 4 , Figure 4 This is a flowchart illustrating the steps of a robot fault analysis method provided in another embodiment of this application.
[0082] like Figure 4 As shown, the robot fault analysis method includes steps S201 to S204.
[0083] Step S201: Obtain the task data corresponding to each of the multiple sub-tasks in the task to be analyzed executed by the robot.
[0084] The task data includes multiple image data and multiple sensor data. The image data can be obtained from images taken by the robot during the execution of each sub-task. Images can be acquired through imaging devices mounted on the robot itself or through external imaging devices deployed in the task environment surrounding the robot. Images can include environmental images, target object images, and sub-task execution status images. Environmental images record the working environment during task execution and can include images of surrounding objects, obstacles, and the operating area. Target object images characterize objects or scenes related to the task objectives of each sub-task, aiding the robot in identification and localization. Task execution status images show the robot's state while performing a specific sub-task, such as the position of the robotic arm or the state of an object being grasped.
[0085] The sensor data in the task data can be various sensory data collected by the robot during the execution of each sub-task. Sensor data can come from different sensors mounted on the robot, providing detailed information about the environment, objects, and the robot's state. Specifically, sensor data can include position and attitude sensor data providing information such as the robot's position, speed, and angle changes; force and torque sensor data providing information about the contact strength between the robot and objects; and tactile sensor data detecting contact between the robot and its environment, and so on.
[0086] Step S202: Based on the multiple image data corresponding to each subtask, determine the environmental change data corresponding to each subtask.
[0087] Each image data point reflects the environmental state at a specific point in time when the robot performs each sub-task. Conversely, multiple image data points that are sequential in time reflect the environmental changes throughout the entire process of the robot performing the sub-task. Therefore, by comparing and analyzing multiple image data points corresponding to each sub-task, changes in relevant entities in the environment during the robot's execution of the sub-task can be identified and determined. These changes can include: the presence and disappearance of objects, changes in object positions, changes in the robot's position, and whether or not the target object is identified and grasped. By recording these changes in detail, environmental change data corresponding to each sub-task can be obtained. Environmental change data can include the changing objects, the type of change, the location of the change, and the time of the change.
[0088] Step S203: Based on the multiple sensor data corresponding to each target sub-task, determine the parameter change data corresponding to each sub-task.
[0089] Sensor data can be raw data collected from various sensors installed on the robot. It's understandable that, given the diverse types of sensors used in the robot, the collected data is not limited to reflecting changes in the robot's own state, but can also reflect changes in the surrounding environment. Specifically, sensors can be position sensors, force / torque sensors, temperature sensors, tactile sensors, and acoustic sensors, etc. Parameter change data can include position parameters (such as the robot's current position, orientation, and speed), force / torque parameters (such as the force or torque applied when the robot performs grasping, pushing, or other interactive operations), state parameters (such as the robot's working state during execution), and so on. It's understandable that by collecting and analyzing this sensor data, and identifying key parameter changes when the robot performs each target sub-task, a strong basis can be provided for subsequent decision-making and task execution.
[0090] Step S204: Input the environmental change data and parameter change data corresponding to multiple subtasks into the large language model, so that the large language model can determine the target subtask that failed and the cause of the failure of the target subtask based on the environmental change data and parameter change data of each subtask.
[0091] The large language model can be trained using data from a pre-defined task database. The task database can include second image data, target environment change data, and target parameter change data corresponding to the successful execution of multiple sub-tasks.
[0092] For example, the large language model is provided with the names of each subtask, as well as the corresponding environmental change data and parameter change data, and is required to explain the specific situation at the time of the failure and the specific cause of the failure.
[0093] In one embodiment, the process by which a large language model determines the target subtask that failed to execute and the cause of failure of the target subtask based on the environmental change data and parameter change data of each subtask includes: determining the target subtask that failed to execute from multiple subtasks based on the environmental change data of each subtask; and determining the cause of failure of the target subtask based on the environmental change data and parameter change data of the target subtask that failed to execute.
[0094] In one embodiment, the process of constructing the task database includes: acquiring target environment change data to characterize the environmental changes at the end of each subtask execution; acquiring second image data to characterize the environmental changes at the end of each subtask execution; acquiring target parameter change data to characterize the robot parameter changes at the end of each subtask execution; and constructing a task database corresponding to multiple subtasks based on the second image data, target environment change data, and target parameter change data corresponding to each of the multiple subtasks.
[0095] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described device and its modules and units can be referred to the corresponding processes in the aforementioned robot fault analysis method embodiments, and will not be repeated here.
[0096] Please refer to Figure 5 , Figure 5 This is a schematic block diagram of a robot fault analysis device provided in an embodiment of this application.
[0097] like Figure 5 As shown, the robot fault analysis device 300 includes:
[0098] The data acquisition module 301 is used to acquire task data collected when the robot performs multiple sub-tasks in the task to be analyzed. The task data includes multiple image data and multiple sensor data.
[0099] The task determination module 302 is used to determine the target subtask that failed to execute from multiple subtasks based on multiple image data corresponding to each subtask.
[0100] The environment determination module 303 is used to determine the environmental change data when the robot performs each target sub-task based on multiple image data corresponding to each target sub-task.
[0101] The parameter determination module 304 is used to determine the parameter change data when the robot performs each target sub-task based on multiple sensor data corresponding to each target sub-task.
[0102] The fault determination module 305 is used to determine the cause of failure of each target sub-task based on the environmental change data and parameter change data when the robot performs each target sub-task.
[0103] The apparatus provided in the above embodiments can be implemented as a computer program, which can be used in, for example... Figure 6 It runs on the computer device shown.
[0104] Please see Figure 6 , Figure 6This is a schematic block diagram of the structure of a computer device provided in an embodiment of this application.
[0105] like Figure 6 As shown, the computer device includes a processor, memory, and network interface connected via a system bus. The memory may include a storage medium and internal memory, and the storage medium may be non-volatile or volatile.
[0106] The storage medium can store the operating system and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any robot fault analysis method.
[0107] The processor provides computing and control capabilities, supporting the operation of the entire computer device.
[0108] Internal memory provides an environment for the execution of computer programs stored in the storage medium. When these computer programs are executed by the processor, the processor can perform any robot fault analysis method.
[0109] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0110] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.
[0111] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps:
[0112] Acquire task data collected by the robot when it performs multiple sub-tasks in the task to be analyzed. The task data includes multiple image data and multiple sensor data.
[0113] Based on multiple image data corresponding to each subtask, the target subtask that failed to execute is determined from multiple subtasks;
[0114] Based on multiple image data corresponding to each target sub-task, determine the environmental change data when the robot performs each target sub-task;
[0115] Based on multiple sensor data corresponding to each target sub-task, determine the parameter changes of the robot when performing each target sub-task;
[0116] Based on the environmental and parameter change data of the robot when performing each target sub-task, the cause of failure for each target sub-task is determined.
[0117] In one embodiment, when the processor determines the target subtask that failed to execute from multiple subtasks based on multiple image data corresponding to each subtask, it is configured to:
[0118] Obtain first image data from the task data of each subtask to characterize the environmental conditions at the end of the subtask execution, and obtain second image data from the end of each subtask execution to characterize the environmental conditions at the end of the subtask execution to characterize the successful execution.
[0119] The first image data corresponding to each subtask is matched with the second image data to obtain the matching degree corresponding to each subtask;
[0120] In each subtask, the subtask with a matching degree less than the preset matching degree threshold is identified as the target subtask that fails to execute.
[0121] In one embodiment, when the processor determines the cause of failure for each target sub-task based on environmental and parameter change data during the robot's execution of each target sub-task, it is configured to:
[0122] Acquire data on changes in the target environment and target parameters when the robot successfully executes the target sub-task;
[0123] The environmental change data when the robot performs each target sub-task is compared with the target environment change data, and the first sub-fault cause of the target sub-task failure is determined based on the comparison results.
[0124] The parameter change data when the robot performs each target sub-task is compared with the target parameter change data, and the second sub-fault cause of the failure of the target sub-task is determined based on the comparison results.
[0125] Based on the first and second sub-fault causes, determine the fault causes for the robot's failure to execute each target sub-task.
[0126] In one embodiment, the environmental change data includes multiple first image data, and the target environmental change data includes multiple second image data. When the processor compares the environmental change data and the target environmental change data during the robot's execution of each target sub-task with the target environmental change data, and determines the first sub-fault cause of the target sub-task's failure based on the comparison results, it is also configured to:
[0127] The first image data corresponding to each target sub-task is compared with the second image data to obtain the image comparison data corresponding to each target sub-task.
[0128] Based on the comparison data of multiple images corresponding to each target sub-task, the primary cause of failure for each target sub-task is analyzed.
[0129] In one embodiment, the parameter change data includes multiple first parameter change curves, and the target parameter change data includes multiple second parameter change curves. When the processor compares the parameter change data and target parameter change data during the robot's execution of each target sub-task with the target parameter change data, and determines the second sub-fault cause of the target sub-task's failure based on the comparison results, it is also configured to:
[0130] By comparing the multiple first parameter change curves and multiple second parameter change curves corresponding to each target sub-task, comparative data of the multiple parameter change curves corresponding to each target sub-task is obtained.
[0131] Based on the comparative data of the parameter change curves corresponding to each target sub-task, the second cause of failure corresponding to each target sub-task is analyzed.
[0132] In one embodiment, the processor is further configured to implement:
[0133] Based on the sensor data of the target sub-task, the first pose data corresponding to the target sub-task is determined. The first pose data is used to characterize the position and attitude of the robot when the target sub-task is completed.
[0134] Determine the second pose data corresponding to the next subtask of the target subtask. The second pose data is used to characterize the position and attitude of the robot when the next subtask starts to be executed.
[0135] Based on the first pose data corresponding to the target subtask and the second pose data corresponding to the next subtask, determine the robot's pose change data.
[0136] Before the robot performs the next subtask of the target subtask, the robot's position and attitude are adjusted based on pose change data.
[0137] In one embodiment, when the processor determines the robot's pose change data based on the first pose data corresponding to the target subtask and the second pose data corresponding to the next subtask, it is also used to:
[0138] Obtain target image data from the image data of the target subtask. The target image data is used to characterize the environmental conditions at the end of the execution of the target subtask.
[0139] Based on the target image data, plan the robot's movement route from the target sub-task to the next sub-task;
[0140] Based on the movement route, the first pose data corresponding to the target subtask, and the second pose data corresponding to the next subtask, the robot's pose change data is determined.
[0141] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the computer equipment described above can be referred to the corresponding process in the aforementioned robot fault analysis method embodiment, and will not be repeated here.
[0142] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0143] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, and the method implemented when the program instructions are executed can refer to various embodiments of the robot fault analysis method of this application.
[0144] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.
[0145] Furthermore, the computer's usable storage medium may primarily include a stored program area and a stored data area. The stored program area may store the operating system, applications required for at least one function, etc.; the stored data area may store data created based on the use of blockchain nodes, etc. The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. A blockchain is essentially a decentralized database, a chain of data blocks linked using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain may include a blockchain underlying platform, a platform product service layer, and an application service layer, etc.
[0146] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.
[0147] It should also be understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. It should be noted that, herein, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.
[0148] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. The above descriptions are merely specific implementations of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A robot fault analysis method, characterized in that, include: The task data collected by the robot when it performs multiple sub-tasks in the task to be analyzed includes multiple image data and multiple sensor data. Based on multiple image data corresponding to each of the subtasks, the target subtask that failed to execute is determined from the multiple subtasks; Based on multiple image data corresponding to each of the target sub-tasks, determine the environmental change data when the robot performs each of the target sub-tasks; Based on multiple sensor data corresponding to each of the target sub-tasks, determine the parameter change data when the robot performs each of the target sub-tasks; Based on the environmental change data and parameter change data when the robot executes each of the target sub-tasks, the cause of failure for each of the target sub-tasks is determined; Based on the sensor data of the target sub-task, the first pose data corresponding to the target sub-task is determined. The first pose data is used to characterize the position and posture of the robot when the target sub-task is completed. Determine the second pose data corresponding to the next subtask of the target subtask, the second pose data being used to characterize the position and attitude of the robot when the next subtask begins execution; Target image data is obtained from the image data of the target sub-task, and the target image data is used to characterize the environmental conditions when the target sub-task is completed; Based on the target image data, the robot plans its movement route from the target sub-task to the next sub-task; Based on the movement route, the first pose data corresponding to the target subtask, and the second pose data corresponding to the next subtask, the pose change data of the robot is determined. Before the robot executes the next subtask of the target subtask, the robot's position and attitude are adjusted based on the pose change data.
2. The robot fault analysis method as described in claim 1, characterized in that, The determination of the cause of failure for each target sub-task based on environmental change data and parameter change data during the robot's execution of each target sub-task includes: Acquire target environment change data and target parameter change data when the robot successfully executes the target sub-task; The environmental change data when the robot performs each of the target sub-tasks is compared with the target environmental change data, and the first sub-fault cause of the failure of the target sub-task is determined based on the comparison results. The parameter change data when the robot executes each target sub-task is compared with the target parameter change data, and the second sub-fault cause of the failure of the target sub-task is determined based on the comparison result; Based on the first sub-fault cause and the second sub-fault cause, determine the fault cause for the robot's failure to execute each of the target sub-tasks.
3. The robot fault analysis method as described in claim 2, characterized in that, The environmental change data includes multiple first image data, and the target environmental change data includes multiple second image data; The step of comparing the environmental change data when the robot executes each of the target sub-tasks with the target environment change data, and determining the first sub-fault cause of the target sub-task execution failure based on the comparison result, includes: The plurality of first image data corresponding to each target sub-task are compared with the plurality of second image data to obtain the plurality of image comparison data corresponding to each target sub-task; Based on the comparison data of multiple images corresponding to each target sub-task, the first cause of failure corresponding to each target sub-task is analyzed.
4. The robot fault analysis method as described in claim 2, characterized in that, The parameter change data includes multiple first parameter change curves, and the target parameter change data includes multiple second parameter change curves. The step of comparing the parameter change data when the robot executes each of the target sub-tasks with the target parameter change data, and determining the second sub-fault cause of the failure of the target sub-task based on the comparison result, includes: The multiple first parameter change curves corresponding to each target sub-task are compared with the multiple second parameter change curves to obtain comparison data of the multiple parameter change curves corresponding to each target sub-task. Based on the comparison data of multiple parameter change curves corresponding to each target sub-task, the second cause of failure corresponding to each target sub-task is analyzed.
5. The robot fault analysis method as described in claim 1, characterized in that, The step of determining the target subtask that failed to execute from among the multiple subtasks based on the multiple image data corresponding to each subtask includes: First image data representing the environmental conditions at the end of execution of each subtask is obtained from the task data of each subtask, and second image data representing the environmental conditions at the end of execution of each subtask is obtained to represent the successful execution. The first image data corresponding to each subtask is matched with the second image data to obtain the matching degree corresponding to each subtask; In each of the subtasks, the subtask with a matching degree less than a preset matching degree threshold is identified as the target subtask that fails to execute.
6. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the robot fault analysis method as described in any one of claims 1 to 5.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the robot fault analysis method as described in any one of claims 1 to 5.