Robot fault analysis method and device and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, is suitable for the medical health field and the financial field, and provides a robot fault analysis method and device and a storage medium, the method comprises the following steps: obtaining task data collected when a robot executes a plurality of subtasks in a to-be-analyzed task, the task data comprising a plurality of image data and a plurality of sensor data; based on the multiple pieces of image data corresponding to the sub-tasks, determining target sub-tasks which fail in execution from the multiple sub-tasks; based on the multiple pieces of image data corresponding to the target sub-tasks, determining environment change data when the robot executes the target sub-tasks; based on the multiple pieces of sensor data corresponding to the target sub-tasks, determining parameter change data when the robot executes the target sub-tasks; and based on the environment change data and the parameter change data when the robot executes each target sub-task, determining a fault cause of execution failure of each target sub-task. The invention aims to improve the accuracy of robot fault analysis.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a robot fault analysis method, device, and storage medium. Background Art

[0002] With the rapid development of robotics technology, robots are increasingly being used across various industries and fields. For example, in healthcare, robots can be used for drug distribution and surgical assistance, effectively improving hospital operational efficiency and the quality of medical services. In finance, robots are used in bank lobbies for customer reception, intelligent guidance, and business processing assistance, effectively enhancing the level of automation in financial services. The efficient operation of robotic systems depends on the accuracy and stability of the robots' tasks. However, robots often face various failures during the execution of their tasks, such as hardware damage, software crashes, and sensor malfunctions.

[0003] To address these issues, large language models can be used to analyze robot task information for fault analysis. However, the fault analysis returned by large language models is limited to the semantic level, lacking a deep understanding of robot behavior. Furthermore, the analysis is based primarily on fault pattern recognition from previous training data, resulting in poor accuracy.

[0004] Therefore, how to improve the accuracy of robot fault analysis is an urgent problem to be solved. Summary of the Invention

[0005] The main purpose of this application is to provide a robot fault analysis method, device and storage medium, aiming to improve the accuracy of robot fault analysis.

[0006] In a first aspect, the present application provides a robot fault analysis method, comprising:

[0007] Acquire task data collected when the robot performs multiple subtasks in the task to be analyzed, wherein the task data includes multiple image data and multiple sensor data;

[0008] Determining a target subtask that has failed to execute from the plurality of subtasks based on the plurality of image data corresponding to the respective subtasks;

[0009] Determining, based on a plurality of image data corresponding to each of the target subtasks, environmental change data when the robot performs each of the target subtasks;

[0010] Determining parameter change data when the robot performs each target subtask based on a plurality of sensor data corresponding to each target subtask;

[0011] Determine the failure causes of each of the target subtasks based on the environmental change data and parameter change data when the robot executes each of the target subtasks.

[0012] In a second aspect, the present application further provides a robot fault analysis method, including:

[0013] Obtain the task data corresponding to each of the multiple subtasks when the robot executes the task to be analyzed; wherein, the task data includes multiple image data and multiple sensor data;

[0014] Based on the multiple image data corresponding to each of the subtasks, determine the environmental change data corresponding to each of the subtasks;

[0015] Based on the multiple sensor data corresponding to each of the target subtasks, determine the parameter change data corresponding to each of the subtasks;

[0016] Input the environmental change data and parameter change data corresponding to the multiple subtasks into a large language model for the large language model to determine the target subtasks that have failed and the failure causes of the target subtasks based on the environmental change data and parameter change data of each of the subtasks.

[0017] In a third aspect, the present application further provides a computer device, the computer device includes a processor, a memory, and a computer program stored on the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the robot fault analysis method as described above are implemented.

[0018] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the robot fault analysis method as described above are implemented.

[0019] The present application provides a robot fault analysis method, device, and storage medium. The present application obtains task data collected when a robot executes multiple subtasks of a task to be analyzed, where the task data includes multiple image data and multiple sensor data; determines a target subtask that fails to execute from multiple subtasks based on the multiple image data corresponding to each subtask; determines environmental change data when the robot executes each target subtask based on the multiple image data corresponding to each target subtask; determines parameter change data when the robot executes each target subtask based on the multiple sensor data corresponding to each target subtask; determines the fault causes for the failure of each target subtask to execute based on the environmental change data and parameter change data when the robot executes each target subtask. By tracing back the image data that can characterize the environmental conditions and environmental change conditions when the robot executes subtasks, as well as the sensor data that can characterize the parameter change conditions when the robot executes subtasks, the influence of environmental factors and the robot's own factors on task execution is more comprehensively considered, so that the robot fault causes can be determined more quickly and accurately, greatly improving the accuracy of robot fault analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the accompanying drawings required for the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, other accompanying drawings can be obtained based on these drawings without creative efforts.

[0021] Figure 1 It is a schematic flowchart of the steps of a robot fault analysis method provided by an embodiment of the present application;

[0022] Figure 2 For Figure 1 the schematic flowchart of the sub-steps of the robot fault analysis method in

[0023] Figure 3 It is a schematic diagram of a scenario of the robot fault analysis method provided by an embodiment of the present application;

[0024] Figure 4 It is a schematic flowchart of the steps of a robot fault analysis method provided by another embodiment of the present application;

[0025] Figure 5 It is a schematic block diagram of a robot fault analysis device provided by an embodiment of the present application;

[0026] Figure 6 It is a schematic block diagram of the structure of a computer device provided by an embodiment of the present application.

[0027] The realization of the objectives of the present application, functional characteristics, and advantages will be further described with reference to the embodiments and the accompanying drawings. Detailed implementation manners

[0028] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0029] The flowchart shown in the accompanying drawings is only an example illustration, and does not necessarily include all the content and operations / steps, nor does it necessarily need to be executed in the described order. For example, some operations / steps can also be decomposed, combined, or partially merged. Therefore, the actual execution order may be changed according to the actual situation.

[0030] The embodiments of the present application provide a robot fault analysis method, device, and storage medium. Among them, the robot fault analysis method can be applied to a terminal device or a server. The terminal device can be an electronic device such as a mobile phone, a tablet computer, a notebook computer, a desktop computer, a personal digital assistant, and a wearable device; the server can be a single server or a server cluster composed of multiple servers.

[0031] Next, in conjunction with the accompanying drawings, some implementation manners of the present application will be described in detail. Without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0032] Please refer to Figure 1 , Figure 1 which is a schematic flowchart of the steps of a robot fault analysis method provided by an embodiment of the present application.

[0033] As Figure 1 shown, the robot fault analysis method includes steps S101 to S105.

[0034] Step S101: Obtain task data collected when the robot executes multiple subtasks in the to-be-analyzed task. The task data includes multiple image data and multiple sensor data.

[0035] Among them, the task to be analyzed can be a task with successful execution or a task with failed execution. The task to be analyzed is an operation process with a clear goal. For example, taking an item on the desktop. A subtask is the basic operation unit that makes up the task to be analyzed and is responsible for completing a specific step or stage in the operation process of the task to be analyzed. For example, identifying and locating the target object on the desktop, planning the movement path of the robotic arm, controlling the robotic arm to move to the target position, performing a grasping operation, moving the object to the specified area, and so on. It can be understood that the successful execution of the task to be analyzed does not necessarily mean that all of its subtasks are successfully executed, and the failed execution of the task to be analyzed does not necessarily mean that any of its subtasks fails. Specifically, in some cases, one or more of the subtasks in the task to be analyzed fail, and in other cases, all of the subtasks in the task to be analyzed are successfully executed.

[0036] The image data in the task data can be obtained from the images captured during the execution of each subtask by the robot. The acquisition channels of the images can be the imaging devices carried by the robot itself or the external shooting devices deployed in the task environment around the robot, etc. The images can include environmental images, target object images, and subtask execution status images, etc. Among them, the environmental images are used to record the working environment during the task execution and can include images of surrounding objects, obstacles, operation areas, etc.; the target object images are used to represent the objects or scenes related to the task objectives of each subtask and can help the robot with identification and positioning, etc.; the subtask execution status images are the status images of the robot when executing a certain subtask, such as the position of the robotic arm, the grasped state of the object, etc.

[0037] The sensor data in the task data can be various perception data collected by the robot during the execution of each subtask. The sensor data can come from different sensors carried by the robot and is used to provide detailed information about the environment, objects, and the state of the robot, etc. Specifically, the sensor data can be the position and attitude sensor data used to provide data such as the position information, speed, and angle change of the robot, the force and torque sensor data used to provide the contact force between the robot and the object, and the tactile sensor data used to detect the contact between the robot and the environment, and so on.

[0038] It should be noted that to further ensure the privacy and security of the above-mentioned target watermark and other related information, the above-mentioned target watermark and other related information can also be stored in a node of a blockchain. The technical solution of this application can also be applied to adding other data files stored on the blockchain. The blockchain referred to in this application is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0039] In one embodiment, before obtaining the task data collected when the robot executes multiple subtasks of the task to be analyzed, where the task data includes multiple image data and multiple sensor data, it further includes: obtaining multiple image data and multiple sensor data corresponding to the successful execution of multiple subtasks; and constructing a task database based on the multiple image data and multiple sensor data corresponding to the successful execution of multiple subtasks.

[0040] It can be understood that by constructing a task database, a reliable data basis can be provided for subsequent fault backtracking and correction strategy optimization, etc.

[0041] Step S102: Based on the multiple image data corresponding to each subtask, determine the target subtask that fails to execute from multiple subtasks.

[0042] Among them, each image data can be used to characterize the environmental situation at a certain time node during the execution of each subtask. It can be understood that under the condition of no external environmental factors (such as temporary obstacles blocking, etc.), the image data corresponding to the successful execution of any subtask each time should be consistent. For example, among the multiple image data corresponding to the successful execution each time, features such as the position of objects in the environment, the angle of the robot arm, and surrounding obstacles will show a consistent change pattern. However, if during a certain execution of a subtask, these features representing consistency deviate significantly, then the image data will also show abnormal changes. These abnormal changes usually indicate that there are problems during the execution process, which may lead to the failure of the subtask. Therefore, by comparing and analyzing one or more of the multiple image data corresponding to each subtask, the target subtask that fails to execute can be effectively identified.

[0043] In one embodiment, based on the multiple image data corresponding to each subtask, determining the target subtask that fails to execute from multiple subtasks includes: obtaining the first image data used to characterize the environmental situation at the end of the execution of each subtask from the task data of each subtask, and obtaining the second image data used to characterize the environmental situation of successful execution at the end of the execution of each subtask; matching the first image data corresponding to each subtask with the second image data to obtain the matching degree corresponding to each subtask; and determining the subtask with a matching degree less than the preset matching degree threshold among each subtask as the target subtask that fails to execute.

[0044] Among them, the second image data can utilize the preset task database. Both the first image data and the second image data can be based on one or more. It should be noted that to ensure the accuracy and effectiveness of the analysis, the first image data and the second image data should be equal in quantity to ensure that the environmental situation of each subtask can be compared one-to-one with the corresponding environmental situation of successful execution, avoiding misjudgment caused by inconsistent data.

[0045] Specifically, matching the first image data corresponding to each subtask with the second image data can be achieved by calculating the matching degree corresponding to each subtask through an image matching algorithm (such as image feature extraction, similarity calculation, etc.). The matching degree is used to reflect the similarity between the actual execution result of the subtask and the expected successful execution result. For all subtasks, if their matching degree is less than the preset matching degree threshold, then the subtask is determined as the target subtask that fails to execute. Among them, the matching degree threshold can be set according to actual needs, and a matching degree lower than this threshold indicates that an exception that can cause the subtask to fail occurs during the execution process of the subtask.

[0046] In one embodiment, obtaining the second image data for characterizing the successful execution environment when each subtask ends includes: obtaining multiple environmental image data for characterizing the successful execution of multiple subtasks; constructing a task database based on the multiple environmental image data corresponding to the multiple subtasks; and obtaining the second image data corresponding to each subtask from the task database.

[0047] Among them, the multiple environmental image data for characterizing the successful execution can be obtained by screening all the image data acquired during the successful execution process of the subtask.

[0048] Step S103: Based on the multiple image data corresponding to each target subtask, determine the environmental change data when the robot executes each target subtask.

[0049] Among them, each image data can reflect the environmental state of the robot at a certain time node when executing each subtask. Correspondingly, multiple image data that are continuous in time can reflect the environmental changes during the entire process of the robot executing the subtask. Therefore, by comparing and analyzing the multiple image data corresponding to each subtask, it is possible to identify and determine the changes that occur to relevant entities in the environment when the robot executes the subtask. These changes can include: the presence and disappearance of objects, the position changes of objects, the position changes of the robot, and the recognition and grasping of the target object or not. By recording these changes in detail, the environmental change data corresponding to each subtask can be obtained. The environmental change data can include the changed object, the type of change, the position of change, and the time of change, etc.

[0050] Step S104: Based on the multiple sensor data corresponding to each target subtask, determine the parameter change data when the robot executes each target subtask.

[0051] Among them, the sensor data can be the raw data collected by various sensors installed on the robot. It can be understood that in the case of diverse types of sensors used by the robot, the collected data is not limited to reflecting the state changes of the robot itself, but can also reflect changes in the surrounding environment, etc. Specifically, the sensors can be position sensors, force / torque sensors, temperature sensors, tactile sensors, acoustic sensors, etc. The parameter change data can be the change data of position parameters (such as the current position, direction, and speed of the robot, etc.), force / torque parameters (such as the force or torque applied when the robot performs grasping, pushing, or other interaction operations, etc.), state parameters (such as the working state during the robot's execution process, etc.), and so on. It can be understood that by collecting and analyzing these sensor data and identifying the key parameter changes when the robot executes each target subtask, it can provide a strong basis for subsequent decision-making and task execution.

[0052] Step S105: Based on the environmental change data and parameter change data when the robot executes each target subtask, determine the failure causes of each target subtask execution.

[0053] When the robot executes a task, the success or failure of the task execution is closely related to the environmental state of the robot and the state of the robot's own parameters. The environmental change data associated with the environmental state of the robot can reflect the dynamic changes in the surrounding environment during the task execution process, such as the movement of obstacles, the deviation of the target object's position, etc. The parameter change data associated with the state of the robot's own parameters reflects the robot's own state and the process of executing actions, such as position, speed, and torque, etc. Therefore, when a target subtask execution fails, by tracing back the data in these two dimensions of environmental change data and parameter change data, it is possible to analyze which state change and / or abnormal parameter fluctuation has caused the target subtask to fail to be successfully executed, thereby accurately determining the failure cause of the target subtask execution and effectively improving the accuracy of robot fault analysis.

[0054] Taking the example of a robot applied in the field of medical and health, assume that the task of the robot is to distribute drugs, and assume that the target subtask is "grasp the drug". The specific analysis process of the environmental change data and parameter change data for this target subtask is as follows: Analyzing the parameter change data of this target subtask, it is known that the image data shows that the position of the target drug is offset and not in the expected area. Further, analyzing the parameter change data of this target subtask, it is known that the force sensor data before grasping shows normal (gradually increasing), but the force value suddenly fluctuates abnormally during the grasping process, indicating abnormal contact of the robot's end effector.

[0055] Therefore, it is speculated that the failure of the target subtask to be grasped may be jointly caused by the movement of the object position and the mechanical grasping deviation, and thus the failure cause of the target subtask execution failure is determined.

[0056] In one embodiment, as Figure 2 shown, step S105 includes: sub-steps S1051 to S1054.

[0057] Sub-step S1051, obtain the target environment change data and target parameter change data when the robot successfully executes the target subtask.

[0058] Among them, the target environment change data and target parameter change data can be obtained through a pre-constructed task database, and the construction of this task database can specifically refer to the construction of the task database in the above embodiment. It should be noted that the task database in this application can be one or more, as long as it can meet the requirements of data comparison analysis and fault backtracking, etc., and will not be elaborated here.

[0059] Sub-step S1052, compare the environment change data when the robot executes each target subtask with the target environment change data, and determine the first sub-failure cause of the target subtask execution failure based on the comparison result.

[0060] Among them, the environment change data is the data used to characterize the environment change when the target subtask fails, and the target environment change data is the data used to characterize the environment change when the target subtask is successful. By comparing the environment change data and the target environment change data, the difference between the two can be identified. Specifically, the comparison of the environment change data and the target environment change data can be carried out using traditional image processing algorithms, such as image difference comparison methods, feature point matching comparison methods, etc., or using deep learning models, such as convolutional neural networks, generative adversarial networks, etc., or using machine learning methods, such as support vector machines, decision trees, etc.

[0061] In some cases, if there is a difference between the environment change data and the target environment change data, it indicates that the change in the external environment causes the failure of the target subtask to be executed. Further, by analyzing the difference in the data, the first sub-failure cause of the target subtask execution failure can be determined. For example, during the execution of the target subtask, a new obstacle appears, causing the target subtask to fail to be executed according to the original steps, and ultimately resulting in the failure of the target subtask to be executed. In some cases, if there is no difference between the environment change data and the target environment change data, it can indicate that the failure cause of the target subtask execution failure has nothing to do with the external environment.

[0062] In one embodiment, the environmental change data includes a plurality of first image data, and the target environmental change data includes a plurality of second image data. Comparing the environmental change data when the robot executes each target subtask with the target environmental change data, and determining the first sub-failure cause of the failure of the target subtask based on the comparison result, includes: comparing the plurality of first image data corresponding to each target subtask with the plurality of second image data to obtain a plurality of image comparison data corresponding to each target subtask; analyzing the first failure cause corresponding to each target subtask based on the plurality of image comparison data corresponding to each target subtask.

[0063] Exemplarily, assume that the environmental change data includes first image data A1 - E1, the target environmental change data includes second image data A2 - E2, and there is a one-to-one correspondence between the acquisition times of the images corresponding to the first image data A1 - E1 and the images corresponding to the second image data A2 - E2.

[0064] By comparing these two sets of image data, it is determined that there is a difference starting from the first image data D1 and the second image data D2. Further comparison shows that there is an object P in the first image data D1 that does not exist in the second image data D2. Thus, it can be determined that the first failure cause of the target subtask is the appearance of an obstacle.

[0065] Sub-step S1053: Compare the parameter change data when the robot executes each target subtask with the target parameter change data, and determine the second sub-failure cause of the failure of the target subtask based on the comparison result.

[0066] Among them, the parameter change data is the data used to characterize the parameter change of the robot when the target subtask fails, and the target parameter change data is the data used to characterize the parameter change of the robot when the target subtask is successful. For the specific process and description of determining the second sub-failure cause through the comparison result between the parameter change data and the target parameter change data, reference can be made to the specific process of determining the first failure sub-cause based on the comparison result of the environmental change data and the target environmental change data, which will not be elaborated here.

[0067] In one embodiment, the parameter change data includes a plurality of first parameter change curves, and the target parameter change data includes a plurality of second parameter change curves. Comparing the parameter change data when the robot executes each target subtask with the target parameter change data, and determining the second sub-failure cause of the failure of the target subtask based on the comparison result, includes: comparing the plurality of first parameter change curves corresponding to each target subtask with the plurality of second parameter change curves to obtain comparison data of the plurality of parameter change curves corresponding to each target subtask; analyzing the second failure cause corresponding to each target subtask based on the comparison data of the plurality of parameter change curves corresponding to each target subtask.

[0068] Exemplarily, the first parameter change curve includes an actual grasping force curve and an actual position trajectory curve; the second parameter change curve includes a reference grasping force and a reference position trajectory curve.

[0069] Compare the actual grasping force curve with the reference grasping force curve as follows: The actual grasping force curve rapidly rises to 20 N within 0.3 s and fluctuates; the reference grasping force curve slowly rises to 10 N within 1 s; Therefore, it is preliminarily inferred that the grasping force control of the robot is abnormal.

[0070] Compare the actual position trajectory curve with the reference position trajectory curve as follows: There are small displacement fluctuations of "pauses" or "jitters" in the actual position trajectory curve; the reference position trajectory curve moves smoothly and the path is simple. Therefore, it is preliminarily inferred that the robot actuator may cause motion jitter due to unstable load or abnormal control instructions. Finally, based on these preliminary inference results of the abnormal grasping force control of the robot and the possible motion jitter of the actuator due to unstable load or abnormal control instructions, the second failure cause corresponding to the target subtask is determined as the abnormal grasping force control of the robot and the unstable execution path.

[0071] Sub-step S1054: Determine the failure cause of the robot's failure to execute each target subtask according to the first sub-failure cause and the second sub-failure cause.

[0072] In some cases, if the failure of the target subtask is caused by external environmental factors, the failure cause of the robot's failure to execute each target subtask can be determined through the first sub-failure cause; in some cases, if the failure of the target subtask is caused by the robot's own factors, the failure cause of the robot's failure to execute each target subtask can be determined through the second sub-failure cause; in other cases, the failure of the target subtask is caused by the combined action of external environmental factors and the robot's own factors. In this case, it is necessary to combine the first sub-failure cause and the second sub-failure cause to determine the failure cause of the ultimate failure of the target subtask. In this way, the failure can be diagnosed more accurately, and it helps to take targeted repair and optimization measures according to different failure causes subsequently, thereby improving the reliability and task execution efficiency of the robot.

[0073] In one embodiment, the robot fault analysis method further includes: determining, based on the sensor data of the target subtask, first pose data corresponding to the target subtask, where the first pose data is used to characterize the position and pose of the robot when the target subtask is executed to completion; determining second pose data corresponding to the next subtask of the target subtask, where the second pose data is used to characterize the position and pose of the robot when the next subtask starts to be executed; determining pose change data of the robot based on the first pose data corresponding to the target subtask and the second pose data corresponding to the next subtask; and adjusting the position and pose of the robot based on the pose change data before the robot executes the next subtask of the target subtask.

[0074] Specifically, when the failed target subtask is executed to completion, there is a difference between its current pose (i.e., the first pose data) and the pose when the successfully executed target subtask is executed to completion, which may result in the inability to correctly complete the next subtask and may also cause task deviations or task failures in subsequent subtasks. Therefore, it is necessary to correct the pose of the robot to ensure the successful execution of subsequent subtasks.

[0075] As Figure 3 shown, Figure 3 FIG. is a schematic diagram of a scenario of the robot fault analysis method provided in an embodiment of the present application.

[0076] Exemplarily, as Figure 3 shown, the task includes subtask 1, subtask 2, and subtask 3, where subtask 2 is the failed target subtask and an error occurs in subtask 2. Let the node when subtask 2 is executed to completion be the breakpoint D of the task. According to the first pose data corresponding to the breakpoint D (i.e., the position and pose of the robot when subtask 2 is executed to completion) and the position and pose of the robot when subtask 3 starts to be executed, calculate how to correct from the breakpoint D to subtask 3, and generate a corresponding motion instruction (such as moving left) according to the calculation result, ensuring that when subtask 2 of the robot fails, the robot corrects its actions through this instruction, so that it can correctly execute the subsequent subtask 3.

[0077] It should be noted that by performing correction planning from the failed target subtask, the correction execution time of the robot is effectively reduced, and the correction efficiency of the robot and the overall execution efficiency and success rate of the task are greatly improved.

[0078] In one embodiment, determining the pose change data of the robot based on the first pose data corresponding to the target subtask and the second pose data corresponding to the next subtask includes: obtaining target image data from the image data of the target subtask, where the target image data is used to characterize the environmental situation at the end of the execution of the target subtask; planning the movement route of the robot from the execution of the target subtask to the next subtask based on the target image data; and determining the pose change data of the robot based on the movement route, the first pose data corresponding to the target subtask, and the second pose data corresponding to the next subtask.

[0079] In some cases, the reason for the failure of the target subtask execution is caused by the external environment, such as the sudden appearance of an obstacle. Therefore, when performing corrective planning, it is necessary to incorporate the factor of the environmental situation at the end of the execution of the target subtask into the calculation process of path planning and pose adjustment to ensure the obstacle avoidance safety and accurate pose of the robot and improve the success rate of task execution.

[0080] The robot fault analysis method provided in the above embodiment can more comprehensively consider the influence of environmental factors and the robot's own factors on task execution by backtracking the image data that can characterize the environmental situation and environmental change situation when the robot executes subtasks, as well as the sensor data that characterizes the parameter change situation when the robot executes subtasks. Therefore, it can more quickly and accurately determine the cause of the robot fault, and at the same time helps to take targeted repair and optimization measures according to different fault causes in the future, improving the reliability and efficiency of the robot task execution.

[0081] Please refer to Figure 4 , Figure 4 which is a schematic flowchart of the steps of a robot fault analysis method provided in another embodiment of the present application.

[0082] As Figure 4 shown, the robot fault analysis method includes steps S201 to S204.

[0083] Step S201: Obtain the task data corresponding to each of the multiple subtasks when the robot executes the task to be analyzed.

[0084] Among them, the task data includes multiple image data and multiple sensor data. The image data in the task data can be obtained from images captured during the robot's execution of each subtask. The acquisition channels of the images can be the imaging devices carried by the robot itself or external imaging devices deployed in the task environment around the robot, etc. The images can include environmental images, target object images, and subtask execution status images, etc. Among them, the environmental images are used to record the working environment during the task execution, and can include images of surrounding objects, obstacles, operation areas, etc.; the target object images are used to represent the objects or scenes related to the task objectives of each subtask, and can help the robot with identification and positioning, etc.; the subtask execution status images are the status images of the robot when executing a certain subtask, such as the position of the robotic arm, the state of the object being grasped, etc.

[0085] The sensor data in the task data can be various perception data collected by the robot during the execution of each subtask. The sensor data can come from different sensors carried by the robot and is used to provide detailed information about the environment, objects, and the robot's state, etc. Specifically, the sensor data can be position and attitude sensor data for providing data such as the robot's position information, speed, and angle change, force and torque sensor data for providing the contact force between the robot and the object, and tactile sensor data for detecting the contact between the robot and the environment, etc.

[0086] Step S202: Based on the multiple image data corresponding to each subtask, determine the environmental change data corresponding to each subtask.

[0087] Among them, each image data can reflect the environmental state of the robot at a certain time node when executing each subtask. Correspondingly, multiple image data that are continuous in time can reflect the environmental changes during the entire process of the robot executing the subtask. Therefore, by comparing and analyzing the multiple image data corresponding to each subtask, it is possible to identify and determine the changes that occur to relevant entities in the environment when the robot executes the subtask. These changes can include: the presence and disappearance of objects, the position changes of objects, the position changes of the robot, and the recognition and grasping of the target object, etc. By recording these changes in detail, the environmental change data corresponding to each subtask can be obtained. The environmental change data can include the changed objects, the types of changes, the positions of changes, and the times of changes, etc.

[0088] Step S203: Based on the multiple sensor data corresponding to each target subtask, determine the parameter change data corresponding to each subtask.

[0089] Among them, the sensor data can be the raw data collected by various sensors installed on the robot. It can be understood that in the case of the diversification of the types of sensors used by the robot, the collected data is not limited to reflecting the state changes of the robot itself, but can also reflect the changes in the surrounding environment, etc. Specifically, the sensors can be position sensors, force / torque sensors, temperature sensors, tactile sensors, acoustic sensors, etc., and the parameter change data can be position parameters (such as the current position, direction, and speed of the robot, etc.), force / torque parameters (such as the force or torque applied when the robot performs grasping, pushing, or other interaction operations, etc.), state parameters (such as the working state during the execution of the robot, etc.), and so on. It can be understood that by collecting and analyzing these sensor data and identifying the key parameter changes when the robot executes each target subtask, it can provide a strong basis for subsequent decision-making and task execution.

[0090] Step S204: Input the environmental change data and parameter change data corresponding to multiple subtasks into the large language model, so that the large language model can determine the target subtask that fails to execute and the failure reason of the target subtask based on the environmental change data and parameter change data of each subtask.

[0091] Among them, the large language model can be trained through the data in the preset task database, and the task database can include the second image data, target environmental change data, target parameter change data, etc. corresponding to the successful execution of multiple subtasks.

[0092] Exemplarily, provide the name of each subtask and the environmental change data and parameter change data corresponding to each subtask to the large language model, and require the large language model to explain the specific situation at the failure moment and the specific reason for the failure.

[0093] In one embodiment, the process in which the large language model determines the target subtask that fails to execute and the failure reason of the target subtask based on the environmental change data and parameter change data of each subtask includes: determining the target subtask that fails to execute from multiple subtasks based on the environmental change data of each subtask; determining the failure reason for the execution failure of the target subtask of each parameter change data based on the environmental change data and parameter change data of the target subtask that fails to execute.

[0094] In one embodiment, the process of constructing the task database includes: obtaining target environmental change data for characterizing the successful execution environment change when each subtask ends; obtaining second image data for characterizing the successful execution environment when each subtask ends; obtaining target parameter change data for characterizing the successful execution robot parameter change when each subtask ends; and constructing a task database corresponding to multiple subtasks based on the second image data, target environmental change data, and target parameter change data corresponding to each of the multiple subtasks.

[0095] It should be noted that those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described device and each module and unit can refer to the corresponding processes in the foregoing embodiments of the robot fault analysis method, and will not be described in detail herein.

[0096] Please refer to Figure 5 , Figure 5 which is a schematic block diagram of a robot fault analysis device provided by an embodiment of the present application.

[0097] As Figure 5 shown, the robot fault analysis device 300 includes:

[0098] A data acquisition module 301, configured to acquire task data collected when a robot executes multiple subtasks in a task to be analyzed, where the task data includes multiple image data and multiple sensor data.

[0099] A task determination module 302, configured to determine a target subtask that fails to execute from multiple subtasks based on the multiple image data corresponding to each subtask.

[0100] An environment determination module 303, configured to determine environmental change data when the robot executes each target subtask based on the multiple image data corresponding to each target subtask.

[0101] A parameter determination module 304, configured to determine parameter change data when the robot executes each target subtask based on the multiple sensor data corresponding to each target subtask.

[0102] A fault determination module 305, configured to determine the fault cause of each target subtask's failure to execute based on the environmental change data and parameter change data when the robot executes each target subtask.

[0103] The device provided in the above embodiment can be implemented in the form of a computer program, and the computer program can run on a computer device as Figure 6 shown.

[0104] Please refer to Figure 6 , Figure 6A schematic block diagram of the structure of a computer device provided by an embodiment of the present application.

[0105] As Figure 6 shown, the computer device includes a processor, a memory, and a network interface connected by a system bus. Among them, the memory may include a storage medium and an internal memory, and the storage medium may be non-volatile or volatile.

[0106] The storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can be made to execute any one of the robot fault analysis methods.

[0107] The processor is used to provide computing and control capabilities to support the operation of the entire computer device.

[0108] The internal memory provides an environment for the operation of the computer program in the storage medium. When the computer program is executed by the processor, the processor can be made to execute any one of the robot fault analysis methods.

[0109] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art can understand that Figure 6 the structure shown in

[0110] is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0111] Among them, in one embodiment, the processor is used to run the computer program stored in the memory to implement the following steps:

[0112] Obtain the task data collected when the robot executes multiple subtasks of the task to be analyzed, where the task data includes multiple image data and multiple sensor data;

[0113] Based on multiple image data corresponding to each subtask, determine a target subtask that fails to execute from multiple subtasks;

[0114] Based on multiple image data corresponding to each target subtask, determine environmental change data when the robot executes each target subtask;

[0115] Based on multiple sensor data corresponding to each target subtask, determine parameter change data when the robot executes each target subtask;

[0116] Based on the environmental change data and parameter change data when the robot executes each target subtask, determine the failure causes of each target subtask.

[0117] In one embodiment, when the processor implements determining a target subtask that fails to execute from multiple subtasks based on multiple image data corresponding to each subtask, it is used to implement:

[0118] Obtain first image data for characterizing the environmental situation at the end of the execution of the subtask from the task data of each subtask, and obtain second image data for characterizing the successful execution environmental situation at the end of the execution of each subtask;

[0119] Match the first image data corresponding to each subtask with the second image data to obtain the matching degree corresponding to each subtask;

[0120] Among each subtask, determine the subtask with a matching degree less than the preset matching degree threshold as the target subtask that fails to execute.

[0121] In one embodiment, when the processor implements determining the failure causes of each target subtask based on the environmental change data and parameter change data when the robot executes each target subtask, it is used to implement:

[0122] Obtain target environmental change data and target parameter change data when the robot successfully executes the target subtask;

[0123] Compare the environmental change data when the robot executes each target subtask with the target environmental change data, and determine the first sub-failure cause of the target subtask failure based on the comparison result;

[0124] Compare the parameter change data when the robot executes each target subtask with the target parameter change data, and determine the second sub-failure cause of the target subtask failure based on the comparison result;

[0125] According to the first sub-failure cause and the second sub-failure cause, determine the failure cause of the robot's execution of each target subtask.

[0126] In one embodiment, the environmental change data includes a plurality of first image data, and the target environmental change data includes a plurality of second image data. When the processor compares the environmental change data and the target environmental change data when the robot executes each target subtask and determines the first sub-failure cause of the failure of the target subtask based on the comparison result, the processor is further configured to:

[0127] Compare the plurality of first image data corresponding to each target subtask with the plurality of second image data to obtain a plurality of image comparison data corresponding to each target subtask;

[0128] Analyze the first failure cause corresponding to each target subtask based on the plurality of image comparison data corresponding to each target subtask.

[0129] In one embodiment, the parameter change data includes a plurality of first parameter change curves, and the target parameter change data includes a plurality of second parameter change curves. When the processor compares the parameter change data and the target parameter change data when the robot executes each target subtask and determines the second sub-failure cause of the failure of the target subtask based on the comparison result, the processor is further configured to:

[0130] Compare the plurality of first parameter change curves corresponding to each target subtask with the plurality of second parameter change curves to obtain comparison data of the plurality of parameter change curves corresponding to each target subtask;

[0131] Analyze the second failure cause corresponding to each target subtask based on the comparison data of the plurality of parameter change curves corresponding to each target subtask.

[0132] In one embodiment, the processor is further configured to:

[0133] Determine the first pose data corresponding to the target subtask based on the sensor data of the target subtask, where the first pose data is used to characterize the position and pose of the robot when the target subtask is executed;

[0134] Determine the second pose data corresponding to the next subtask of the target subtask, where the second pose data is used to characterize the position and pose of the robot when the next subtask starts to be executed;

[0135] Determine the pose change data of the robot based on the first pose data corresponding to the target subtask and the second pose data corresponding to the next subtask;

[0136] Before the robot executes the next subtask of the target subtask, adjust the position and pose of the robot based on the pose change data.

[0137] In one embodiment, when the processor determines the pose change data of the robot based on the first pose data corresponding to the target subtask and the second pose data corresponding to the next subtask, the processor is further configured to:

[0138] Obtain target image data from the image data of the target subtask, where the target image data is used to characterize the environmental situation at the end of the execution of the target subtask;

[0139] Based on the target image data, plan the movement route of the robot from the execution of the target subtask to the next subtask;

[0140] Based on the movement route, the first pose data corresponding to the target subtask, and the second pose data corresponding to the next subtask, determine the pose change data of the robot.

[0141] It should be noted that those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the above-described computer device can refer to the corresponding process in the foregoing embodiments of the robot fault analysis method, and will not be elaborated herein.

[0142] This application can be used in many general-purpose or special-purpose computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and so on. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This application can also be practiced in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0143] The embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions, and the method implemented when the program instructions are executed can refer to the various embodiments of the robot fault analysis method of the present application.

[0144] Among them, the computer-readable storage medium may be the internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the computer device.

[0145] Furthermore, the computer-usable storage medium may mainly include a storage program area and a storage data area. Among them, the storage program area may store an operating system, application programs required for at least one function, etc.; the storage data area may store data created according to the use of the blockchain node, etc. The blockchain referred to in this application is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm. Blockchain, in essence, is a decentralized database, a string of data blocks generated by using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity (anti-counterfeiting) of the information and generate the next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, etc.

[0146] It should be understood that the terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. As used in the specification of this application and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms.

[0147] It should also be understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations. It should be noted that in this article, the term "comprises", "comprising", or any other variant thereof is intended to cover a non-exclusive inclusion, so that a process, method, article, or system including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article, or system. Without further limitation, an element defined by the statement "comprising a..." does not exclude the existence of additional identical elements in the process, method, article, or system including that element.

[0148] The serial numbers of the embodiments of the present application above are only for description and do not represent the superiority or inferiority of the embodiments. The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for analyzing robot faults, characterized in that, Including: Obtain task data collected when a robot executes multiple subtasks of a task to be analyzed, where the task data includes multiple image data and multiple sensor data; Based on the multiple image data corresponding to each of the subtasks, determine a target subtask that fails to execute from the multiple subtasks; Based on the multiple image data corresponding to each of the target subtasks, determine environmental change data when the robot executes each of the target subtasks; Based on the multiple sensor data corresponding to each of the target subtasks, determine parameter change data when the robot executes each of the target subtasks; Based on the environmental change data and parameter change data when the robot executes each of the target subtasks, determine the failure causes of each of the target subtasks.

2. The robot fault analysis method according to claim 1, characterized in that The determining the failure causes of each of the target subtasks based on the environmental change data and parameter change data when the robot executes each of the target subtasks includes: Obtain target environmental change data and target parameter change data when the robot successfully executes the target subtask; Compare the environmental change data when the robot executes each of the target subtasks with the target environmental change data, and determine a first sub-failure cause of the target subtask based on the comparison result; Compare the parameter change data when the robot executes each of the target subtasks with the target parameter change data, and determine a second sub-failure cause of the target subtask based on the comparison result; Based on the first sub-failure cause and the second sub-failure cause, determine the failure cause of the robot's failure to execute each of the target subtasks.

3. The robot fault analysis method according to claim 2, characterized in that The environmental change data includes multiple first image data, and the target environmental change data includes multiple second image data; The comparing the environmental change data when the robot executes each of the target subtasks with the target environmental change data, and determining a first sub-failure cause of the target subtask based on the comparison result includes: Compare the multiple first image data corresponding to each of the target subtasks with the multiple second image data to obtain multiple image comparison data corresponding to each of the target subtasks; Based on the multiple image comparison data corresponding to each of the target subtasks, analyze the first failure cause corresponding to each of the target subtasks.

4. The robot fault analysis method according to claim 2, characterized in that The parameter change data includes multiple first parameter change curves, and the target parameter change data includes multiple second parameter change curves; The comparing the parameter change data when the robot executes each of the target subtasks with the target parameter change data, and determining a second sub-failure cause of the target subtask based on the comparison result includes: Compare the multiple first parameter change curves corresponding to each of the target subtasks with the multiple second parameter change curves to obtain comparison data of the multiple parameter change curves corresponding to each of the target subtasks; Based on the comparison data of the multiple parameter change curves corresponding to each of the target subtasks, analyze the second failure cause corresponding to each of the target subtasks.

5. The robot fault analysis method according to claim 1, wherein, Determining a target subtask that fails to execute from multiple subtasks based on multiple pieces of image data corresponding to each of the subtasks includes: Obtaining first image data for characterizing the environmental conditions at the end of the execution of each subtask from the task data of each subtask, and obtaining second image data for characterizing the environmental conditions of successful execution at the end of the execution of each subtask; Matching the first image data corresponding to each subtask with the second image data to obtain the matching degree corresponding to each subtask; Determining, among each of the subtasks, the subtask with a matching degree less than a preset matching degree threshold as the target subtask that fails to execute.

6. The robot fault analysis method according to any one of claims 1-5, characterized in that, The method further includes: Determining first pose data corresponding to the target subtask based on the sensor data of the target subtask, where the first pose data is used to characterize the position and pose of the robot at the end of the execution of the target subtask; Determining second pose data corresponding to the next subtask of the target subtask, where the second pose data is used to characterize the position and pose of the robot when the next subtask starts to execute; Determining the pose change data of the robot based on the first pose data corresponding to the target subtask and the second pose data corresponding to the next subtask; Before the robot executes the next subtask of the target subtask, adjusting the position and pose of the robot based on the pose change data.

7. The robot fault analysis method according to claim 6, wherein, The determining the pose change data of the robot based on the first pose data corresponding to the target subtask and the second pose data corresponding to the next subtask includes: Obtaining target image data from the image data of the target subtask, where the target image data is used to characterize the environmental conditions at the end of the execution of the target subtask; Planning a movement route of the robot from the execution of the target subtask to the next subtask based on the target image data; Determining the pose change data of the robot based on the movement route, the first pose data corresponding to the target subtask, and the second pose data corresponding to the next subtask.

8. A method for analyzing robot faults, characterized in that, Including: Obtaining the task data corresponding to each of the multiple subtasks in the task to be analyzed executed by the robot; wherein, the task data includes multiple pieces of image data and multiple pieces of sensor data; Determining the environmental change data corresponding to each subtask based on the multiple pieces of image data corresponding to each subtask; Determining the parameter change data corresponding to each subtask based on the multiple pieces of sensor data corresponding to each target subtask; Inputting the environmental change data and parameter change data corresponding to the multiple subtasks into a large language model for the large language model to determine the target subtask that fails to execute and the failure reason of the target subtask based on the environmental change data and parameter change data of each subtask.

9. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored on the memory and executable by the processor, where when the computer program is executed by the processor, it implements the robot fault analysis method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the robot fault analysis method according to any one of claims 1 to 8 is implemented.

Images