Robot monitoring methods, devices, equipment and storage media

Abstract

This application discloses a robot monitoring method, apparatus, device, and storage medium. To avoid the problem of monitoring information errors caused by interference from environmental factors due to a single algorithm, this application employs at least two pose detection algorithms to obtain the robot's current pose detection result. The pose detection algorithms may include those based on the robot's own sensors and those based on external sensors. These two algorithms perform pose detection based on data collected from different sensors, thus complementing each other. Furthermore, the results of these at least two pose detection algorithms are fused to obtain a fused pose. The reliability of the fused pose is verified based on the pose detection results of each algorithm. When the reliability of the fused pose is determined, it is saved as the robot's current valid pose. The valid pose obtained by this application is more accurate and has higher reliability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, and more particularly to a robot monitoring method, device, equipment and storage medium. BACKGROUND

[0002] With the development of science and technology, robots are increasingly widely used in industries, medical treatment, families and other fields, especially in logistics, distribution and other industries. Mobile robots have become an important means of transportation. However, due to the complexity and uncertainty of mobile robots, they may encounter various problems in actual application, such as inaccurate positioning, unreasonable path planning, abnormal robot running state and the like. Therefore, it is of great practical significance to develop a monitoring system capable of monitoring the state of mobile robots in real time.

[0003] Robot monitoring can include monitoring the pose and running state of the robot. Among them, the pose is the basis for robot navigation and processing other tasks, so the monitoring of the pose of the robot is particularly important. The prior art provides many monitoring schemes based on sensors and algorithms, and the commonly used SLAM algorithm (Simultaneous Localization And Mapping, simultaneous localization and mapping) can locate the pose information of the robot.

[0004] Although these methods and technologies solve the problem of mobile robot state monitoring to some extent, there are still some limitations, such as single algorithm being easily affected by complex environment and other factors, which cannot guarantee the accuracy of monitoring information, and the credibility is low. SUMMARY

[0005] In view of the above problems, the present application is proposed to provide a robot monitoring method, device, equipment and storage medium, to provide a robot monitoring scheme with high credibility and improve the accuracy of monitoring information. The specific scheme is as follows:

[0006] In a first aspect, a robot monitoring method is provided, comprising:

[0007] At least two pose detection algorithms are used to obtain the current pose detection result of the robot, the at least two pose detection algorithms including a pose detection algorithm based on the robot's own sensor and a pose detection algorithm based on an external sensor;

[0008] The pose detection results of the at least two pose detection algorithms are fused to obtain a fused pose, and whether the fused pose is credible is verified based on the pose detection results of each pose detection algorithm;

[0009] When the fused pose is credible, the fused pose is saved as the current valid pose of the robot.

[0010] Preferably, the at least two pose detection algorithms comprise:

[0011] a simultaneous localization and mapping (SLAM) algorithm, and a special marker based visual positioning detection algorithm.

[0012] Preferably, the special marker based visual positioning detection algorithm comprises an ArUco marker based pose detection algorithm.

[0013] Preferably, the at least two pose detection algorithms further comprise:

[0014] a pose detection algorithm based on a deep neural network model, wherein a training sample of the deep neural network model during training comprises a current frame training picture, a target pose and a historical pose of the robot, and a sample label is a pose of the robot at a current time;

[0015] and a process of obtaining a current pose detection result of the robot by using the pose detection algorithm based on the deep neural network model, comprising:

[0016] obtaining a current frame picture, a target pose corresponding to the current frame and a historical pose of the robot to form input state data;

[0017] sending the input state data into the deep neural network model after training to obtain a current pose detection result of the robot output by the model.

[0018] Preferably, at least two pose detection algorithms are used to obtain current pose detection results of the robot respectively, the pose detection results of the at least two pose detection algorithms are fused to obtain a fused pose, and a process of verifying whether the fused pose is credible based on the pose detection results of the pose detection algorithms, comprising:

[0019] a first pose of the robot is obtained by using the SLAM algorithm, and a second pose of the robot is obtained by using the special marker based visual positioning detection algorithm;

[0020] the first pose and the second pose are subjected to pose fusion voting to obtain a fused pose;

[0021] a first credibility judgment of the fused pose is performed according to a difference size of the fused pose and the first pose and the second pose respectively;

[0022] if the first credibility judgment determines that the fused pose is credible, it is finally determined that the fused pose is credible, and if the first credibility judgment determines that the fused pose is not credible, a third pose of the robot and a confidence degree are obtained by using the pose detection algorithm based on the neural network model;

[0023] If the difference between the fusion pose information and the third pose is within a set difference range, and the confidence is not lower than a set confidence threshold, it is finally determined that the fusion pose is reliable, otherwise, it is finally determined that the fusion pose is unreliable.

[0024] Preferably, before saving the fusion pose as the current valid pose of the robot, further comprising:

[0025] When the fusion pose is reliable, determining a reasonable range of the current pose according to a planned target pose of the robot and a historical motion state;

[0026] Determining whether the fusion pose is within the reasonable range of the current pose, if yes, determining that the fusion pose is reasonable, and executing the step of saving the fusion pose as the current valid pose of the robot, if no, determining that the fusion pose is unreasonable, and discarding the current frame.

[0027] Preferably, the pose fusion voting on the first pose and the second pose to obtain the fusion pose comprises:

[0028] Setting a covariance matrix of the first pose and the second pose;

[0029] Calculating a filtered fusion result of the first pose and the second pose through Kalman filtering and the covariance matrix to obtain the fusion pose.

[0030] Preferably, further comprising:

[0031] Obtaining an actual running state trajectory of the robot, the running state trajectory comprising a timestamp and running state information corresponding to each timestamp;

[0032] Comparing the difference between the actual running state trajectory and a planned theoretical running state trajectory, and outputting a prompt information if the difference exceeds a set tolerance.

[0033] Preferably, the running state information comprises any one or more of the following:

[0034] Internal state class, action state class, and event state class.

[0035] Preferably, further comprising:

[0036] Determining a target state of abnormality according to the difference between the actual running state trajectory and the planned theoretical running state trajectory;

[0037] Determining each related state associated with the target state and the dependency relationship between each related state according to a pre-configured association relationship between states;

[0038] According to the dependency relationship, each of the related states is sequentially traversed, attribute information of an attribute item of the related state is called, and reliability analysis of the attribute item is performed based on the attribute information;

[0039] The target attribute item that does not satisfy the reliability condition is screened, and an abnormal label corresponding to the target attribute item is obtained;

[0040] The abnormal reason set of the target state is composed of each of the abnormal labels.

[0041] Preferably, the method further comprises:

[0042] The effective pose of the robot at each timestamp within a set time period before the current time is stored in a real-time pose queue, and the target pose of the robot at each timestamp is stored in a regulation and control pose queue;

[0043] The real-time pose queue and the regulation and control pose queue are aligned according to the timestamps, and the pose difference of each timestamp is calculated;

[0044] It is judged whether the current pose of the robot needs to be corrected based on the pose difference, and if so, a correction instruction is sent to the robot.

[0045] In a second aspect, a robot monitoring device is provided, comprising:

[0046] A pose calculation unit is configured to obtain a current pose detection result of a robot by using at least two pose detection algorithms, wherein the at least two pose detection algorithms include a pose detection algorithm based on a sensor of the robot itself and a pose detection algorithm based on an external sensor;

[0047] A pose fusion unit is configured to fuse the pose detection results of the at least two pose detection algorithms to obtain a fused pose, and verify whether the fused pose is reliable based on the pose detection results of each of the pose detection algorithms;

[0048] An effective pose saving unit is configured to save the fused pose as a current effective pose of the robot when the fused pose is reliable.

[0049] In a third aspect, a robot monitoring device is provided, comprising a memory and a processor;

[0050] The memory is configured to store a program;

[0051] The processor is configured to execute the program to implement each step of the robot monitoring method as described above.

[0052] In a fourth aspect, a storage medium is provided, and the storage medium has stored thereon a computer program, and the computer program is executed by a processor to implement each step of the robot monitoring method as described above.

[0053] By the above technical solution, in order to avoid the problem that a single algorithm is easily disturbed by environmental factors and causes errors in monitoring information, at least two pose detection algorithms are adopted to obtain the current pose detection result of the robot, wherein the pose detection algorithm can include a pose detection algorithm based on the sensor of the robot itself and a pose detection algorithm based on an external sensor, and the two algorithms perform pose detection based on data collected by different sensors, which can complement each other. On this basis, the pose detection results of the at least two pose detection algorithms are fused to obtain a fused pose, and further, the fused pose can be cross-verified based on the pose detection results of each pose detection algorithm to verify the credibility of the fused pose, and when it is determined that the fused pose is credible, it is saved as the current effective pose of the robot. Obviously, the application fuses the pose detection results of multiple different pose detection algorithms to obtain a fused pose, and verifies the credibility of the fused pose, which can combine the advantages of each different pose detection algorithm, make up for the defects of a single algorithm affected by environmental factors and errors, and finally obtain a more accurate effective pose with higher credibility. BRIEF DESCRIPTION OF DRAWINGS

[0054] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments with reference made to the accompanying drawings. The drawings are for purposes of illustration only and are not considered a limitation of the present application. Furthermore, like reference numerals are used to denote identical components across the various drawings. In the drawings:

[0055] Figure 1 A flowchart of a robot monitoring method provided by an embodiment of the application is shown;

[0056] Figure 2 An ArUco marker is exemplified;

[0057] Figure 3 A robot monitoring method flowchart using three pose detection algorithms is exemplified;

[0058] Figure 4 Another robot monitoring method flowchart using three pose detection algorithms is exemplified;

[0059] Figure 5 A pose diagram of a robot advancing process is exemplified;

[0060] Figure 6 A method flowchart for monitoring the moving track of a robot is exemplified;

[0061] Figure 7 A structural schematic diagram of a robot monitoring device provided by an embodiment of the present application is shown in the figure.

[0062] Figure 8 A structural schematic diagram of a robot monitoring device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0063] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0064] The present application provides a robot monitoring scheme, which can be applied to monitoring the running state of a movable robot, which can include the robot pose, sensor state, acceleration and deceleration state, etc. Through the state monitoring of the robot, the working efficiency and safety of the robot can be effectively improved, which provides strong support for the wide application of the robot in various fields.

[0065] The present application scheme can be realized based on a terminal with data processing capability, which can be a robot, i.e. the state monitoring scheme can be realized by the processor carried by the robot itself. In addition, the terminal can also be other devices in communication with the robot, such as a base station or other terminals.

[0066] Next, the robot monitoring method of the present application can include the following steps in combination with the above description. Figure 1

[0067] Step S100, at least two pose detection algorithms are used to obtain the current pose detection result of the robot.

[0068] Among them, the at least two pose detection algorithms can include a pose detection algorithm based on the robot's own sensor, and a pose detection algorithm based on an external sensor. Different pose detection algorithms can rely on different data, so the corresponding data can be obtained according to the requirements of each pose detection algorithm, and then the pose detection algorithm is called to determine the current pose detection result of the robot.

[0069] By using at least two different pose detection algorithms, pose detection is performed based on data collected by different sensors, which can complement each other and avoid the problem of inaccurate pose detection caused by a single algorithm affected by environmental factors.

[0070] ​In this embodiment, the pose detection result of the robot can be determined in real time, that is, the pose detection result of the robot at different time instants is determined in time sequence.

[0071] In step S110, the pose detection results of the at least two pose detection algorithms are fused to obtain a fused pose, and it is verified whether the fused pose is reliable.

[0072] Specifically, the pose detection results of the pose detection algorithms obtained in the previous step can be fused by using various fusion schemes to obtain a fused pose. Compared with the pose detection result obtained by a single algorithm, the fused pose has higher accuracy.

[0073] Further, in order to improve the reliability of the scheme, the reliability of the fused pose can be further verified in this step. Specifically, whether the fused pose is reliable can be verified based on the pose detection results of the pose detection algorithms. According to the difference of the pose detection algorithms adopted in the present application, the corresponding fusion scheme and reliability verification scheme can also be different, which will be introduced in the following embodiments.

[0074] In step S120, when the fused pose is reliable, the fused pose is saved as the current valid pose of the robot.

[0075] Specifically, if it is determined in the previous step that the fused pose is reliable, the fused pose can be saved as the current valid pose of the robot. If it is determined in the previous step that the fused pose is unreliable, the data of the current frame can be discarded, and the valid pose saved at the most recent time in the history is taken as the valid pose of the robot at the current time.

[0076] The robot monitoring method provided in the embodiments of the present application adopts at least two pose detection algorithms to obtain the current pose detection result of the robot in order to avoid the problem that a single algorithm is easily disturbed by environmental factors and the like, resulting in errors in the monitoring information. The pose detection algorithms can include a pose detection algorithm based on the sensors of the robot itself and a pose detection algorithm based on external sensors. The two algorithms perform pose detection based on the data collected by different sensors, and can complement each other. On this basis, the pose detection results of the at least two pose detection algorithms are fused to obtain a fused pose. Further, the fused pose can be cross-verified based on the pose detection results of the pose detection algorithms to verify the reliability of the fused pose. When it is determined that the fused pose is reliable, it is saved as the current valid pose of the robot. Obviously, the present application fuses the pose detection results of multiple different pose detection algorithms to obtain a fused pose, and verifies the reliability of the fused pose. The advantages of different pose detection algorithms can be combined, and the defects of a single algorithm affected by environmental factors and the like and having errors can be compensated for. The final valid pose is more accurate and has higher reliability.

[0077] In some embodiments of this application, different pose detection algorithms described in the aforementioned step S100 are illustrated by way of example.

[0078] There are various pose detection algorithms based on the robot's own sensors, with SLAM being a common one. SLAM technology is a technique that enables robots to autonomously perceive, understand, and construct environmental information in unknown environments, and it is widely used in robot navigation, autonomous driving, augmented reality, and other fields. The implementation of SLAM technology requires the comprehensive use of data from multiple sensors, such as LiDAR, cameras, and inertial measurement units, combined with machine learning, optimization algorithms, and other techniques for processing and optimization.

[0079] The SLAM algorithm mainly includes the following aspects:

[0080] 1. Sensor selection and data processing

[0081] Sensors are the core of SLAM technology, and their selection and data processing directly affect the accuracy and real-time performance of the SLAM algorithm. Commonly used sensors include LiDAR, cameras, and inertial measurement units (IMUs). LiDAR offers advantages such as high precision and long-range detection, making it suitable for localization and mapping of large robots; cameras, on the other hand, offer advantages such as low cost and ease of integration, making them suitable for localization and mapping of small or mobile robots.

[0082] 2. Feature extraction and matching

[0083] Feature extraction and matching are key steps in SLAM technology. Their purpose is to extract meaningful features from sensor data and match features acquired at different times to determine the robot's position and orientation. Common feature extraction methods include color histograms, Gabor filters, and Local Binary Patterns (LBP); common matching methods include feature point matching, feature vector matching, and deep learning-based feature matching.

[0084] 3. Pose estimation and optimization

[0085] Pose estimation refers to calculating a robot's position and orientation based on sensor data, while optimization refers to optimizing the robot's motion trajectory to enable it to complete tasks more quickly and efficiently. Common pose estimation methods include least squares-based methods, Bayesian filter-based methods, and particle filter-based methods; common optimization methods include gradient descent-based optimization algorithms, genetic algorithm-based optimization algorithms, and ant colony optimization algorithms.

[0086] 4. Map Building

[0087] Map building is a crucial component of SLAM technology. Its purpose is to organize and manage environmental information collected by the robot in unknown environments for subsequent use and analysis. Common map building methods include grid map building, topological map building, and LiDAR cloud map building. Grid map building is suitable for simple scenarios, offering high accuracy and stability; topological map building is suitable for complex scenarios, providing high scalability and flexibility; and LiDAR cloud map building is suitable for large-scale scenarios, offering high resolution and wide coverage.

[0088] This embodiment provides a simplified algorithm flow for the SLAM algorithm, which may include:

[0089] S1. Obtain data from the accelerometer and gyroscope in the IMU, and calculate the robot's angular velocity and angular displacement in the current frame.

[0090] S2. Calculate the robot's rotation matrix R and translation vector T in the current frame based on the angular velocity and angular displacement.

[0091] S3. Convert the rotation matrix R and translation vector T into quaternion representations for easier subsequent processing.

[0092] S4. Add the rotation and translation vectors represented by quaternions to the robot's state vector to represent the robot's state information in the current frame.

[0093] S5. Using the state transition equation, predict the state vector of the current frame based on the state vector of the previous frame and the sensor data of the current frame.

[0094] S6. Compare the predicted state vector of the current frame with the actual observed state vector of the current frame to obtain the state error. Then, use a Kalman filter to estimate and correct the state error to obtain the final state vector.

[0095] S7. Use the state vector as the robot's pose information for map building and optimization processes.

[0096] There are various pose detection algorithms based on external sensors, mainly mobile robot localization methods based on surveillance cameras. These methods use an external surveillance camera to project certain 3D feature points of the robot into 3D coordinates in the world coordinate system, thus enabling the robot's localization in that system. This involves two key technical aspects:

[0097] 1. Three-dimensional feature point detection.

[0098] 3D feature point detection refers to the process of detecting and extracting feature points from the surface of an object in three-dimensional space using computer vision technology. These feature points can be used to build 3D models of objects or for applications such as target tracking and recognition. Common 3D feature point detection algorithms include: Harris corner detection, SIFT feature detection, FAST feature detection, SURF feature detection, and corner detection with special markings.

[0099] 2. Three-dimensional projection transformation of feature points.

[0100] Since the process of a camera imaging a three-dimensional object into a two-dimensional image is a physical process, it is possible to inversely project a two-dimensional image back into three-dimensional space based on the principle of pinhole imaging, provided there is specific information.

[0101] Based on the combination of the above two technical points, it is possible to obtain the two-dimensional coordinates of key points representing the three-dimensional coordinates of the mobile robot in the image coordinate system through an external camera, and then obtain its three-dimensional coordinates in the world coordinate system through three-dimensional projection transformation.

[0102] There are various pose detection algorithms based on external sensors, and they can be further divided into different pose detection algorithms according to their different processing logic.

[0103] For example, visual positioning and detection algorithms based on special markers, such as ArUco-based pose detection algorithms, can be used. Figure 2 An example of an ArUco marker is provided. A pose detection algorithm based on ArUco markers determines the camera's position and orientation by detecting ArUco markers. An ArUco marker is a square marker consisting of a wide black border and an internal binary matrix that determines its identifier (id). The black border of the ArUco marker facilitates its rapid detection in an image, while the internal binary encoding is used for marker identification and to provide error detection and correction.

[0104] ArUco tags can be placed on the robot, allowing images of the robot to be acquired by an externally installed image sensor. Based on the acquired images carrying special tags, the robot's pose information can be determined through image vision algorithms.

[0105] It should be further noted that for visual positioning and detection algorithms based on special markers, these special markers (such as ArUco markers) can also be set in the robot's working environment. In this case, the robot's own image sensor can acquire images carrying the ArUco markers, and then the image vision algorithm can determine the robot's pose information. In this case, the visual positioning and detection algorithm based on special markers can be classified as a pose detection algorithm based on its own sensor.

[0106] In addition to external sensor-based pose detection algorithms, deep neural network-based pose detection algorithms can also be used. This deep neural network model takes environmental images captured by an external image sensor as input and can predict the robot's pose information.

[0107] During the training phase of the deep neural network model, the current frame training image of the robot in the environment, the planned target pose, and the robot's historical poses can be collected as training samples. The robot's pose at the current moment is used as the sample label. The training data is composed of training samples and sample labels to train the deep neural network model.

[0108] When using this deep neural network model to obtain the robot's current pose detection result, the current frame image, the target pose corresponding to the current frame, and the robot's historical poses can be acquired to form input state data; the input state data is then fed into the trained deep neural network model to obtain the robot's current pose detection result output by the model.

[0109] Based on the pose detection algorithms described above, the at least two pose detection algorithms used in step S100 of the above embodiments of this application may include the SLAM algorithm and a visual localization detection algorithm based on special markers. Furthermore, it may also include a pose detection algorithm based on a deep neural network model.

[0110] The three pose detection algorithms illustrated in the above embodiments each have their advantages. The SLAM algorithm is implemented based on sensors on the robot itself, so there is no blind spot problem. However, its accuracy is often not high, and it often produces incorrect pose results in complex dynamic scenes, leading to unstable monitoring results. The ArUco code-based detection algorithm is a fast visual detection algorithm with the advantages of low computational requirements and high detection accuracy. However, due to the small size of the detection target, it often loses the target due to robot rotation, occlusion, etc. The method based on a deep neural network model has better adaptability to occlusion than the ArUco code-based detection algorithm, but it is highly dependent on computational power, placing a heavy burden on the processor, and its accuracy is often lower than that of the ArUco code-based detection algorithm.

[0111] In summary, each algorithm has its advantages and disadvantages. The embodiments of this application can dynamically call the above algorithms according to the actual situation, fuse the output results, and finally output the real-time fused pose of the current robot to the outside.

[0112] Reference Figure 3 The example flowchart illustrates a robot monitoring method employing the three pose detection algorithms described above, which may specifically include:

[0113] Step S1: Use the SLAM algorithm to estimate the pose and obtain the robot's current first pose.

[0114] Specifically, data collected by the robot's own sensors can be obtained, such as radar data collected by LiDAR and inertial data measured by inertial sensors. The SLAM algorithm can then be called to calculate the pose based on the sensor data to obtain the robot's current first pose.

[0115] Step S2: Use an algorithm based on ArUco markers to estimate the pose and obtain the robot's current second pose.

[0116] Specifically, this step can employ visual positioning and detection algorithms based on special markers, such as algorithms based on ArUco markers. The robot's current second position can be calculated using images acquired by an image sensor. The image sensor can be an external image sensor or the robot's own image sensor, depending on whether the special marker is located on the robot or in the environment.

[0117] Step S3: Perform pose fusion voting on the first pose and the second pose to obtain the fused pose.

[0118] This embodiment can employ various fusion voting strategies, such as averaging and weighted averaging. This embodiment further provides another pose fusion voting scheme based on Kalman filtering.

[0119] Specifically, the covariance matrices of the first pose and the second pose can first be set empirically. Then, the first pose and the second pose are fused using Kalman filtering and the covariance matrix to obtain the fused pose.

[0120] Kalman filtering is an algorithm that uses the state equations of a linear system to make an optimal estimate of the system state using the system's input and output observation data. Since the observation data includes the effects of noise and interference in the system, the optimal estimation can also be regarded as a filtering process.

[0121] Step S4: First determine whether the fused pose is reliable. If yes, proceed to step S8; otherwise, proceed to step S5.

[0122] Specifically, the reliability of the fused pose is initially assessed based on the differences between the fused pose and the first and second poses. If the difference between the fused pose and the first pose is within a set range, and the difference between the fused pose and the second pose is also within a set range, the fused pose is confirmed to be reliable, and step S8 is executed. Otherwise, the fused pose is considered to have failed the initial reliability assessment, and step S5 is executed.

[0123] Step S5: Use the pose detection algorithm based on the neural network model to obtain the robot's current third pose and confidence level.

[0124] Specifically, considering that the pose detection algorithm based on the neural network model has higher requirements for computing power, in order to reduce the computational overhead of the processor, in this embodiment, the calculation process of this step is only executed when the fused pose is determined to be unreliable in step S4 above, thereby reducing the amount of computation as much as possible.

[0125] The robot's current third pose and confidence level can be calculated using a neural network model. The confidence level indicates the reliability of the calculated third pose.

[0126] Step S6: Make a second judgment on whether the fused pose is reliable. If yes, proceed to step S8; otherwise, proceed to step S7.

[0127] Specifically, the difference between the fused pose and the third pose can be compared. If the difference is within a set range and the confidence level is not lower than a set confidence threshold, the fused pose is ultimately determined to be reliable, and step S8 is executed. Otherwise, the fused pose is ultimately determined to be unreliable, and step S7 is executed.

[0128] Step S7: Discard the current frame.

[0129] Specifically, if the fused pose is determined to be unreliable, the current frame can be discarded.

[0130] Step S8: Confirm valid pose frame.

[0131] Specifically, when determining that the fused pose is reliable, the valid frame of the current pose can be confirmed.

[0132] Step S9: Output the latest valid frame and save the fused pose as the robot's current valid pose.

[0133] As shown in the method provided in this embodiment, when simultaneously employing the SLAM algorithm, a pose detection algorithm based on special markers, and a pose detection algorithm based on a neural network model, the results of the first two algorithms can be fused first, and an initial reliability judgment of the fused pose can be performed. If the fused pose is determined to be reliable, a valid pose frame can be confirmed, and the latest fused pose can be output. If the initial determination of the fused pose is unreliable, a third pose detection algorithm can be further invoked to perform a second reliability judgment on the fused pose. If the result of the second reliability judgment is reliable, the fused pose can still be considered reliable; if the result of the second reliability judgment is still unreliable, the fused pose is considered indeed unreliable, and the current frame can be discarded. This embodiment's method, while minimizing processor computation, ensures the accuracy of the fused pose through the fusion and cross-validation of multiple algorithms.

[0134] Further reference Figure 4 It illustrates another robot monitoring method using the three pose detection algorithms mentioned above, with a flowchart for comparison. Figure 3 and Figure 4 It can be seen that step S10 can be added between steps S8 and S9 above: determine whether the fused pose is reasonable.

[0135] If the result indicates that the fused pose is reasonable, proceed to step S9; otherwise, proceed to step S7.

[0136] The process of determining whether the fused pose is reasonable may specifically include:

[0137] Based on the planned target pose and historical motion states of the robot, the reasonable range of the current pose is determined. It is then determined whether the fused pose is within the reasonable range of the current pose. If so, the fused pose is determined to be reasonable; otherwise, it is determined to be unreasonable.

[0138] Combination Figure 5 As shown, the forward direction is the planned forward direction, the dashed circle z5 is the planned target pose, the solid circle z1 is the pose at the previous moment, the dashed circle z3 is the ideal current pose during the forward movement, and the dashed circles z2 and z4 are the possible ranges of the current pose (the leftmost and rightmost ones), respectively.

[0139] The reasonable range of the current pose can be determined based on the planned target pose and historical motion state of the robot. If the fused pose is within the reasonable range of the current pose, it means that the fused pose conforms to the trend of the robot's historical trajectory changes, and the pose change does not exceed the reasonable range of change, i.e., the fused pose is reasonable; otherwise, it means that the fused pose is unreasonable.

[0140] In this embodiment, after determining that the fused pose is reliable, the rationality of the fused pose can be further judged. If the fused pose is reasonable, the process of saving the fused pose as the robot's current valid pose can be executed; otherwise, the current frame can be discarded, which further improves the accuracy of robot pose monitoring.

[0141] In some embodiments of this application, another alternative solution for robot monitoring method is provided. Based on the aforementioned monitoring of robot pose, this embodiment can further monitor and analyze the robot's operating status.

[0142] Specifically, the robot's navigation planning unit pre-plans the robot's theoretical operating trajectory, which includes timestamps and corresponding operating state information for each timestamp. Due to obstacles, algorithms, physical factors, etc., the robot's actual operating trajectory may differ from the theoretical trajectory. This application can compare the differences between the actual and planned theoretical trajectories and evaluate them. If the difference exceeds a set tolerance, a prompt message can be output. Examples include issuing a warning message or an error message.

[0143] The robot's operating state can include various types, including but not limited to the following:

[0144] 1) Internal status: Whether various sensors are online, system operating status, etc.

[0145] 2) Action status: robot's movement speed, triggering obstacle avoidance actions, etc.

[0146] 3) Event status: Whether the robot has collided or is avoiding obstacles, etc.

[0147] Furthermore, this application also provides a scheme for analyzing the causes of abnormal states, which can perform correlation analysis on different states and trace the cause of the abnormality. Compared with the conventional approach of detecting each state separately, it can detect the cause of the abnormality faster and more accurately.

[0148] The specific process may include:

[0149] S1. Based on the difference between the actual operating state trajectory and the planned theoretical operating state trajectory, determine the target state in which the anomaly occurred.

[0150] Specifically, by comparing two motion state trajectories, if the actual running state and the theoretical running state at the same time point differ too much, it can be determined that the corresponding target state has become abnormal.

[0151] S2. Based on the pre-configured state association relationships, determine the relevant states associated with the target state, and the subordinate relationships between the relevant states.

[0152] Specifically, embodiments of this application can pre-configure the association relationships between different states based on the relationships between them. For example, for the event state class "collision occurs", it can be associated with the following states in sequence according to the subordinate relationship:

[0153] ① Event status class: Whether a movement task is being executed, etc.

[0154] ② Action status: robot speed, robot pose, obstacle speed, etc.

[0155] ③ Internal status categories: IMU, LiDAR, RGB camera working status, etc.

[0156] Furthermore, it is possible to find the relevant states associated with the target state, as well as the hierarchical relationships between these relevant states.

[0157] S3. According to the subordinate relationship, traverse each of the related states in sequence, retrieve the attribute information of the attribute items of the related states, and perform reliability analysis of the attribute items based on the attribute information.

[0158] Specifically, for each relevant state, the attribute information of the attribute items for that state can be retrieved. Different relevant states may correspond to different attribute items. For example, for the event state class "Is a mobile task being executed?", its attribute items are the event state class itself, and the attribute information corresponding to each item is "yes" or "no". As another example, for the internal state class "LiDAR working status", its attribute items may include: transmission frequency, whether in standby mode, whether abnormal, whether normal, whether a message has been published externally, etc.

[0159] For each relevant state, reliability analysis can be performed on the attribute items based on the attribute information of the relevant state. The attribute information of the attribute items in the relevant state can include historical attribute information, which facilitates reliability analysis.

[0160] Taking the action status class "robot speed" as an example, its attribute value is a specific speed value. If the analysis finds that the robot speed value exceeds the set speed limit, it can be determined that the "robot speed" attribute has become abnormal and does not meet the reliability conditions.

[0161] S4. Filter out target attribute items that do not meet the reliability conditions and obtain the corresponding exception tags for the target attribute items.

[0162] Specifically, for the target attribute items that do not meet the reliability conditions as determined by the above analysis, their corresponding anomaly labels can be obtained. Each attribute item can be pre-configured with a corresponding anomaly label; for example, for the attribute item "robot speed," the configured anomaly label could be "abnormal speed."

[0163] S5. The set of abnormal causes for the target state is composed of each of the aforementioned abnormal labels.

[0164] Through the analysis of the above steps, we can obtain the anomaly labels of each relevant state associated with the target state that has an anomaly, and these anomaly labels form a set of anomaly causes for the target state.

[0165] According to the method provided in this embodiment, by pre-establishing the correlation between each state, after an anomaly in a target state is detected, the possible cause of the anomaly can be quickly traced based on the correlation.

[0166] Next, taking the abnormal state of "collision occurred" as an example, the above process will be explained:

[0167] First, based on the configured state associations, the related states associated with "collision occurred" can be obtained as follows:

[0168] ④ Event status class: Whether a movement task is being executed, etc.

[0169] ⑤ Action status: robot speed, robot pose, obstacle speed, etc.

[0170] ⑥ Internal status: IMU, LiDAR, RGB camera working status, etc.

[0171] Based on the hierarchical relationships between related states, an anomaly analysis process is generated:

[0172] 1. First, trace the event state class being processed—whether a movement task is being performed. If the robot is performing a movement task, then label the exception as "Collision occurred during active movement". Otherwise, label it as "Collision occurred passively".

[0173] 2. Then trace back to the action state class—speed, pose, etc. If the robot exceeds the speed limit or the pose changes abnormally (e.g., the robot pose relocalization has just been triggered, or the localization is abnormal), then add the corresponding tags, such as "abnormal speed" or "abnormal localization".

[0174] 3. Finally, trace back to the internal state class. Check for any anomalies in the internal state, such as message delays in the LiDAR or IMU drift. Similarly, tag the anomalies accordingly.

[0175] In some embodiments of this application, another alternative solution for robot monitoring method is provided. Based on the aforementioned solution, this embodiment can further monitor and analyze the robot's movement trajectory.

[0176] like Figure 6 As shown, the monitoring process includes:

[0177] The valid poses of the robot at each timestamp within the set time period before the current moment are stored in the real-time pose queue, and the planned target poses of the robot at each timestamp are stored in the control pose queue.

[0178] Align the real-time pose queue and the control pose queue according to timestamps, and calculate the pose difference for each timestamp.

[0179] Based on the pose difference, determine whether the robot's current pose needs to be corrected. If so, send a correction command to the robot.

[0180] The method for monitoring and analyzing robot movement trajectories provided in this embodiment offers basic operational assurance during the initial stages when robot navigation / obstacle avoidance algorithms are unstable. This prevents frequent interruptions due to trajectory anomalies, thus avoiding low testing / operation efficiency. Furthermore, combined with the anomaly tracing mechanism described in the aforementioned embodiment, it effectively generates a set of error test cases for algorithm iteration.

[0181] The robot monitoring device provided in the embodiments of this application is described below. The robot monitoring device described below and the robot monitoring method described above can be referred to in correspondence.

[0182] See Figure 7 , Figure 7 This is a schematic diagram of the structure of a robot monitoring device disclosed in an embodiment of this application.

[0183] like Figure 7 As shown, the device may include:

[0184] The pose calculation unit 71 is used to obtain the current pose detection result of the robot by employing at least two pose detection algorithms. The at least two pose detection algorithms include a pose detection algorithm based on the robot's own sensors and a pose detection algorithm based on external sensors.

[0185] The pose fusion unit 72 is used to fuse the pose detection results of the at least two pose detection algorithms to obtain a fused pose, and to verify the reliability of the fused pose based on the pose detection results of each pose detection algorithm.

[0186] The effective pose storage unit 73 is used to save the fused pose as the robot's current effective pose when the fused pose is reliable.

[0187] Optionally, the pose calculation unit employs at least two pose detection algorithms, including:

[0188] Simultaneous Localization and Mapping (SLAM) algorithm, and visual localization and detection algorithm based on special markers.

[0189] The aforementioned visual positioning and detection algorithms based on special markers include: a pose detection algorithm based on ArUco markers.

[0190] Further, optionally, the pose calculation unit may employ at least two pose detection algorithms, including:

[0191] The pose detection algorithm based on a deep neural network model uses training samples that include: the current frame training image, the target pose, and the robot's historical poses. The sample label is the robot's pose at the current moment. Based on this, the pose calculation unit uses the pose detection algorithm based on the deep neural network model to obtain the robot's current pose detection result, which includes:

[0192] The current frame image, the target pose corresponding to the current frame, and the robot's historical poses are obtained to form the input state data.

[0193] The input state data is fed into the trained deep neural network model to obtain the robot's current pose detection result output by the model.

[0194] Optionally, the process by which the pose calculation unit uses at least two pose detection algorithms to obtain the robot's current pose detection result may include:

[0195] First, the SLAM algorithm is used to obtain the robot's current first pose, and a visual positioning and detection algorithm based on special markers is used to obtain the robot's current second pose.

[0196] Upon receiving the first instruction from the pose fusion unit, a pose detection algorithm based on the neural network model is further employed to obtain the robot's current third pose and confidence level.

[0197] The above-mentioned pose fusion unit fuses the pose detection results of the at least two pose detection algorithms to obtain a fused pose, and verifies the reliability of the fused pose based on the pose detection results of each of the pose detection algorithms, including:

[0198] A pose fusion vote is performed on the first pose and the second pose to obtain the fused pose;

[0199] The reliability of the fused pose is initially determined based on the magnitude of the difference between the fused pose and the first pose and the second pose.

[0200] If the initial credibility assessment determines that the fused pose is credible, then the fused pose is ultimately determined to be credible. If the initial credibility assessment determines that the fused pose is unreliable, then the first instruction is sent to the pose calculation unit. After the pose calculation unit calculates the robot's current third pose and confidence level, the difference between the fused pose information and the third pose is compared. If the difference is within a set difference range and the confidence level is not lower than a set confidence level threshold, then the fused pose is ultimately determined to be credible; otherwise, the fused pose is ultimately determined to be unreliable.

[0201] Optionally, the apparatus of this application may further include:

[0202] The rationality judgment unit is used to determine the reasonable range of the current pose based on the planned target pose and historical motion state of the robot before the effective pose saving unit saves the fused pose as the robot's current effective pose. If the fused pose is deemed reliable, the unit determines whether the fused pose is within the reasonable range of the current pose. If so, the unit determines that the fused pose is reasonable and jumps to the effective pose saving unit. If not, the unit determines that the fused pose is unreasonable and discards the current frame.

[0203] Optionally, the process by which the pose fusion unit performs pose fusion voting on the first pose and the second pose to obtain the fused pose includes:

[0204] Define the covariance matrix of the first pose and the second pose;

[0205] The first pose and the second pose are used to calculate the filtered and fused result using Kalman filtering and the covariance matrix to obtain the fused pose.

[0206] Optionally, the apparatus of this application may further include:

[0207] The running status trajectory acquisition unit is used to acquire the actual running status trajectory of the robot, the running status trajectory including timestamps and running status information corresponding to each timestamp;

[0208] The trajectory comparison unit is used to compare the actual operating trajectory with the planned theoretical operating trajectory. If the difference exceeds the set tolerance, a prompt message will be output.

[0209] Optionally, the above operating status information includes any one or more of the following:

[0210] Internal state class, action state class, event state class.

[0211] Optionally, the apparatus of this application may further include:

[0212] An abnormal state determination unit is used to determine the target state in which an abnormality has occurred based on the difference between the actual operating state trajectory and the planned theoretical operating state trajectory.

[0213] The associated state determination unit is used to determine, based on the pre-configured state association relationships, each associated state with the target state, and the subordinate relationships between each associated state;

[0214] The associated state attribute item analysis unit is used to sequentially traverse each of the associated states according to the subordinate relationship, retrieve the attribute information of the attribute items of the associated states, and perform reliability analysis of the attribute items based on the attribute information.

[0215] The anomaly tag acquisition unit is used to filter target attribute items that do not meet reliability conditions and acquire the anomaly tags corresponding to the target attribute items.

[0216] An anomaly cause determination unit is used to form a set of anomaly causes for the target state, which consists of each of the anomaly labels.

[0217] Optionally, the apparatus of this application may further include: a trajectory monitoring unit, used to perform the following process:

[0218] The effective poses of the robot at each timestamp within a set time period before the current time are stored in the real-time pose queue, and the planned target poses of the robot at each timestamp are stored in the planning pose queue.

[0219] Align the real-time pose queue and the control pose queue according to timestamps, and calculate the pose difference between each timestamp;

[0220] Based on the pose difference, determine whether the robot's current pose needs to be corrected. If so, send a correction command to the robot.

[0221] The robot monitoring device provided in this application embodiment can be applied to robot monitoring equipment, which can be the robot being monitored itself, or a base station or other terminal communicating with the robot being monitored. Optionally, Figure 8 The hardware structure block diagram of the robot monitoring equipment is shown. Figure 8 The hardware structure of the robot monitoring equipment may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;

[0222] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;

[0223] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0224] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0225] The memory stores a program, which the processor can call. The program is used for:

[0226] At least two pose detection algorithms are used to obtain the current pose detection result of the robot. The at least two pose detection algorithms include a pose detection algorithm based on the robot's own sensors and a pose detection algorithm based on external sensors.

[0227] The pose detection results of the at least two pose detection algorithms are fused to obtain a fused pose, and the reliability of the fused pose is verified based on the pose detection results of each of the pose detection algorithms.

[0228] When the fused pose is reliable, the fused pose is saved as the robot's current valid pose.

[0229] Optionally, the refined and extended functions of the program can be found in the description above.

[0230] This application embodiment also provides a storage medium that can store a program suitable for execution by a processor, the program being used for:

[0231] At least two pose detection algorithms are used to obtain the current pose detection result of the robot. The at least two pose detection algorithms include a pose detection algorithm based on the robot's own sensors and a pose detection algorithm based on external sensors.

[0232] The pose detection results of the at least two pose detection algorithms are fused to obtain a fused pose, and the reliability of the fused pose is verified based on the pose detection results of each of the pose detection algorithms.

[0233] When the fused pose is reliable, the fused pose is saved as the robot's current valid pose.

[0234] Optionally, the refined and extended functions of the program can be found in the description above.

[0235] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0236] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0237] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A robot monitoring method characterized by, The method comprises the following steps: adopting at least two pose detection algorithms to obtain current pose detection results of the robot respectively, wherein the at least two pose detection algorithms comprise a pose detection algorithm based on a sensor of the robot itself and a pose detection algorithm based on an external sensor; fusing the pose detection results of the at least two pose detection algorithms to obtain a fused pose, and verifying whether the fused pose is credible based on the pose detection results of each of the pose detection algorithms; when the fused pose is credible, saving the fused pose as a current valid pose of the robot; the process of adopting at least two pose detection algorithms to obtain current pose detection results of the robot respectively, fusing the pose detection results of the at least two pose detection algorithms to obtain a fused pose, and verifying whether the fused pose is credible based on the pose detection results of each of the pose detection algorithms comprises the following steps: adopting two pose detection algorithms to obtain a first pose and a second pose of the robot; performing pose fusion voting on the first pose and the second pose to obtain a fused pose; performing a first credibility judgment on the fused pose according to the difference between the fused pose and each of the first pose and the second pose; if the first credibility judgment determines that the fused pose is credible, finally determining that the fused pose is credible; if the first credibility judgment determines that the fused pose is not credible, adopting a third pose detection algorithm to obtain a third pose and a confidence degree of the robot at present; comparing the difference between the fused pose and the third pose, if the difference is within a set difference range and the confidence degree is not lower than a set confidence threshold, finally determining that the fused pose is credible, otherwise, finally determining that the fused pose is not credible.

2. The method of claim 1, wherein, The at least two pose detection algorithms comprise: a simultaneous localization and mapping (SLAM) algorithm and a special marker-based visual positioning detection algorithm.

3. The method of claim 2, wherein, The special marker-based visual positioning detection algorithm comprises an ArUco marker-based pose detection algorithm.

4. The method of claim 2, wherein, The at least two pose detection algorithms further comprise: a pose detection algorithm based on a deep neural network model, wherein the training samples of the deep neural network model during training comprise a current frame training picture, a target pose and a historical pose of the robot, and a sample label is a pose of the robot at a current time; the process of adopting the pose detection algorithm based on the deep neural network model to obtain the current pose detection result of the robot comprises the following steps: obtaining a current frame picture, a target pose corresponding to the current frame and a historical pose of the robot to form input state data; sending the input state data into the trained deep neural network model to obtain a current pose detection result of the robot output by the model.

5. The method of claim 4, wherein, The process of adopting two pose detection algorithms to obtain a first pose and a second pose of the robot at present comprises the following steps: adopting the SLAM algorithm to obtain a first pose of the robot, and adopting a special marker-based visual positioning detection algorithm to obtain a second pose of the robot at present; the process of adopting a third pose detection algorithm to obtain a third pose and a confidence degree of the robot at present comprises the following steps: A third pose of the robot and a confidence level of the third pose are obtained by using a pose detection algorithm based on the neural network model.

6. The method of claim 1, wherein, Before the fusion pose is saved as the current valid pose of the robot, the method further comprises: When the fusion pose is reliable, a reasonable range of the current pose is determined according to a target pose of the robot and a historical motion state of the robot; It is determined whether the fusion pose is within the reasonable range of the current pose, and if so, it is determined that the fusion pose is reasonable, and the step of saving the fusion pose as the current valid pose of the robot is performed, and if not, it is determined that the fusion pose is not reasonable, and the current frame is discarded.

7. The method of claim 1, wherein, The first pose and the second pose are fused to obtain a fusion pose, comprising: The covariance matrix of the first pose and the second pose is set; The first pose and the second pose are fused by Kalman filtering and the covariance matrix to obtain a fusion pose.

8. The method of claim 1, wherein, Further comprising: An actual running state trajectory of the robot is obtained, and the running state trajectory comprises a timestamp and running state information corresponding to each timestamp; The difference between the actual running state trajectory and a planned theoretical running state trajectory is compared, and if the difference exceeds a set tolerance, a prompt message is output.

9. The method of claim 8, wherein, The running state information comprises any one or more of the following: Internal state class, action state class, event state class.

10. The method of claim 8, wherein, Further comprising: According to the difference between the actual running state trajectory and the planned theoretical running state trajectory, a target state of an abnormality is determined; According to a pre-configured association relationship between states, each related state associated with the target state and a dependency relationship between the related states are determined; According to the dependency relationship, each related state is traversed in turn, attribute information of an attribute item of the related state is retrieved, and reliability analysis of the attribute item is performed based on the attribute information; Target attribute items that do not meet the reliability condition are screened, and an abnormality label corresponding to the target attribute item is obtained; An abnormality cause set of the target state is composed of each abnormality label.

11. The method according to any one of claims 1 to 10, characterized in that, Further comprising: The valid poses of the robot at each timestamp within a set time period before the current time are stored in a real-time pose queue, and the target poses of the robot at each timestamp are stored in a regulation and control pose queue; The real-time pose queue and the regulation and control pose queue are aligned according to the timestamps, and the pose difference at each timestamp is calculated; Based on the pose difference, it is determined whether the current pose of the robot needs to be corrected, and if so, a correction instruction is sent to the robot.

12. A robot monitoring device, characterized by The method comprises: A pose calculation unit is configured to obtain pose detection results of the robot by using at least two pose detection algorithms, wherein the at least two pose detection algorithms comprise a pose detection algorithm based on a sensor of the robot and a pose detection algorithm based on an external sensor; A pose fusion unit is configured to fuse the pose detection results of the at least two pose detection algorithms to obtain a fusion pose, and verify whether the fusion pose is reliable based on the pose detection results of each pose detection algorithm; An effective pose saving unit configured to save the fusion pose as a current effective pose of the robot when the fusion pose is trusted; The pose calculation unit and the pose fusion unit employ at least two pose detection algorithms to obtain current pose detection results of the robot, fuse the pose detection results of the at least two pose detection algorithms to obtain a fusion pose, and verify whether the fusion pose is trusted based on the pose detection results of each of the pose detection algorithms, including: employing two pose detection algorithms to obtain a first pose and a second pose of the robot; performing pose fusion voting on the first pose and the second pose to obtain a fusion pose; performing a first trustworthiness judgment on the fusion pose according to the difference between the fusion pose and each of the first pose and the second pose; if the first trustworthiness judgment determines that the fusion pose is trusted, then finally determining that the fusion pose is trusted; if the first trustworthiness judgment determines that the fusion pose is not trusted, then employing a third pose detection algorithm to obtain a third pose and a confidence of the robot; comparing the difference between the fusion pose and the third pose, and if the difference is within a set difference range and the confidence is not lower than a set confidence threshold, then finally determining that the fusion pose is trusted, otherwise, finally determining that the fusion pose is not trusted.

13. A robot monitoring device, characterized by comprising: a memory and a processor; the memory is configured to store a program; the processor is configured to execute the program to implement each step of the robot monitoring method according to any one of claims 1-11.

14. A storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement each step of the robot monitoring method according to any one of claims 1-11.

Images