Multi-sensor anomaly detection method and device, storage medium and program product

Abstract

The embodiment of the invention provides a multi-sensor anomaly detection method and device, a storage medium and a program product, and the method comprises the steps: receiving topic information sent by a plurality of subscribed target sensors through an ROS node, obtaining the current system time for each target sensor in response to the received sensing data sent by the target sensor, and sending the current system time to the ROS node; determining the current system time as the last message timestamp of the target sensor, generating log information, analyzing the log information corresponding to the plurality of target sensors based on a preset period, and for each target sensor, determining the current system time as the last message timestamp of the target sensor; and if the difference value between the current time and the last message timestamp newly recorded in the log information corresponding to the target sensor is greater than a preset threshold value, marking the target sensor as an abnormal sensor. According to the method, the problems such as delay and packet loss of sensor data transmission can be found in time, and the real-time performance of sensor monitoring and the system robustness are improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of robotics technology, and in particular, to a multi-sensor anomaly detection method, device, storage medium, and program product. Background Art

[0002] In the field of modern robotics, multi-sensor data fusion technology has become a key technology for achieving high-precision environmental perception and decision-making.

[0003] In the related art, whether the sensor is abnormal can be detected by checking whether the sensor noise amplitude is too large.

[0004] However, in the process of implementing the present application, the inventors found that there are at least the following problems in the prior art: the above-mentioned method cannot timely detect the problem of sensor data delay or loss, which may easily lead to misjudgment. Summary of the invention

[0005] The embodiments of the present application provide a multi-sensor anomaly detection method, device, storage medium and program product to timely discover the problem of sensor data delay or loss, and improve the real-time and system robustness of sensor monitoring.

[0006] In a first aspect, an embodiment of the present application provides a multi-sensor anomaly detection method, comprising:

[0007] Receiving topic information sent by a plurality of subscribed target sensors through a robot operating system ROS node; the topic information includes sensor data corresponding to the target sensor;

[0008] For each target sensor, in response to receiving the sensor data sent by the target sensor, obtaining the current system time, determining the current system time as the last message timestamp of the target sensor, and generating log information;

[0009] Based on a preset period, the log information corresponding to the plurality of target sensors is analyzed respectively;

[0010] For each target sensor, if the difference between the current time and the last message timestamp of the latest record in the log information corresponding to the target sensor is greater than a preset threshold, the target sensor is marked as an abnormal sensor.

[0011] In a possible design, the target sensor includes an IMU sensor, an image sensor, or a point cloud sensor; and the preset period is twice the preset threshold.

[0012] In a possible design, if the difference between the current time and the last message timestamp of the latest record in the log information corresponding to the target sensor is greater than a preset threshold, the target sensor is marked as an abnormal sensor, including:

[0013] If the time difference between the latest recorded last message timestamp and the previous last message timestamp in the log information corresponding to the target sensor is greater than the latest receiving cycle, the target sensor is determined as an abnormal sensor; the latest receiving cycle is determined based on a preset number of last message timestamps before the previous last message timestamp recorded in the log information.

[0014] In a possible design, if the difference between the current time and the last message timestamp of the latest record in the log information corresponding to the target sensor is greater than a preset threshold, the target sensor is marked as an abnormal sensor, including:

[0015] If the difference between the current time and the last message timestamp of the latest record in the log information corresponding to the target sensor is greater than a preset threshold, the log information corresponding to the target sensor is input into the deep learning model for abnormality identification, and an identification result is obtained, and the abnormal sensor is determined according to the identification result.

[0016] In one possible design, the method further includes:

[0017] Creating a ROS node and assigning a unique identifier to the ROS node;

[0018] Based on the unique identifier, configuring the ROS node as a subscriber of multiple target sensors;

[0019] A timer is created, and the preset period is set by the timer.

[0020] In one possible design, the method further includes:

[0021] In response to detecting an abnormal sensor, performing at least one of the following:

[0022] Generate a warning log, and display the warning log on a display screen of a console so that a user can make a preliminary diagnosis of an abnormality based on the warning log; the warning log includes an identification of an abnormal sensor;

[0023] Publishing a topic of error information so that other ROS nodes except the ROS node receive the topic of error information and perform corresponding response processing according to the error information;

[0024] Capturing information of the abnormal sensor to obtain abnormal information of the abnormal sensor, so that a user can locate and repair the abnormality according to the abnormal information; the abnormal information includes stack information;

[0025] Switching the abnormal sensor to a corresponding standby sensor, determining the standby sensor as a new target sensor, and receiving topic information sent by the new target sensor;

[0026] Continue to perform abnormality detection on the remaining target sensors except the abnormal sensor, and do not issue a control signal for terminating the operation of the system.

[0027] In a second aspect, an embodiment of the present application provides a multi-sensor anomaly detection device, including:

[0028] A receiving module, used for receiving topic information sent by a plurality of subscribed target sensors through a robot operating system ROS node; the topic information includes sensor data corresponding to the target sensor;

[0029] a determination module, configured to obtain, for each target sensor, a current system time in response to receiving sensor data sent by the target sensor, determine the current system time as the last message timestamp of the target sensor, and generate log information;

[0030] The parsing module is used to parse the log information corresponding to the plurality of target sensors respectively based on a preset period; for each target sensor, if the difference between the current time and the last message timestamp of the latest record in the log information corresponding to the target sensor is greater than a preset threshold, the target sensor is marked as an abnormal sensor.

[0031] In a third aspect, an embodiment of the present application provides a multi-sensor anomaly detection device, including: at least one processor and a memory;

[0032] The memory stores computer-executable instructions;

[0033] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the method described in the first aspect and various possible designs of the first aspect.

[0034] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method described in the first aspect and various possible designs of the first aspect are implemented.

[0035] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the method described in the first aspect and various possible designs of the first aspect.

[0036] The multi-sensor anomaly detection method, device, storage medium and program product provided in this embodiment include receiving topic information sent by multiple subscribed target sensors through a robot operating system ROS node, the topic information includes sensor data of the corresponding target sensor, for each target sensor, in response to receiving the sensor data sent by the target sensor, obtaining the current system time, determining the current system time as the last message timestamp of the target sensor, and generating log information, parsing the log information corresponding to the multiple target sensors based on a preset period, and for each target sensor, if the difference between the current time and the last message timestamp of the latest record in the log information corresponding to the target sensor is greater than a preset threshold, then the target sensor is marked as an abnormal sensor. The method provided in this embodiment receives the sensor data of multiple sensors in real time through the ROS node, and quickly determines the abnormal sensor by periodically detecting the timestamp when the sensor data is received, which can timely discover problems such as sensor data transmission delay and packet loss, and improve the real-time and system robustness of sensor monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0038] Figure 1 A schematic diagram of an application scenario of the multi-sensor anomaly detection method provided in an embodiment of the present application;

[0039] Figure 2 A schematic diagram of a multi-sensor anomaly detection method according to an embodiment of the present application;

[0040] Figure 3 A schematic diagram of the process of initializing a ROS node provided in an embodiment of the present application;

[0041] Figure 4 A schematic diagram of the process of receiving sensor data and updating timestamps provided in an embodiment of the present application;

[0042] Figure 5 A schematic diagram of a flow chart of sensor status detection provided in an embodiment of the present application;

[0043] Figure 6 A flowchart of a process for processing an abnormal sensor after an abnormal sensor is detected provided in an embodiment of the present application;

[0044] Figure 7A flowchart of a process for processing a sensor that does not have an abnormality after an abnormal sensor is detected provided by an embodiment of the present application;

[0045] Figure 8 A schematic diagram of the structure of a multi-sensor anomaly detection device provided in an embodiment of the present application;

[0046] Fig. 9 A schematic diagram of the hardware structure of a multi-sensor anomaly detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0048] It should be noted that the multi-sensor anomaly detection method, device, storage medium and program product provided in the present application can be used in the field of robotics, and can also be used in any field other than the field of robotics. The application field of the multi-sensor anomaly detection method, device, storage medium and program product provided in the present application is not limited.

[0049] In the field of modern robotics and autonomous driving, multi-sensor data fusion technology has become a key technology for achieving high-precision environmental perception and decision-making. By simultaneously acquiring data from different types of sensors (such as inertial measurement units (IMUs), cameras, and lidars), robots can obtain more comprehensive and reliable environmental information in dynamic and complex environments. However, existing technologies still have problems such as sensor data loss and insufficient real-time monitoring of sensor anomalies in multi-sensor data reception and management. Specifically, in practical applications, sensor data may be intermittently lost or delayed due to network delays, hardware failures, or other unexpected factors. This problem will cause the robot to lack key information during the data fusion stage, thereby affecting the robustness and accuracy of the system. Existing multi-sensor systems usually lack a real-time monitoring mechanism for sensor status. When a sensor does not send data for a long time, it is often difficult for the system to detect and handle the problem in a timely manner. This defect may cause potential risks in the operation of the robot, especially when performing autonomous navigation or dynamic object tracking in complex environments.

[0050] Therefore, how to achieve real-time monitoring and management of multi-sensor data and timely detect and handle abnormal phenomena such as sensor data loss and delay has become an important research topic in the field of robot perception and control. Especially in dynamic environments, having efficient sensor status monitoring capabilities will directly affect the stability of the robot system and the efficiency of task completion.

[0051] In response to the above technical problems, the inventors of this application have found that by introducing a monitoring node for multiple sensors, real-time data reception and status check of multiple sensors such as IMU, camera and lidar can be achieved, and sensor abnormalities such as data loss or delay can be detected and discovered in a timely manner by periodically and regularly checking the latest message time of each sensor, thereby improving the real-time performance and system robustness of sensor monitoring. Based on this, the embodiment of this application improves a multi-sensor abnormality detection method.

[0052] Figure 1 Schematic diagram of application scenarios of the multi-sensor anomaly detection method provided in the embodiment of the present application. Figure 1 As shown, the humanoid robot includes multiple sensors. The sensor may be an IMU sensor, an image sensor, or a point cloud sensor. A Robot Operating System (ROS) node is a monitoring node specially created to detect multiple sensors.

[0053] In the specific implementation process, the ROS node can be configured as a subscriber of multiple sensors, and then the sensor data is sent to the ROS node in the form of a topic message. The method provided in this embodiment receives topic information sent by multiple subscribed target sensors through the ROS node, and the topic information includes the sensor data of the corresponding target sensor. For each target sensor, in response to receiving the sensor data sent by the target sensor, the current system time is obtained, the current system time is determined as the last message timestamp of the target sensor, and log information is generated. Based on a preset period, the log information corresponding to the multiple target sensors is parsed. For each target sensor, if the difference between the current time and the last message timestamp of the latest record in the log information corresponding to the target sensor is greater than a preset threshold, the target sensor is marked as an abnormal sensor. The multi-sensor anomaly detection method provided in the embodiment of the present application receives the sensor data of multiple sensors in real time through the ROS node, and quickly determines the abnormal sensor by periodically detecting the timestamp when the sensor data is received, and can timely discover problems such as sensor data transmission delay and packet loss, thereby improving the real-time and system robustness of sensor monitoring.

[0054] It should be noted that Figure 1The scenario diagram shown is only an example. The multi-sensor anomaly detection method and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the system and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0055] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0056] Figure 2 The flowchart of the multi-sensor anomaly detection method provided in the embodiment of the present application is shown in FIG. Figure 2 As shown, the method includes:

[0057] 201. Receive topic information sent by multiple subscribed target sensors through a robot operating system ROS node; the topic information includes sensor data corresponding to the target sensors.

[0058] The execution subject of this embodiment is a robot, or a ROS node in the robot.

[0059] In this embodiment, the target sensor includes an IMU sensor, an image sensor or a point cloud sensor.

[0060] In this embodiment, a robot operating system ROS node can be created, a unique identifier can be assigned to the ROS node, the ROS node can be configured as a subscriber of multiple target sensors based on the unique identifier, a timer can be created, and the preset period can be set by the timer. Then, the topic information published by multiple target sensors can be received through the ROS node.

[0061] For example, Figure 3As shown in the figure, during the node initialization process, a ROS node named multi_sensor_monitor can be created and assigned a unique name to ensure that it can be uniquely identified in the ROS network. Initialize the timestamp of the ROS node. Configure subscribers for the IMU, image, and point cloud sensors respectively, and specify the corresponding topic names and message types: IMU topic: / imu_topic, message type is sensor_msgs / Imu. Image topic: / camera / image, message type is sensor_msgs / Image. Point cloud topic: / lidar / points, message type is sensor_msgs / PointCloud2. The subscriber will be responsible for receiving data from each sensor and passing it to the subsequent processing module. In order to achieve periodic detection, a timer with a preset period (for example, 0.5 seconds) can be created. The timer will periodically trigger the sensor status check function to achieve real-time monitoring of the sensor status. Among them, the period of the timer can be adjusted according to actual needs to ensure timely inspection of the sensor status without affecting the system performance. Optionally, in order to enhance the scalability and maintainability of the system, initialization tasks such as sensor subscription configuration and timer creation can be encapsulated into independent functions or classes. The method provided in this embodiment can realize real-time monitoring and exception handling of a multi-sensor system through ROS nodes, providing a reliable data source for subsequent data processing and decision support.

[0062] 202. For each target sensor, in response to receiving sensor data sent by the target sensor, obtain a current system time, determine the current system time as a last message timestamp of the target sensor, and generate log information.

[0063] Specifically, you can define an independent message callback function for each target sensor, get the current system time when receiving data, and update the last message timestamp of the sensor. Print log information in the callback function to confirm the data reception status.

[0064] For example, Figure 4As shown in the figure, for each target sensor (IMU, camera, lidar), a separate message callback function is defined in the ROS node. These callback functions will be triggered when the topic message of the corresponding sensor is received. In the callback function, first get the current system time (usually using the timestamp function provided by ROS), then associate the timestamp with the received sensor data, and update the last message timestamp of the sensor. In the callback function, a data structure (such as a dictionary or class attribute) can be maintained to store the last message timestamp of each sensor. When new sensor data is received, update the timestamp of the corresponding sensor to the current system time. In addition, in the callback function, in addition to updating the timestamp, log information can also be printed to confirm the reception status of the data. The log information can include the received sensor data type, topic name, timestamp, and possible data summary (such as image size, point cloud point count, etc.).

[0065] In the specific implementation process, in ROS, the message callback function is usually implemented through the callback interface of the subscriber. Developers need to define a callback function for each sensor data type and register it to the corresponding subscriber. In order to ensure the accuracy of the timestamp, the timestamp function provided by ROS can be used to obtain the current system time. For example, the ROS time management API is used, such as rclcpp::Time::now(). When printing log information, the ROS log system (such as RCLCPP_INFO, RCLCPP_WARN and other macros) can be used to record log information of different levels. The method provided in this embodiment can receive the data of each sensor in real time through the ROS node, and update the corresponding timestamp when the data is received, thereby ensuring the timeliness and accuracy of the data. At the same time, by printing the log information, it is convenient to further analyze the log information to determine the abnormal sensor.

[0066] 203. Based on a preset period, parse the log information corresponding to the plurality of target sensors respectively.

[0067] 204. For each target sensor, if the difference between the current time and the last message timestamp of the latest record in the log information corresponding to the target sensor is greater than a preset threshold, the target sensor is marked as an abnormal sensor.

[0068] Specifically, the periodic timer trigger mechanism can be used to check the status of each sensor and determine whether there is data delay or loss to ensure the stability and reliability of the system. now and the latest timestamp T of each sensor sensor . Calculate the time difference △t, if △t>△t timeout(The preset threshold can be set to 1 second), the sensor is marked as timed out and the sensor is determined to be an abnormal sensor. The time difference calculation formula is:

[0069] △t=t now -T sensor (1)

[0070] Among them, △t is the time difference, t now is the current system time, T sensor It is the last message timestamp, that is, the latest timestamp of the sensor.

[0071] If the conditions are met:

[0072] △t>△t timeout (2)

[0073] Among them, △t is the time difference, △t timeout is the preset threshold.

[0074] The sensor is judged to have timed out.

[0075] In this embodiment, the current time refers to the triggering time of the current cycle, that is, the time when the time difference calculation is performed. The current time can be the current system time, or it can be the time obtained by correcting the current system time.

[0076] For example, Figure 5 As shown, firstly, the timer callback is triggered, and then the sensor status is checked to determine whether the sensor has timed out. Specifically, when the timer is triggered, the current system time is first obtained, and then the latest timestamp of each sensor (that is, the timestamp when the data was last received) is obtained from the previously maintained data structure to check the sensor status. For each sensor, the difference between the current time and the latest timestamp of the sensor is calculated. A timeout threshold is set (for example, 1 second). If the time difference exceeds the threshold, it is determined that the sensor timeout is abnormal, that is, there is data delay or loss. The timed-out sensor is marked as an abnormal sensor, and log information can be recorded or the corresponding exception handling mechanism can be triggered.

[0077] In the specific implementation process, the timer can be implemented by the rclcpp::TimerBase class of ROS or its subclasses. In the callback function of the timer, the above-mentioned sensor status check logic is implemented. The timeout threshold can be adjusted according to the needs of the actual application. For example, for applications that require high real-time performance, the timeout threshold can be set shorter; while for applications that do not require high real-time performance, it can be set longer. When a sensor is marked as an abnormal state, a series of exception handling mechanisms can be triggered, such as sending warning messages, restarting sensor nodes, recording error logs, etc. The specific implementation of these exception handling mechanisms depends on the specific needs and design of the system. The method provided in this embodiment can periodically check the status of each sensor through the ROS node, and promptly detect and handle data delays or losses, thereby ensuring the stability and reliability of the system.

[0078] In some embodiments, in order to reduce misjudgment, the preset period is twice the preset threshold.

[0079] From the above description, it can be seen that the multi-sensor anomaly detection method provided in this embodiment receives the sensor data of multiple sensors in real time through the ROS node, and quickly determines the abnormal sensors by periodically detecting the timestamp when the sensor data is received. It can promptly discover problems such as sensor data transmission delay and packet loss, thereby improving the real-time performance and system robustness of sensor monitoring.

[0080] In some embodiments, if the time difference between the latest recorded last message timestamp and the previous last message timestamp in the log information corresponding to the target sensor is greater than the latest receiving cycle, the target sensor is determined as an abnormal sensor; the latest receiving cycle is determined based on a preset number of last message timestamps before the previous last message timestamp recorded in the log information.

[0081] Specifically, considering that the data transmission cycles of different sensors may be different, and the cycle of the same sensor may be offset, it is possible to dynamically detect the cycle received from the sensor, and determine whether packet loss or delay occurs based on the latest cycle detected. Exemplarily, when receiving the 100th sensor data, the latest cycle can be calculated based on the timestamps of the reception moments of the 80th to 99th sensor data. For example, the average value of the intervals between these timestamps can be taken as the latest cycle. Furthermore, when the time difference between the timestamp at the time of receiving the 100th sensor data and the corresponding timestamp of the 99th is greater than the latest cycle, it indicates that a data reception delay has occurred, and the sensor can be marked as an abnormal sensor.

[0082] In some embodiments, if the difference between the current time and the latest recorded last message timestamp in the log information corresponding to the target sensor is greater than a preset threshold, the target sensor is marked as an abnormal sensor, which may include: if the difference between the current time and the latest recorded last message timestamp in the log information corresponding to the target sensor is greater than a preset threshold, the log information corresponding to the target sensor is input into a deep learning model for abnormality identification, an identification result is obtained, and the abnormal sensor is determined according to the identification result.

[0083] Specifically, in order to improve the accuracy of anomaly detection, the log information of sensors that are judged to be abnormal based on time difference can be further input into the deep learning model for further detection. Specifically, the sensors are initially screened by periodically detecting whether the difference between the current time and the last message timestamp is greater than a preset threshold. If the difference is greater than the preset threshold, it indicates that the sensor is very likely to be abnormal. Then, the log information of the sensor is input into the deep learning model for further judgment. Since the deep learning model usually has a higher accuracy rate for samples with abnormalities, it can also reduce the misjudgment of the deep learning model.

[0084] In some embodiments, the method may further include: in response to detecting an abnormal sensor, executing at least one of the following: first item: generating a warning log, and displaying the warning log on the display screen of the console so that the user can make a preliminary diagnosis of the abnormality based on the warning log; the warning log includes the identification of the abnormal sensor; second item: publishing the topic of the error information so that other ROS nodes other than the ROS node receive the topic of the error information and perform corresponding response processing based on the error information; third item: capturing information of the abnormal sensor and obtaining abnormal information of the abnormal sensor so that the user can locate and repair the abnormality based on the abnormal information; the abnormal information includes stack information; fourth item: switching the abnormal sensor to a corresponding backup sensor, and determining the backup sensor as a new target sensor, and receiving the topic information sent by the new target sensor; fifth item: continuing to perform abnormality detection on the remaining target sensors except the abnormal sensor, and not issuing a control signal to terminate the system operation.

[0085] Specifically, for the first, second, and third items, when a sensor timeout is detected, exception handling is performed to ensure the robustness and debuggability of the system. Exception handling may include printing a warning log when a timeout occurs, and the warning log contains the sensor name and timeout description information. It may also include publishing ROS error information to the / sensor_error topic, and the message format may be a string: Example: "Sensor timeout detected: imu". It may also include capturing and recording exception information, and the exception information may include stack information for subsequent debugging. Among them, the warning log may be written to the console or file. Developers can check the logs regularly, or analyze the log information through log aggregation tools (such as rosbag, rqt_console, logfile) to troubleshoot abnormal conditions in system operation. After the ROS error information is published to the / sensor_error topic, other modules (such as health monitoring modules, alarm systems) can respond to the error message, such as sending an email, triggering a device restart, or reminding the operator through the UI interface. This helps to respond quickly when the system is running. The stack information captured by the exception can be viewed through debugging tools (such as GDB, rclcpp's debugging mode), helping developers to deeply analyze the cause of the system failure. In a production environment, this information can also be written to the log file, which facilitates in-depth analysis of the problem without affecting the operation of the system.

[0086] For example, Figure 6As shown in the figure, after a sensor timeout is detected, a warning log can be recorded, an error message can be published, and an exception can be captured and recorded. Among them, when a sensor timeout is detected, a warning log containing the sensor name and timeout description information can be printed immediately. This helps developers quickly identify and locate problems. The ROS2 logging system (such as the RCLCPP_WARN macro) can be used to print warning logs to ensure that log information can be recorded and displayed according to the predetermined log level. When publishing ROS error information, a ROS topic named / sensor_error can be created to publish sensor error information. When a sensor timeout is detected, a string message containing the sensor name and timeout description can be constructed and published to the / sensor_error topic. Other ROS nodes can subscribe to the / sensor_error topic to receive and process error information when a sensor times out. In the process of capturing and recording exceptions, a try-catch block can be used in the exception handling code to capture possible exceptions (such as exceptions in the timer callback function). When an exception is captured, the exception information is recorded, including the exception type and stack information. This facilitates developers to perform detailed debugging and analysis later. The ROS2 log system or third-party log library can be used to record abnormal information to ensure that the information can be properly stored and retrieved. The method provided in this embodiment can promptly execute abnormal handling measures when a sensor timeout is detected, and provide strong support for system maintenance and troubleshooting through detailed log records and error information release. This will help improve the robustness and debuggability of the system and ensure the stable operation of the system.

[0087] In some embodiments, for the second item, considering that the continuity and accuracy of sensor data such as IMU, image, point cloud, etc. are crucial, when a sensor is abnormal, other ROS nodes can use data compensation algorithms such as curve fitting to fill the data gap to maintain the normal operation of the system. Specifically, after publishing the topic of error information, other ROS nodes other than the ROS node receive the topic of the error information, and the corresponding response processing according to the error information may include: determining the fault type according to the error information; the fault type may include hardware fault, communication fault or data fault; if the fault type belongs to the preset type, the sensor data of the abnormal sensor is compensated based on the data compensation algorithm. Among them, the data step algorithm may include table lookup method, linear interpolation method, cubic spline interpolation method, curve fitting method, etc. In the specific implementation process, valid data before the abnormal sensor fails can be collected as a basis for compensation or interpolation. The collected valid data is input into the selected compensation algorithm. According to the calculation results of the algorithm, a compensation value or interpolation point to fill the data gap is generated. The generated compensation value or interpolation point is verified to ensure that it meets the data requirements and accuracy standards of the system. If outliers or data points that do not meet the requirements are found, the algorithm should be reapplied or the parameters should be adjusted.

[0088] Regarding the fourth item, in order to ensure the normal operation of the robot, a backup sensor can be set for the sensor. When the main sensor is abnormal, it will automatically switch to the backup sensor and continue to detect the backup sensor; the main sensor will automatically recover after being repaired.

[0089] For the fifth item, whether it is the main sensor or the backup sensor, the system can continuously monitor its status to form a closed-loop process of abnormality detection, switching, and recovery. By centrally monitoring the sensor status, the system stability is improved, and it can be further expanded to an intelligent management system that predicts abnormalities. For example, Figure 7 As shown, the sensor status can continue to be detected. For non-abnormal sensors that have not timed out, record the active status log. Ensure that the system can still run stably after detecting sensor anomalies, and provide users with clear operation results. Specifically, for the results of each periodic check, output the current sensor status log ("active" or "timeout"). Ensure that the system does not terminate after capturing an anomaly, but continues to monitor the status of other sensors.

[0090] Figure 8 This is a schematic diagram of the structure of a multi-sensor anomaly detection device provided in an embodiment of the present application. Figure 8 As shown, the multi-sensor anomaly detection device 80 includes: a receiving module 801, a determining module 802 and a parsing module 803.

[0091] The receiving module 801 is used to receive topic information sent by a plurality of subscribed target sensors through a robot operating system ROS node; the topic information includes sensor data corresponding to the target sensor;

[0092] A determination module 802 is used for, for each target sensor, in response to receiving the sensor data sent by the target sensor, obtaining the current system time, determining the current system time as the last message timestamp of the target sensor, and generating log information;

[0093] The parsing module 803 is used to parse the log information corresponding to the multiple target sensors respectively based on a preset period; for each target sensor, if the difference between the current time and the last message timestamp of the latest record in the log information corresponding to the target sensor is greater than a preset threshold, the target sensor is marked as an abnormal sensor.

[0094] The multi-sensor anomaly detection device provided in the embodiment of the present application receives the sensor data of multiple sensors in real time through the ROS node, and quickly determines the abnormal sensors by periodically detecting the timestamp when the sensor data is received. It can promptly discover problems such as sensor data transmission delay and packet loss, thereby improving the real-time performance and system robustness of sensor monitoring.

[0095] In some embodiments, the target sensor includes an IMU sensor, an image sensor, or a point cloud sensor; and the preset period is twice the preset threshold.

[0096] In some embodiments, the parsing module 803 is specifically used to: if the time difference between the latest recorded last message timestamp and the previous last message timestamp in the log information corresponding to the target sensor is greater than the latest receiving cycle, the target sensor is determined as an abnormal sensor; the latest receiving cycle is determined based on a preset number of last message timestamps before the previous last message timestamp recorded in the log information.

[0097] In some embodiments, the parsing module 803 is specifically used to: if the difference between the current time and the last message timestamp of the latest record in the log information corresponding to the target sensor is greater than a preset threshold, the log information corresponding to the target sensor is input into the deep learning model for anomaly identification, and an identification result is obtained, and the abnormal sensor is determined according to the identification result.

[0098] In some embodiments, the receiving module 801 is further used to: create a robot operating system ROS node and assign a unique identifier to the ROS node; based on the unique identifier, configure the ROS node as a subscriber of multiple target sensors; create a timer and set the preset period through the timer.

[0099] In some embodiments, the analysis module 803 is further configured to: in response to detecting an abnormal sensor, perform at least one of the following:

[0100] Generate a warning log, and display the warning log on a display screen of a console so that a user can make a preliminary diagnosis of an abnormality based on the warning log; the warning log includes an identification of an abnormal sensor;

[0101] Publishing a topic of error information so that other ROS nodes except the ROS node receive the topic of error information and perform corresponding response processing according to the error information;

[0102] Capturing information of the abnormal sensor to obtain abnormal information of the abnormal sensor, so that a user can locate and repair the abnormality according to the abnormal information; the abnormal information includes stack information;

[0103] Switching the abnormal sensor to a corresponding standby sensor, determining the standby sensor as a new target sensor, and receiving topic information sent by the new target sensor;

[0104] Continue to perform abnormality detection on the remaining target sensors except the abnormal sensor, and do not issue a control signal for terminating the operation of the system.

[0105] The multi-sensor anomaly detection device provided in the embodiment of the present application can be used to execute the above-mentioned method embodiment. Its implementation principle and technical effects are similar, and this embodiment will not be repeated here.

[0106] Fig. 9 A schematic diagram of the hardware structure of a multi-sensor anomaly detection device provided in an embodiment of the present application. The device may be a humanoid robot or other device that requires multi-sensor fusion processing.

[0107] Device 90 may include one or more of the following components: a processing component 901 , a memory 902 , a power component 903 , a multimedia component 904 , an audio component 905 , an input / output (I / O) interface 906 , a sensor component 907 , and a communication component 908 .

[0108] The processing component 901 generally controls the overall operation of the device 90, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 901 may include one or more processors 909 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 901 may include one or more modules to facilitate the interaction between the processing component 901 and other components. For example, the processing component 901 may include a multimedia module to facilitate the interaction between the multimedia component 904 and the processing component 901.

[0109] The memory 902 is configured to store various types of data to support operations on the device 90. Examples of such data include instructions for any application or method operating on the device 90, contact data, phone book data, messages, pictures, videos, etc. The memory 902 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0110] The power supply component 903 provides power to the various components of the device 90. The power supply component 903 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 90.

[0111] The multimedia component 904 includes a screen that provides an output interface between the device 90 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 904 includes a front camera and / or a rear camera. When the device 90 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0112] The audio component 905 is configured to output and / or input audio signals. For example, the audio component 905 includes a microphone (MIC), and when the device 90 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 902 or sent via the communication component 908. In some embodiments, the audio component 905 also includes a speaker for outputting audio signals.

[0113] I / O interface 906 provides an interface between processing component 901 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0114] The sensor assembly 907 includes one or more sensors for providing various aspects of status assessment for the device 90. For example, the sensor assembly 907 can detect the open / closed state of the device 90, the relative positioning of components, such as the display and keypad of the device 90, and the sensor assembly 907 can also detect the position change of the device 90 or a component of the device 90, the presence or absence of user contact with the device 90, the orientation or acceleration / deceleration of the device 90, and the temperature change of the device 90. The sensor assembly 907 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 907 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 907 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0115] The communication component 908 is configured to facilitate wired or wireless communication between the device 90 and other devices. The device 90 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 908 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 908 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0116] In an exemplary embodiment, the device 90 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0117] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 902 including instructions, and the instructions can be executed by a processor 909 of the device 90 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0118] The computer-readable storage medium mentioned above may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium may be any available medium that can be accessed by a general or special-purpose computer.

[0119] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0120] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0121] An embodiment of the present application further provides a computer program product, including a computer program. When the computer program is executed by a processor, the multi-sensor anomaly detection method performed by the multi-sensor anomaly detection device as described above is implemented.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-sensor anomaly detection method, characterized in that: include: Receive topic information sent by multiple subscribed target sensors through the robot operating system ROS node; The topic information includes sensor data corresponding to the target sensor; For each target sensor, in response to receiving the sensor data sent by the target sensor, obtaining the current system time, determining the current system time as the last message timestamp of the target sensor, and generating log information; Based on a preset period, the log information corresponding to the plurality of target sensors is analyzed respectively; For each target sensor, if the difference between the current time and the last message timestamp of the latest record in the log information corresponding to the target sensor is greater than a preset threshold, the target sensor is marked as an abnormal sensor.

2. The method according to claim 1, characterized in that The target sensor includes an IMU sensor, an image sensor or a point cloud sensor; the preset period is twice the preset threshold.

3. The method according to claim 1, characterized in that The method further comprises: If the time difference between the latest recorded last message timestamp and the previous last message timestamp in the log information corresponding to the target sensor is greater than the latest receiving cycle, the target sensor is determined as an abnormal sensor; the latest receiving cycle is determined based on a preset number of last message timestamps before the previous last message timestamp recorded in the log information.

4. The method according to claim 1, characterized in that If the difference between the current time and the last message timestamp of the latest record in the log information corresponding to the target sensor is greater than a preset threshold, the target sensor is marked as an abnormal sensor, including: If the difference between the current time and the last message timestamp of the latest record in the log information corresponding to the target sensor is greater than a preset threshold, the log information corresponding to the target sensor is input into the deep learning model for abnormality identification, and an identification result is obtained, and the abnormal sensor is determined according to the identification result.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Creating a ROS node and assigning a unique identifier to the ROS node; Based on the unique identifier, configuring the ROS node as a subscriber of multiple target sensors; A timer is created, and the preset period is set by the timer.

6. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: In response to detecting an abnormal sensor, performing at least one of the following: Generate a warning log, and display the warning log on a display screen of a console so that a user can make a preliminary diagnosis of an abnormality based on the warning log; the warning log includes an identification of an abnormal sensor; Publishing a topic of error information so that other ROS nodes except the ROS node receive the topic of error information and perform corresponding response processing according to the error information; Capturing information of the abnormal sensor to obtain abnormal information of the abnormal sensor, so that a user can locate and repair the abnormality according to the abnormal information; the abnormal information includes stack information; Switching the abnormal sensor to a corresponding standby sensor, determining the standby sensor as a new target sensor, and receiving topic information sent by the new target sensor; Continue to perform abnormality detection on the remaining target sensors except the abnormal sensor.

7. A multi-sensor anomaly detection device, characterized in that: include: A receiving module, used for receiving topic information sent by multiple subscribed target sensors through a robot operating system ROS node; The topic information includes sensor data corresponding to the target sensor; a determination module, configured to obtain, for each target sensor, a current system time in response to receiving sensor data sent by the target sensor, determine the current system time as the last message timestamp of the target sensor, and generate log information; The parsing module is used to parse the log information corresponding to the plurality of target sensors respectively based on a preset period; for each target sensor, if the difference between the current time and the last message timestamp of the latest record in the log information corresponding to the target sensor is greater than a preset threshold, the target sensor is marked as an abnormal sensor.

8. A multi-sensor anomaly detection device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the multi-sensor anomaly detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, the multi-sensor anomaly detection method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the multi-sensor anomaly detection method according to any one of claims 1 to 6 is implemented.

Images