Quadruped robot real-time abnormal event identification method based on visual perception

Abstract

The invention relates to the technical field of robots, in particular to a visual perception-based real-time abnormal event recognition method for a quadruped robot, which comprises the following steps of: acquiring body motion data and an environment image sequence of the robot in real time through an inertial measurement unit and a visual sensor; predicting and generating a short-term expected movement track based on the data and the control instruction; meanwhile, estimating an actual motion track of the robot by utilizing a visual odometer technology; fusing the deviation value of the expected trajectory and the actual trajectory with the optical flow statistical features extracted from the dense optical flow field to form a multi-modal fusion feature vector; and calculating a real-time abnormal confidence coefficient score by adopting a pre-trained long-short-term memory network model. Multi-mode sensing information is effectively fused, limitation of a single sensor is overcome, high-precision and real-time recognition and classification of abnormities such as foot slipping, collision and terrain abrupt change are achieved, corresponding emergency control strategies can be triggered, and autonomy and reliability of the quadruped robot in a complex environment are improved.

Description

Technical Field

[0001] This invention relates to the field of robotics, specifically to a method for real-time abnormal event recognition of quadruped robots based on visual perception. Background Technology

[0002] As quadruped robots gradually move from the laboratory to complex real-world applications (such as emergency rescue, field exploration, and industrial inspection), the environmental uncertainties they face increase dramatically. Ensuring high reliability and autonomy for robots in such dynamic and unstructured environments hinges on their ability to perceive and respond to sudden anomalies in real time and accurately. These anomalies, including but not limited to foot slippage, sudden collisions, joint malfunctions, and abrupt terrain changes, are major causes of robot task interruptions, instability, and even damage.

[0003] Currently, the technical approaches for anomaly detection in robots have several limitations, making it difficult to meet the stringent requirements of quadruped robots for real-time performance, accuracy, and environmental adaptability. These limitations are manifested in the following aspects:

[0004] 1. Detection methods based on a single sensor mode lack robustness:

[0005] Pure Inertial Measurement Unit (IMU) Dependence: Many existing methods rely solely on data from the robot's onboard IMU for anomaly detection, such as determining collisions by detecting thresholds for acceleration or angular velocity. However, IMU data only reflects changes in the robot's own motion state and cannot distinguish whether these changes originate from internal malfunctions (such as joint jamming) or external interactions (such as misstepping or collisions). More importantly, for some progressive anomalies (such as slowly sinking into soft mud), the IMU signal changes are not obvious, easily leading to missed detections.

[0006] Pure visual odometry-dependent methods rely entirely on the deviation between the trajectory calculated by visual SLAM or odometry and the expected trajectory for judgment. However, visual sensors are highly susceptible to interference from factors such as changes in lighting, missing textures, and motion blur. In dim lighting or when turning around quickly, the visual trajectory estimation itself may experience severe jitter or be lost, leading to a large number of false alarms.

[0007] 2. Lack of forward-looking forecasting and short-term trajectory planning as a benchmark for comparison:

[0008] Most existing anomaly detection systems perform "post-hoc" comparisons, that is, comparing the current actual state with the current expected state. However, robot motion has inertia, and the impact of anomalies often has a time delay. Current technology lacks a predictive trajectory for a short-term future time window (e.g., a few hundred milliseconds) based on current control commands and kinematic models. Without this predicted trajectory as a fine-grained comparison benchmark, the system struggles to detect minute but trending deviations in the early stages of anomaly occurrence, resulting in insufficient detection sensitivity.

[0009] 3. Superficial use of visual information, lacking in-depth extraction of motion features:

[0010] Even with the introduction of visual sensors in some studies, the utilization of visual data has largely remained at the level of odometry calculations or simple scene recognition. The rich motion information contained within image sequences has not been fully explored. For example, dense optical flow fields contain the motion vectors of every pixel in the environment, providing an excellent source of information for perceiving relative motion between oneself and the environment. Current technologies lack effective means to extract statistical features (such as the distribution of motion amplitude and the consistency of direction) from optical flow fields, failing to transform this information "gold mine" into quantifiable feature indicators for assessing anomalies.

[0011] 4. Static threshold determination mechanisms are difficult to adapt to dynamically changing environments:

[0012] Traditional anomaly detection often uses fixed thresholds, such as a fixed acceleration threshold or trajectory deviation threshold. However, the normal fluctuation range of robot vibration and trajectory deviation varies greatly under different motion states (e.g., walking versus running) and different terrains (e.g., flat ground versus climbing). Fixed thresholds are prone to false alarms during vigorous motion and may be insensitive during stable motion. The lack of a dynamic threshold mechanism that can adaptively adjust based on the robot's recent operating state is a major reason for the excessively high false alarm rate of existing systems.

[0013] 5. Detection and response are disconnected, failing to form a closed-loop intelligence:

[0014] Many technical solutions treat anomaly detection as a standalone sensing module, with the detection result typically being only a simple "yes / no" binary signal or a confidence score. The type of anomaly, its severity, and the specific recovery or adaptation strategies to be implemented are left to upper-level systems or operators. This open-loop design delays response time and, in scenarios requiring millisecond-level reactions, fails to achieve true autonomous intelligence.

[0015] In summary, existing technologies have significant shortcomings in multimodal perception fusion, forward-looking prediction, utilization of deep visual features, adaptive decision-making, and closed-loop response, which restrict the reliable operation of quadruped robots in complex real-world environments. Therefore, there is an urgent need for a novel real-time anomaly event recognition solution that can overcome these limitations. Summary of the Invention

[0016] The purpose of this invention is to provide a real-time abnormal event recognition method for quadruped robots based on visual perception, so as to solve the problems mentioned in the background art, which are that existing quadruped robots have obvious defects in multimodal perception fusion, forward prediction, utilization of deep visual features, adaptive decision-making and closed-loop response.

[0017] To achieve the above objectives, the present invention provides the following technical solution:

[0018] A vision-based method for real-time abnormal event recognition in quadruped robots includes the following steps:

[0019] S1. Real-time acquisition of multimodal data: The robot's motion state data is acquired in real time through the inertial measurement unit (IMU) and vision sensors mounted on the quadruped robot's body. Visual image sequences of the external environment ;in, Let represent the triaxial acceleration, triaxial angular velocity, and triaxial Euler angles at time t, respectively. This represents the sequence of images from the start time to time t.

[0020] S2. Robot-World Coordinate System Transformation and Desired Trajectory Generation: Based on the IMU data acquired in S1, the robot's real-time pose in the world coordinate system is calculated using quaternion differential equations. and combined with the robot's control commands This includes the desired velocity and steering angle, and uses a kinematic model to predict and generate the robot's desired trajectory within a short-term time window Δt. ;

[0021] S3. Visual Perception and Actual Trajectory Estimation: The sequence of visual images acquired in S1 Real-time processing is performed, and dense optical flow fields are estimated using a deep learning-based optical flow network. Furthermore, by extracting and matching feature points, and combining camera models and the SfM (Structure for Motion Reconstruction) principle, the robot's actual motion trajectory relative to its surrounding environment is calculated. ;

[0022] S4. Multimodal Feature Fusion and Anomaly Index Calculation: The desired trajectory generated in S2... The actual trajectory estimated by S3 Perform spatiotemporal alignment and calculate trajectory deviation. Simultaneously, the dense optical flow field obtained from S3 Extracting optical flow statistical feature vectors ; trajectory deviation Optical flow statistical characteristics Fusion to form a multimodal fused feature vector The data is then input into a pre-trained anomaly assessment model to calculate real-time anomaly confidence scores. ∈[0,1];

[0023] S5. Dynamic Threshold Judgment and Abnormal Event Identification: A dynamic threshold algorithm based on a time window is adopted, based on the recent abnormal confidence score sequence { Based on the statistical characteristics of}, the anomaly detection threshold at the current moment is adaptively calculated. ;when > When an abnormal event occurs, an alarm signal containing the type, level, and spatiotemporal information of the abnormality is generated.

[0024] Preferably, in step S2, the specific algorithm for predicting and generating the desired motion trajectory using a kinematic model includes:

[0025] Based on the robot's current linear velocity control commands and angular velocity control commands The trajectory is extrapolated using a differential driving model, and its discrete form is as follows:

[0026] ;

[0027] in,[ , , ] T Let t be the robot pose in the world coordinate system calculated by the IMU. , , ] T To predict the future pose point, by connecting Multiple predicted pose points within the time window form the desired trajectory. .

[0028] Preferably, in step S3, the specific method for estimating the actual trajectory is as follows: using a lightweight convolutional neural network (CNN) to extract images in real time. Feature points and their descriptors, compared with the previous frame image Perform feature matching to obtain a set of matching point pairs. After filtering out mismatched point pairs using the Random Sample Consensus (RANSAC) algorithm, the relative pose transformation between two frames is solved by Essential Matrix decomposition. By accumulating the relative pose transformations, the actual trajectory of the robot in the world coordinate system with the initial time as the origin is obtained: .

[0029] Preferably, in step S4, the optical flow statistical feature vector The extraction includes the following steps: for dense optical flow fields Perform grid partitioning, dividing the image into M×N grids; for each grid... Calculate the mean value of optical flow amplitude within the grid. and standard deviation Calculate the histogram of optical flow directions within the grid, and take the difference between the two directions with the largest amplitudes in the histogram as a measure of directional consistency. ; Statistical characteristics of all grids ( , , The features are concatenated to form a high-dimensional feature vector, and then dimensionality is reduced by principal component analysis (PCA) to obtain the final optical flow statistical feature vector. .

[0030] Preferably, the anomaly evaluation model is a lightweight Long Short-Term Memory (LSTM) network, whose input is a multimodal fusion feature vector. The output is the anomaly confidence score. The hidden state update formula for this LSTM network is:

[0031] ;

[0032] ;

[0033] ;

[0034] ∘ + ∘ ;

[0035] ;

[0036] ∘ ;

[0037] in, The sigmoid activation function is used, and ∘ represents the Hadamard product; finally, the last hidden state is processed through a fully connected layer and the sigmoid activation function. Mapped to :

[0038] .

[0039] Preferably, in step S5, the dynamic threshold algorithm specifically involves: setting a time window of length L and storing the anomaly confidence scores for the last L time points. Calculate the mean of the scores within this window. and standard deviation The dynamic threshold at the current moment; The calculation is as follows:

[0040] ;

[0041] in, This is the sensitivity coefficient, used to adjust how sensitive the threshold is to fluctuations in the fraction. It is a base threshold bias used to prevent false alarms when the score is extremely stable.

[0042] Preferably, after determining that an abnormal event has occurred, step S6 is also included: anomaly type classification and response strategy generation; based on the multimodal fusion features within a period of time before and after the anomaly occurred. Harmony flow field The system uses a pre-trained classifier to identify anomaly types, which include at least: foot slippage, sudden joint failure, collision with unknown obstacles, and terrain instability. Based on the identified anomaly types, the system generates corresponding robot motion response strategies, which include: emergency braking, gait switching, cadence adjustment, foot force control replanning, or sending a distress signal to a remote operator.

[0043] Preferably, an offline training phase is included before step S1:

[0044] Multimodal data of quadruped robots and corresponding abnormal event labels are collected under various typical environments and abnormal scenarios to construct a training dataset. The optical flow network and feature extraction network in step S3 and the abnormal evaluation model in step S4 are then subjected to supervised end-to-end or phased joint training until the model converges.

[0045] As a preferred option, a real-time abnormal event recognition system for quadruped robots based on vision perception includes:

[0046] The data acquisition module, including an IMU and a vision sensor, is used to acquire real-time data on the robot's motion state and visual images of the environment.

[0047] The data processing and calculation module, embedded in the robot's onboard computing unit, is configured to perform: expected trajectory generation, actual trajectory estimation, multimodal feature fusion, anomaly index calculation, and dynamic threshold judgment.

[0048] The control and response module is configured to: when an abnormal event is detected, invoke a preset emergency control strategy according to the type of abnormality, generate control commands and send them to the robot's underlying motion controller.

[0049] As a preferred option, a quadruped robot equipped with a vision-based real-time abnormal event recognition system for quadruped robots is included.

[0050] Compared with the prior art, the beneficial effects of the present invention are:

[0051] 1. Multimodal information fusion provides comprehensive and robust perception.

[0052] The core advantage of this method lies in its deep integration of information from both ontology perception and environment perception modalities, rather than relying on a single data source.

[0053] Complementarity: IMUs provide high-frequency, accurate data on the robot's own motion, but they are insensitive to environmental changes and suffer from cumulative errors. Visual sensors, on the other hand, can directly perceive the robot's motion relative to its external environment and dynamic changes in the scene, effectively compensating for the drift problem of IMUs. The combination of the two achieves comprehensive perception, both internal and external.

[0054] Redundancy: When one sensor fails temporarily or is interfered with, another sensor can still provide some information, ensuring the system's robustness and continuous operation in complex environments.

[0055] Cross-validation: The comparison between the expected trajectory and the actual trajectory provides a very reliable and physically meaningful anomaly criterion, which greatly reduces the false alarm rate.

[0056] 2. Balancing real-time performance and computational efficiency

[0057] The method is designed to fully consider the need for real-time operation on the airborne computing unit.

[0058] Lightweight model: Explicitly adopt a lightweight network structure in key steps to ensure that the algorithm can run at high speed on limited hardware resources.

[0059] Feature engineering: Gridded statistical feature extraction of the optical flow field and dimensionality reduction through PCA is an effective strategy that significantly reduces the computational load of subsequent models while preserving key information, and is much more efficient than directly processing the raw optical flow data.

[0060] Dynamic threshold: The adaptive dynamic threshold algorithm is simple to calculate and can achieve adaptive adjustment of the threshold without complex calculations, making it very suitable for real-time systems.

[0061] 3. High-precision anomaly detection with low false alarm rate

[0062] Through a series of technical means, we have achieved both sensitive and reliable anomaly detection.

[0063] Deep temporal modeling: LSTM networks are used as an anomaly evaluation model, enabling them to learn and remember the dependencies between events in a time series. This means that it can not only identify instantaneous anomalies, but also recognize gradually developing, subtle anomaly trends, significantly improving detection accuracy.

[0064] Contextual Awareness: Optical Flow Statistical Features V flow The introduction of (t) enables the system to not only perceive that "it is not moving as expected," but also that "the environment looks abnormal." This provides crucial information for distinguishing different types of anomalies.

[0065] Adaptive threshold: The dynamic threshold τ(t) overcomes the shortcomings of the fixed threshold. During smooth operation, the threshold is low, which can capture minor anomalies; in normal bumpy road sections, the confidence score itself will fluctuate, and the dynamic threshold will also increase accordingly to prevent false alarms, thus achieving adaptation to the environment.

[0066] 4. Highly scalable functionality: from detection to diagnosis and response.

[0067] This method goes beyond simply "detecting anomalies" and extends to "what happened" and "what to do," forming a complete closed-loop system.

[0068] Classifiable anomalies: This method can subsequently identify the anomaly type. This is because the fused feature vector F fusion (t) and the original optical flow field F t It contains a wealth of information that can be used to distinguish failure modes.

[0069] Support for decision-making and response: Based on accurate anomaly type identification, the system can trigger refined and differentiated response strategies, rather than a simple "emergency stop." For example, slippage triggers force control adjustments, collisions trigger an emergency stop and alarm, and terrain instability triggers a gait change. This greatly enhances the robot's autonomy and practicality.

[0070] Clear system architecture: From data acquisition and computation to control response, the modular design is clear and easy to implement and deploy.

[0071] The core advantages of the visual perception-based real-time anomaly event recognition method for quadruped robots in this invention are: multimodal fusion solves the problem of the limitations of single-sensor perception; lightweight design and efficient feature processing ensure real-time performance; time-series modeling and dynamic thresholding achieve high-precision, low-false-alarm detection; and finally, anomaly classification and response strategy generation complete the closed loop from perception to decision-making, significantly improving the autonomous survivability and task reliability of quadruped robots in complex and unknown environments. Attached Figure Description

[0072] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are explained in detail together with the embodiments of the invention, but do not constitute a limitation thereof.

[0073] Figure 1 This is a flowchart of the real-time abnormal event recognition method for quadruped robots based on visual perception according to the present invention.

[0074] Figure 2 This is a block diagram of the real-time abnormal event recognition system for quadruped robots based on visual perception, as described in this invention.

[0075] Figure 3 This is a line graph showing the changes in trajectory deviation and anomaly score in the foot slippage scenario of Embodiment 1 of the present invention;

[0076] Figure 4 This is a line graph showing the changes in IMU signals and anomaly scores under a collision scenario in Embodiment 2 of the present invention.

[0077] Figure 5 This is a line graph showing the cumulative deviation and anomaly score trend under terrain abrupt changes in Embodiment 3 of the present invention. Detailed Implementation

[0078] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0079] Example 1: Feet slipping on wet and slippery ground

[0080] Scene description:

[0081] A quadruped robot is walking normally on a smooth indoor tile floor. Ahead is a patch of undetected water. Following instructions, the robot moves forward in a straight line at a speed of 0.5 m / s. When its right foreleg steps onto the water patch, it briefly slips. The trajectory deviation and anomaly score changes during this slippage scenario are as follows: Figure 3 As shown, the slippage event occurs at t=0.3s. The trajectory deviation (ΔTraj) exhibits a sharp peak (0.005m), while the anomaly confidence (S) also increases. anomaly The readings also surged dramatically, far exceeding the previous baseline level, accurately capturing this anomaly.

[0082] Detailed process of the implementation example:

[0083] Data Acquisition (S1):

[0084] IMU Data: The robot's IMU continuously outputs acceleration, angular velocity, and Euler angle data. At the moment of slippage, the IMU detects a slight, non-commanded change in lateral acceleration and yaw rate.

[0085] Visual data: The airborne camera continuously captures image sequences of the surrounding environment. Ground texture features can be seen in the images.

[0086] Expected vs. Actual Trajectory (S2 & S3):

[0087] Expected trajectory (Traj) exp The control system issues a command to move in a straight line. Therefore, the kinematic model predicts a smooth, forward-moving short-term trajectory.

[0088] Actual trajectory (Traj) actual Visual odometry (SfM) calculates the robot's actual trajectory by processing image sequences. At the moment of slippage, due to the sudden decrease in foot friction, the robot's body undergoes a small, unexpected lateral displacement and rotation, causing a deviation (ΔTraj) between the actual trajectory and the expected trajectory.

[0089] Optical flow characteristics (S4):

[0090] The optical flow network calculates the dense optical flow field (F) of the current frame. t At the moment of slippage, the optical flow field will exhibit a local motion pattern that is inconsistent with the robot's overall forward direction (for example, ground texture features produce abnormal lateral flow in the image).

[0091] Anomaly detection (S4 & S5):

[0092] Multimodal fusion feature F fusion (t) is input into the LSTM anomaly assessment model. The model integrates the two pieces of information, "the trajectory deviation is increasing" and "anomalies are appearing in the optical flow pattern," and outputs an anomaly confidence score S. anomaly (t) begins to rise rapidly.

[0093] The dynamic threshold τ(t) is calculated based on recent stationary fractions (e.g., within the previous second), with a low initial value. When S anomaly When (t) rapidly exceeds τ(t) at the moment of slippage, the system immediately determines that an abnormal event has occurred.

[0094] Response (S7):

[0095] The classifier determines the anomaly type as foot slipping based on the features.

[0096] The system generates a response strategy: immediately fine-tunes the foot force control of the slipping leg to increase grip; and briefly adjusts the body's center of gravity to compensate for instability, rather than braking suddenly, thereby ensuring the continuity of movement.

[0097] Data table:

[0098]

[0099] Example 2: Collision with a low, unknown obstacle

[0100] Scene description:

[0101] A robot is performing an inspection task in a warehouse environment. A small cardboard box is suddenly placed in its path. Because the box's color is similar to the ground and its height is low, the visual SLAM system fails to recognize it as an obstacle in advance. The robot's left front leg directly collidees with the box. The changes in the IMU signal and anomaly score during the collision scenario are as follows: Figure 4 As shown, at the impact time t=0.3s, the IMU acceleration record shows an extremely high pulse peak (15.8 m / s²), indicating a severe impact. Anomaly confidence level (S) anomaly The response was extremely rapid, instantly reaching near-perfect (0.98) and triggering the highest level of anomaly alert. Both then slowly declined, but the anomaly confidence score, due to its time-series memory characteristics, declined more slowly than the IMU signal.

[0102] Detailed process of the implementation example:

[0103] Data Acquisition (S1):

[0104] IMU data: At the moment of impact, the IMU will detect a violent, high-frequency deceleration (negative acceleration) impact signal, as well as possible sudden changes in roll angle.

[0105] Visual data: In a sequence of images, a scene suddenly exhibits large, discontinuous motion.

[0106] Expected vs. Actual Trajectory (S2 & S3):

[0107] Expected trajectory: A smooth trajectory predicted based on forward instructions.

[0108] Actual trajectory: The collision momentarily impedes the robot's movement. Visual odometry calculates a sharp jolt or brief pause in the robot's posture, resulting in a significant deviation (ΔTraj) from the expected trajectory.

[0109] Optical flow characteristics (S4):

[0110] When a collision occurs, the optical flow field of the entire image becomes extremely chaotic. Due to the violent camera shake, the optical flow vectors become short and disordered, with highly inconsistent directions. Optical flow statistical characteristics V flow The amplitude standard deviation and direction consistency measures in (t) will be significantly abnormal.

[0111] Anomaly detection (S4 & S5):

[0112] The anomaly assessment model receives huge trajectory deviations and extremely anomalous optical flow characteristics, and outputs S anomaly The (t) score instantly jumped to nearly 1.0, far exceeding the dynamic threshold, immediately triggering an anomaly alarm.

[0113] Response (S7):

[0114] The classifier judges the collision as "collision with an unknown obstacle" based on the characteristics of severe impact and chaotic optical flow.

[0115] The system generates a response strategy: immediately triggering emergency braking, stopping all gait generation, and putting all joints into damping mode to absorb impact energy. Simultaneously, it sends an alarm signal and a snapshot of the moment of impact to the remote operator.

[0116] Data table:

[0117]

[0118] Example 3: Abrupt topography leading to instability (from hard ground to soft sand)

[0119] Scene description:

[0120] The robot transitioned from a solid, hard surface to soft sand outdoors. This transition was gradual, rather than a sudden impact; the cumulative deviation and anomaly score trend under the abrupt terrain change were as follows: Figure 5 As shown in the chart, this graph illustrates a gradual anomaly, distinct from the previously described sudden anomalies. Starting from t=0s upon entering the sandy area, the cumulative trajectory deviation steadily increases over time (from 0.01m to 0.07m). Anomaly confidence level (S) anomaly The error did not jump instantaneously, but rather steadily increased following this deviation trend, exceeding the dynamic threshold at t=3s and triggering an anomaly. Subsequently, due to the system's adjustment strategy (gait switching), the rate of deviation accumulation slowed down, the anomaly score also began to decrease slowly, and the system entered a new "normal" stable state.

[0121] Detailed process of the implementation example:

[0122] Data Acquisition (S1):

[0123] IMU data: Upon entering the sand, due to the feet sinking and unstable support, the robot's body began to exhibit continuous, low-frequency, small-amplitude swaying. The IMU continuously measured greater roll and pitch fluctuations than on hard ground.

[0124] Visual data: Images show the ground texture changing from asphalt to gravel.

[0125] Expected vs. Actual Trajectory (S2 & S3):

[0126] Expected trajectory: The control system still issues commands and predicts the trajectory according to the hard ground model.

[0127] Actual trajectory: Due to increased movement resistance and foot slippage, the robot's actual forward speed was lower than expected, causing the displacement deviation ΔTraj to gradually accumulate and increase over time (although the deviation at each moment was not large).

[0128] Optical flow characteristics (S4):

[0129] The optical flow field shows that the movement speed of the ground texture slows down (because the actual speed slows down), and due to the shaking of the aircraft, the optical flow field exhibits a slight, overall undulating pattern.

[0130] Anomaly detection (S4 & S5):

[0131] The advantages of the LSTM anomaly assessment model are thus demonstrated. It can memorize time-series features and detect trends such as "trajectory deviation is slowly and continuously increasing" and "continuous low-frequency fluctuations in IMU and optical flow." Therefore, S... anomaly (t) The score will show a slow but steady upward trend.

[0132] After a few seconds, S anomaly The cumulative value of (t) eventually exceeds the dynamic threshold τ(t) calculated based on recent history, and the system determines that it has entered an abnormal state.

[0133] Response (S7):

[0134] The classifier identifies the anomaly type as "topographical instability".

[0135] The system generates a response strategy: switching gait from a fast gait suitable for hard surfaces (such as jogging) to a slow, high-frequency gait suitable for soft surfaces (such as crawling) to increase stability. It may also adjust the foot contact control strategy.

[0136] Data table:

[0137]

[0138] The advantages of this vision-based real-time abnormal event recognition method for quadruped robots are as follows:

[0139] 1. Multimodal information fusion provides comprehensive and robust perception.

[0140] The core advantage of this method lies in its deep integration of information from both ontology perception and environment perception modalities, rather than relying on a single data source.

[0141] Complementarity: IMUs provide high-frequency, accurate data on the robot's own motion, but they are insensitive to environmental changes and suffer from cumulative errors. Visual sensors, on the other hand, can directly perceive the robot's motion relative to its external environment and dynamic changes in the scene, effectively compensating for the drift problem of IMUs. The combination of the two achieves comprehensive perception, both internally and externally.

[0142] Redundancy: When one sensor fails temporarily or is interfered with, another sensor can still provide some information, ensuring the system's robustness and continuous operation in complex environments.

[0143] Cross-validation: The comparison between the expected trajectory and the actual trajectory provides a very reliable and physically meaningful anomaly criterion, which greatly reduces the false alarm rate.

[0144] 2. Balancing real-time performance and computational efficiency

[0145] The method is designed to fully consider the need for real-time operation on the airborne computing unit.

[0146] Lightweight model: Explicitly adopt a lightweight network structure in key steps to ensure that the algorithm can run at high speed on limited hardware resources.

[0147] Feature engineering: Gridded statistical feature extraction of the optical flow field and dimensionality reduction through PCA is an effective strategy that significantly reduces the computational load of subsequent models while preserving key information, and is much more efficient than directly processing the raw optical flow data.

[0148] Dynamic threshold: The adaptive dynamic threshold algorithm is simple to calculate and can achieve adaptive adjustment of the threshold without complex calculations, making it very suitable for real-time systems.

[0149] 3. High-precision anomaly detection with low false alarm rate

[0150] Through a series of technical means, we have achieved both sensitive and reliable anomaly detection.

[0151] Deep temporal modeling: LSTM networks are used as an anomaly evaluation model, enabling them to learn and remember the dependencies between events in a time series. This means that it can not only identify instantaneous anomalies, but also recognize gradually developing, subtle anomaly trends, significantly improving detection accuracy.

[0152] Contextual Awareness: Optical Flow Statistical Features V flow The introduction of (t) enables the system to not only perceive that "it is not moving as expected," but also that "the environment looks abnormal." This provides crucial information for distinguishing different types of anomalies.

[0153] Adaptive threshold: The dynamic threshold τ(t) overcomes the shortcomings of the fixed threshold. During smooth operation, the threshold is low, which can capture minor anomalies; in normal bumpy road sections, the confidence score itself will fluctuate, and the dynamic threshold will also increase accordingly to prevent false alarms, thus achieving adaptation to the environment.

[0154] 4. Highly scalable functionality: from detection to diagnosis and response.

[0155] This method goes beyond simply "detecting anomalies" and extends to "what happened" and "what to do," forming a complete closed-loop system.

[0156] Classifiable anomalies: This method can subsequently identify the anomaly type. This is because the fused feature vector F fusion (t) and the original optical flow field F t It contains a wealth of information that can be used to distinguish failure modes.

[0157] Support for decision-making and response: Based on accurate anomaly type identification, the system can trigger refined and differentiated response strategies, rather than a simple "emergency stop." For example, slippage triggers force control adjustments, collisions trigger an emergency stop and alarm, and terrain instability triggers a gait change. This greatly enhances the robot's autonomy and practicality.

[0158] Clear system architecture: From data acquisition and computation to control response, the modular design is clear and easy to implement and deploy.

[0159] The core advantages of the visual perception-based real-time anomaly event recognition method for quadruped robots in this invention are: multimodal fusion solves the problem of the limitations of single-sensor perception; lightweight design and efficient feature processing ensure real-time performance; time-series modeling and dynamic thresholding achieve high-precision, low-false-alarm detection; and finally, anomaly classification and response strategy generation complete the closed loop from perception to decision-making, significantly improving the autonomous survivability and task reliability of quadruped robots in complex and unknown environments.

[0160] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for real-time abnormal event recognition of quadruped robots based on visual perception, characterized in that, Includes the following steps: S1. Real-time acquisition of multimodal data: The robot's motion state data is acquired in real time through the inertial measurement unit (IMU) and vision sensors mounted on the quadruped robot's body. Visual image sequences of the external environment ;in, Let represent the triaxial acceleration, triaxial angular velocity, and triaxial Euler angles at time t, respectively. This represents the sequence of images from the start time to time t. S2. Robot-World Coordinate System Transformation and Desired Trajectory Generation: Based on the IMU data acquired in S1, the robot's real-time pose in the world coordinate system is calculated using quaternion differential equations. and combined with the robot's control commands This includes the desired velocity and steering angle, and uses a kinematic model to predict and generate the robot's desired trajectory within a short-term time window Δt. ; S3. Visual Perception and Actual Trajectory Estimation: The sequence of visual images acquired in S1 Real-time processing is performed, and dense optical flow fields are estimated using a deep learning-based optical flow network. Furthermore, by extracting and matching feature points, and combining camera models and the SfM (Structure for Motion Reconstruction) principle, the robot's actual motion trajectory relative to its surrounding environment is calculated. ; S4. Multimodal Feature Fusion and Anomaly Index Calculation: The desired trajectory generated in S2... The actual trajectory estimated by S3 Perform spatiotemporal alignment and calculate trajectory deviation. Simultaneously, the dense optical flow field obtained from S3 Extracting optical flow statistical feature vectors ; trajectory deviation Optical flow statistical characteristics Fusion to form a multimodal fused feature vector The data is then input into a pre-trained anomaly assessment model to calculate real-time anomaly confidence scores. ∈[0,1]; S5. Dynamic Threshold Judgment and Abnormal Event Identification: A dynamic threshold algorithm based on a time window is adopted, based on the recent abnormal confidence score sequence { Based on the statistical characteristics of}, the anomaly detection threshold at the current moment is adaptively calculated. ;when > When an abnormal event occurs, an alarm signal containing the type, level, and spatiotemporal information of the abnormality is generated.

2. The method for real-time abnormal event recognition of a quadruped robot based on visual perception according to claim 1, characterized in that, In step S2, the specific algorithm for predicting and generating the desired motion trajectory using a kinematic model includes: Based on the robot's current linear velocity control commands and angular velocity control commands The trajectory is extrapolated using a differential driving model, and its discrete form is as follows: ； in,[ , , ] T Let t be the robot pose in the world coordinate system calculated by the IMU. , , ] T To predict the future pose point, by connecting Multiple predicted pose points within the time window form the desired trajectory. .

3. The method for real-time abnormal event recognition of a quadruped robot based on visual perception according to claim 1, characterized in that, In step S3, the specific method for estimating the actual trajectory is as follows: using a lightweight convolutional neural network (CNN) to extract images in real time. Feature points and their descriptors, compared with the previous frame image Perform feature matching to obtain a set of matching point pairs. After filtering out mismatched point pairs using the Random Sample Consensus (RANSAC) algorithm, the relative pose transformation between the two frames is solved by Essential Matrix decomposition. By accumulating the relative pose transformations, the actual trajectory of the robot in the world coordinate system with the initial time as the origin is obtained: .

4. The method for real-time abnormal event recognition of a quadruped robot based on visual perception according to claim 1, characterized in that, In step S4, the optical flow statistical feature vector The extraction includes the following steps: for dense optical flow fields Perform grid partitioning, dividing the image into M×N grids; for each grid... Calculate the mean value of optical flow amplitude within the grid. and standard deviation Calculate the histogram of optical flow directions within the grid, and take the difference between the two directions with the largest amplitudes in the histogram as a measure of directional consistency. ; Statistical characteristics of all grids ( , , The features are concatenated to form a high-dimensional feature vector, and then dimensionality is reduced by principal component analysis (PCA) to obtain the final optical flow statistical feature vector. .

5. The method for real-time abnormal event recognition of a quadruped robot based on visual perception according to claim 1, characterized in that, The anomaly assessment model is a lightweight Long Short-Term Memory (LSTM) network, whose input is a multimodal fusion feature vector. The output is the anomaly confidence score. The hidden state update formula for this LSTM network is: ； ； ； ∘ + ∘ ； ； ∘ ； in, The sigmoid activation function is used, and ∘ represents the Hadamard product; finally, the last hidden state is processed through a fully connected layer and the sigmoid activation function. Mapped to : 。 6. The method for real-time abnormal event recognition of a quadruped robot based on visual perception according to claim 1, characterized in that, In step S5, the dynamic threshold algorithm specifically involves: setting a time window of length L and storing the anomaly confidence scores for the last L time moments. Calculate the mean of the scores within this window. and standard deviation The dynamic threshold at the current moment; The calculation is as follows: ； in, This is the sensitivity coefficient, used to adjust how sensitive the threshold is to fluctuations in the fraction. It is a base threshold bias used to prevent false alarms when the score is extremely stable.

7. The method for real-time abnormal event recognition of a quadruped robot based on visual perception according to claim 1, characterized in that, After determining that an abnormal event has occurred, step S6 is also included: anomaly type classification and response strategy generation; based on the multimodal fusion features within a period of time before and after the anomaly occurred. Harmony flow field The system uses a pre-trained classifier to identify anomaly types, which include at least: foot slippage, sudden joint failure, collision with unknown obstacles, and terrain instability. Based on the identified anomaly types, the system generates corresponding robot motion response strategies, which include: emergency braking, gait switching, cadence adjustment, foot force control replanning, or sending a distress signal to a remote operator.

8. The method for real-time abnormal event recognition of a quadruped robot based on visual perception according to claim 1, characterized in that, Before step S1, there is also an offline training phase: In various typical environments and abnormal scenarios, multimodal data of quadruped robots and corresponding abnormal event labels are collected to construct a training dataset. The optical flow network and feature extraction network in step S3 and the abnormal evaluation model in step S4 are then subjected to supervised end-to-end or phased joint training until the model converges.

9. The method for real-time abnormal event recognition of a quadruped robot based on visual perception according to claim 1, characterized in that, A vision-based real-time abnormal event recognition system for quadruped robots includes: The data acquisition module, including an IMU and a vision sensor, is used to acquire real-time data on the robot's motion state and visual images of the environment. The data processing and calculation module, embedded in the robot's onboard computing unit, is configured to perform: expected trajectory generation, actual trajectory estimation, multimodal feature fusion, anomaly index calculation, and dynamic threshold judgment. The control and response module is configured to: when an abnormal event is detected, invoke a preset emergency control strategy according to the type of abnormality, generate control commands and send them to the robot's underlying motion controller.

10. The method for real-time abnormal event recognition of a quadruped robot based on visual perception according to claim 9, characterized in that, The invention includes a quadruped robot equipped with the vision-based real-time abnormal event recognition system for quadruped robots as described in claim 9.

Images