Unknown environment adaptive navigation control system based on deep learning
By employing technologies such as heterogeneous sensor array interfaces and multimodal perception optimization modules, the adaptability and error problems of traditional navigation methods in dynamic environments have been solved, enabling adaptive navigation and real-time perception in unknown environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- STATE GRID HUBEI ELECTRIC POWER INFORMATION & TELECOMMUNICATION COMPANY
- Filing Date
- 2025-12-17
- Publication Date
- 2026-05-05
AI Technical Summary
Traditional navigation methods are poorly adaptable to dynamic environments, have low efficiency in multimodal perception fusion, accumulate serious errors in long-term tasks, and existing research is limited by the high cost of real-world data acquisition, making it difficult to achieve real-time perception and adaptive navigation in unknown environments.
The system employs a heterogeneous sensor array interface, a time-series environment dynamic modeling module, a multimodal perception optimization module, an unknown environment risk prediction module, an adaptive navigation decision module, and an operational error self-correction module. Through real-time data acquisition, dynamic modeling, online sensor calibration, risk assessment, and navigation strategy adjustment, the system achieves adaptive navigation.
It improves the system's adaptability in dynamic scenarios, enhances the accuracy of multimodal perception fusion, dynamically corrects long-term errors, and ensures the stability and accuracy of navigation.
Smart Images
Figure CN121979192A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of autonomous navigation control technology, specifically to an adaptive navigation control system for unknown environments based on deep learning. Background Technology
[0002] As autonomous systems extend into scenarios with high complexity and strong uncertainty, traditional navigation methods face multiple technical bottlenecks. First, they suffer from significant defects in adaptability to dynamic environments—existing algorithms generally rely on static environment assumptions, leading to delayed system responses and the need for manual recalibration when faced with moving obstacles or sudden interference, resulting in a sharp drop in positioning accuracy. Second, multimodal perception fusion suffers from structural defects. The spatiotemporal asynchronous characteristics and measurement noise differences between LiDAR, millimeter-wave radar, and cameras make it difficult for traditional Early / Late Fusion strategies to effectively extract complementary features, especially under extreme conditions (such as visual occlusion), which can easily lead to systemic failures. Third, the problem of accumulated errors is prominent in long-term tasks. Deep learning modules such as visual odometry are affected by iterative drift, and coupled with unreliable closed-loop detection mechanisms, the path redundancy rate exceeds industry thresholds. Current research is limited by the high cost of real-world data acquisition, and most results are only at the simulation verification stage. There is an urgent need to build an intelligent navigation framework with real-time environment modeling, cross-modal feature co-evolution, and incremental topology learning to overcome core bottlenecks such as insufficient generalization ability in dynamic scenarios, low fault tolerance of heterogeneous sensors, and error divergence in long-cycle tasks. Summary of the Invention
[0003] The purpose of this invention is to overcome the problems existing in the prior art. This invention provides an adaptive navigation control system for unknown environments based on deep learning, which has the advantages of real-time dynamic understanding of the environment, multimodal perception and collaborative evolution, and long-term operation error self-correction. It solves the problems of poor adaptability of traditional methods in dynamic scenarios, inefficient fusion of heterogeneous sensors, and navigation inaccuracy caused by accumulated errors in long-term tasks.
[0004] To achieve the above objectives, the technical solution of the present invention is:
[0005] In a first aspect, the present invention provides an adaptive navigation control system for unknown environments based on deep learning, specifically including: a heterogeneous sensor array interface, a time-series environment dynamic modeling module, a multimodal perception optimization module, an unknown environment risk prediction module, an adaptive navigation decision module, an operation error self-correction module, and a dynamic path planning control module;
[0006] The heterogeneous sensor array interface integrates multiple types of sensor devices to collect spatial characteristics of the unknown environment, dynamic target information, and its own operating status data in real time, thereby acquiring multimodal data streams.
[0007] The time-series environment dynamic modeling module distinguishes the current environment structure and dynamic elements based on multimodal data streams, and constructs a real-time updated environment map;
[0008] The multimodal perception optimization module calibrates the contribution of each sensor online based on real-time updated environmental maps, multimodal data streams, and historical navigation data, and isolates or downgrades the operation of sensors whose contribution is lower than a set value.
[0009] The unknown environment risk prediction module predicts potential collision risks and sensor failure risks based on multimodal data streams and historical navigation data, and obtains a comprehensive risk assessment value.
[0010] The adaptive navigation decision module dynamically adjusts the navigation strategy based on the real-time updated environmental map and comprehensive risk assessment value, and generates the optimal path and navigation strategy instructions.
[0011] The dynamic path planning and control module receives the optimal path and navigation strategy instructions, and adjusts the execution mechanism in real time to achieve dynamic correction and precise execution of the path;
[0012] The self-correction module for operational errors corrects the long-term drift errors of each sensor in real time through loop closure detection.
[0013] The heterogeneous sensor array interface serves as the system's data input front end, integrating a solid-state lidar array, a 4D millimeter-wave radar module, a panoramic visual perception unit, a high-precision IMU / GNSS integrated navigation module, and environmental meteorological sensors. Through hardware timestamp synchronization and a flexible sampling interface, it collects point clouds, images, millimeter-wave echoes, inertial data, and environmental parameters in real time, and outputs a standardized multimodal data stream with spatiotemporal alignment.
[0014] The temporal environment dynamic modeling module performs temporal alignment preprocessing and dynamic feature extraction on multimodal data streams. Based on dynamic SLAM technology, it separates static environmental structures and dynamic obstacles in real time, performs dynamic point cloud filtering, and constructs a real-time updated environmental map with an update frequency of ≥10Hz.
[0015] The calculation formula for dynamic point cloud filtering in the temporal environment dynamic modeling module is as follows:
[0016] ;
[0017] In the formula, This represents the dynamic point probability mask for the current frame t. The number of points is represented by the number of points, and each element represents the probability that the corresponding 3D point belongs to a dynamic target. This represents a dynamic target segmentation network for 3D point clouds, trained end-to-end using multimodal input data. This represents the laser point cloud of the current frame. Represents the visual image of the current frame. This represents aggregated historical frame data. This represents the number of historical map points. This represents the learnable parameters of the network.
[0018] The multimodal perception optimization module calibrates the contribution of each sensor online based on real-time updated environmental maps, multimodal data streams, and historical navigation data. Its fusion weight calculation formula is as follows:
[0019] ;
[0020] In the formula, Indicates the system The combined scores of the individual sensors are summed. Indicates the number of sensors; Let represent the final fusion weight of the i-th sensor at time t. This represents the quality score of sensor i as evaluated by the deep Q-network. This represents the Kalman filter confidence level of sensor i. This represents the predicted failure probability value of sensor i. Indicates the probability of failure compensation. This indicates the influence of controlling for empirical quality scores. This indicates the influence of controlling the real-time statistical confidence level. This indicates the influence of controlling forward-looking failure prediction. Represents an exponential function;
[0021] The multimodal perception optimization module can identify solid-state LiDAR arrays, 4D millimeter-wave radar modules, panoramic vision perception units, or environmental and meteorological sensors. When this happens, the module automatically initiates sensor failure isolation and system degradation protection, and dynamically switches between perception control and navigation mode.
[0022] The multimodal perception optimization module identified a high-precision IMU / GNSS integrated navigation module or two or more sensors. If necessary, request the driver or remote operator to take over;
[0023] The unknown environment risk prediction module predicts potential collision risks and sensor failure risks based on multimodal data streams and historical navigation data, and obtains a comprehensive risk assessment value.
[0024] The unknown environment risk prediction module contains a spatiotemporal graph neural network and calculates a comprehensive risk assessment value based on multimodal data streams and historical data. The calculation formula is as follows:
[0025] ;
[0026] In the formula, This represents the overall risk assessment value. This represents the total number of predicted scenarios, that is, the number of all possible future developments considered by the system. This represents the probability of the k-th scenario occurring, with a value ranging from 0 to 1. This represents the risk value assessment for the k-th scenario. This represents the risk sensitivity coefficient, used to adjust the system's sensitivity to risk. This represents the risk attenuation factor.
[0027] The adaptive navigation decision module dynamically adjusts the navigation strategy based on the constantly updated environmental map and comprehensive risk assessment value:
[0028] After dynamic point cloud filtering, the adaptive navigation decision module adjusts the navigation decision based on the updated environmental map and comprehensive risk assessment value. Dynamically adjust navigation strategy:
[0029] when When necessary, adjust the performance-first strategy;
[0030] when At that time, a conservative traffic strategy should be adjusted.
[0031] when When necessary, proactive safety strategies should be adjusted, and the driver or remote operator should be requested to take over.
[0032] Generate the optimal path and navigation strategy instructions based on the corresponding navigation strategy.
[0033] The operation error self-correction module calculates the confidence level of loop closure detection. The calculation formula is as follows:
[0034] ;
[0035] In the formula, This indicates the confidence level of the loop closure detection. ∈(0,1), Represents ORB feature similarity, calculated by brute-force matching of ORB feature points of the current frame and candidate closed-loop frames, and the ratio of the number of successful matches to the total number of features, ranging from [0,1]. The pose consistency score is represented by the PnP algorithm, which is used to solve the relative pose of the candidate closed-loop frame and evaluate its deviation from the odometry pose prediction value. The smaller the deviation, the higher the score. This represents the time difference between the current moment and the candidate closed-loop frame moment. The longer the time, the lower the confidence level of the closed loop. , , These represent the penalty weight coefficients for feature similarity, pose consistency score, and temporal difference, respectively. Represents the sigmoid function;
[0036] when When the value is greater than 0.85, global pose graph optimization is automatically triggered, suppressing long-term drift error to within 0.5%.
[0037] The navigation control system also includes: a navigation status visualization module and a system operation and maintenance management module;
[0038] The navigation status visualization module presents the heterogeneous sensor array interface module, the time-series environment dynamic modeling module, the multimodal perception optimization module, the unknown environment risk prediction module, the adaptive navigation decision module, the operation error self-correction module, and the dynamic path planning control module in a visual interface, while also supporting manual monitoring and intervention.
[0039] The system operation and maintenance management module is responsible for equipment status monitoring, data storage and backup, algorithm model iteration and update, and access control.
[0040] Secondly, the present invention provides a deep learning-based adaptive navigation control method for unknown environments, which is implemented based on the aforementioned deep learning-based adaptive navigation control system for unknown environments.
[0041] The heterogeneous sensor array interface integrates multiple types of sensor devices to collect spatial characteristics of the unknown environment, dynamic target information, and its own operating status data in real time, thereby acquiring multimodal data streams.
[0042] Based on multimodal data streams, the current environmental structure and dynamic elements are distinguished to construct a real-time updated environmental map;
[0043] Based on real-time updated environmental maps, multimodal data streams, and historical navigation data, the contribution of each sensor is calibrated online, and sensors with a contribution below the set value are isolated or degraded.
[0044] Based on multimodal data streams and historical navigation data, potential collision risks and sensor failure risks are predicted, and a comprehensive risk assessment value is obtained.
[0045] Based on the real-time updated environmental map and comprehensive risk assessment values, the navigation strategy is dynamically adjusted to generate the optimal path and navigation strategy instructions.
[0046] It receives instructions on the optimal path and navigation strategy, adjusts the execution mechanism in real time, and achieves dynamic path correction and precise execution.
[0047] The self-correction module for operational errors corrects the long-term drift errors of each sensor in real time through loop closure detection.
[0048] Thirdly, the present invention provides a deep learning-based adaptive navigation control device for unknown environments, the deep learning-based adaptive navigation control device for unknown environments includes a memory and a processor, the memory being used to store computer program code and transmit the computer program code to the processor;
[0049] The processor is configured to execute the aforementioned process of the deep learning-based adaptive navigation control system for unknown environments according to the instructions in the computer program code.
[0050] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0051] 1. The present invention discloses an adaptive navigation control system for unknown environments based on deep learning. By using the spatiotemporal alignment technology of the heterogeneous sensor array interface module and the dynamic point cloud filtering algorithm of the temporal environment dynamic modeling module, the system can realize real-time perception and modeling of dynamic environment. This breaks the static environment assumption in traditional navigation, and makes the system more adaptable to dynamic scenes while avoiding positioning deviation caused by dynamic obstacles.
[0052] 2. The present invention provides a deep learning-based adaptive navigation control system for unknown environments. Through the dynamic weight allocation algorithm of the multimodal perception optimization module and the sensor failure isolation mechanism, it effectively solves the problems of spatiotemporal asynchrony of heterogeneous sensors, measurement noise differences, and perception failure under extreme conditions. It achieves the effect of improving the accuracy and robustness of multimodal perception fusion and ensuring stable and reliable perception in complex environments.
[0053] 3. The present invention discloses an adaptive navigation control system for unknown environments based on deep learning. By combining the loop closure detection confidence algorithm of the self-correction module and the hierarchical strategy of the adaptive navigation decision module, the accumulated drift error in long-term tasks is dynamically corrected, and the real-time matching of navigation decisions and environmental risks is achieved. This results in suppressing the divergence of long-period errors and balancing navigation safety and efficiency. Attached Figure Description
[0054] Figure 1 This is a schematic diagram of the structure of the present invention.
[0055] Figure 2 This is a diagram of the device of the present invention. Detailed Implementation
[0056] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0057] Example 1:
[0058] See Figure 1An adaptive navigation control system for unknown environments based on deep learning specifically includes: a heterogeneous sensor array interface, a time-series environment dynamic modeling module, a multimodal perception optimization module, an unknown environment risk prediction module, an adaptive navigation decision module, an operation error self-correction module, and a dynamic path planning control module.
[0059] The heterogeneous sensor array interface integrates multiple types of sensor devices to collect spatial characteristics of the unknown environment, dynamic target information, and its own operating status data in real time, thereby acquiring multimodal data streams.
[0060] The time-series environment dynamic modeling module distinguishes the current environment structure and dynamic elements based on multimodal data streams, and constructs a real-time updated environment map;
[0061] The multimodal perception optimization module calibrates the contribution of each sensor online based on real-time updated environmental maps, multimodal data streams, and historical navigation data, and isolates or downgrades the operation of sensors whose contribution is lower than a set value.
[0062] The unknown environment risk prediction module predicts potential collision risks and sensor failure risks based on multimodal data streams and historical navigation data, and obtains a comprehensive risk assessment value.
[0063] The adaptive navigation decision module dynamically adjusts the navigation strategy based on the real-time updated environmental map and comprehensive risk assessment value, and generates the optimal path and navigation strategy instructions.
[0064] The dynamic path planning and control module receives the optimal path and navigation strategy instructions, and adjusts the execution mechanism in real time to achieve dynamic correction and precise execution of the path;
[0065] The self-correction module for operational errors corrects the long-term drift errors of each sensor in real time through loop closure detection.
[0066] The heterogeneous sensor array interface serves as the system's data input front end, integrating a solid-state lidar array, a 4D millimeter-wave radar module, a panoramic visual perception unit, a high-precision IMU / GNSS integrated navigation module, and environmental meteorological sensors. Through hardware timestamp synchronization and a flexible sampling interface, it collects point clouds, images, millimeter-wave echoes, inertial data, and environmental parameters in real time, and outputs a standardized multimodal data stream with spatiotemporal alignment.
[0067] The temporal environment dynamic modeling module performs temporal alignment preprocessing and dynamic feature extraction on multimodal data streams. Based on dynamic SLAM technology, it separates static environmental structures and dynamic obstacles in real time, performs dynamic point cloud filtering, and constructs a real-time updated environmental map with an update frequency of ≥10Hz.
[0068] The calculation formula for dynamic point cloud filtering in the temporal environment dynamic modeling module is as follows:
[0069] ;
[0070] In the formula, This represents the dynamic point probability mask for the current frame t. The number of points is represented by the number of points, and each element represents the probability that the corresponding 3D point belongs to a dynamic target. This represents a dynamic target segmentation network for 3D point clouds, trained end-to-end using multimodal input data. This represents the laser point cloud of the current frame. Represents the visual image of the current frame. This represents aggregated historical frame data. This represents the number of historical map points. This represents the learnable parameters of the network.
[0071] The multimodal perception optimization module calibrates the contribution of each sensor online based on real-time updated environmental maps, multimodal data streams, and historical navigation data. Its fusion weight calculation formula is as follows:
[0072] ;
[0073] In the formula, Indicates the system One sensor (from) arrive Sum the combined scores of ) Indicates the number of sensors; This represents the final fusion weight of the i-th sensor at time t, and is between 0 and 1. The closer the value is to 0, the lower the reliability of the sensor data at a previous moment, and the lower the degree to which it is adopted. The quality score of sensor i, as evaluated by the deep Q-network, is derived from the environment-sensor coupling feature vector. The input includes a joint encoding of current frame environmental parameters (visibility, temperature, humidity, vibration level) and sensor echo statistical features (point cloud density, image signal-to-noise ratio, millimeter-wave reflection intensity). The quality score is output in real-time through an offline-trained empirical deep Q-network. The Kalman filter confidence score of sensor i is derived from the innovation covariance matrix. It is calculated using a Kalman filter to determine the current measurement residual (the deviation between the predicted value and the actual point cloud / image / inertial data) and its covariance trace, thus characterizing the statistical consistency of the sensor under the current dynamic state. The predicted failure probability of sensor i is derived from the sensor physical state monitoring network. Inputs include real-time hardware status codes (temperature, power consumption, communication latency) and historical failure mode data. A lightweight LSTM network is used to predict the short-term failure probability. Indicates the probability of failure compensation. This indicates the influence of controlling for empirical quality scores. This indicates the influence of controlling the real-time statistical confidence level. The influence of forward-looking fault prediction is indicated by the use of optimization algorithms such as gradient descent and training with a large amount of data, so that the system can automatically find the optimal trade-off strategy in different scenarios and improve the perceived reliability under extreme conditions. Represents an exponential function;
[0074] The multimodal perception optimization module can identify solid-state LiDAR arrays, 4D millimeter-wave radar modules, panoramic vision perception units, or environmental and meteorological sensors. In such cases, the module automatically initiates sensor failure isolation and system degradation protection, and dynamically switches between perception dominance and navigation mode to prevent systemic misjudgments and navigation failures caused by the failure of a single perception path.
[0075] The multimodal perception optimization module identified a high-precision IMU / GNSS integrated navigation module or two or more sensors. If necessary, request the driver or remote operator to take over;
[0076] The quality score of sensor i as evaluated by the deep Q network is as follows: Based on sensor observation data, the deep Q network is used as a reinforcement learning agent. The current multi-sensor observation and system state are used as the state input, and the weight allocation strategy of each sensor is used as the action. The reward function is constructed with collision avoidance, improved positioning accuracy, trajectory smoothness and energy consumption reduction as navigation performance indicators. The "effectiveness" or "reliability" of different sensors in the current environment is scored to obtain the sensor quality score evaluated by the deep Q network.
[0077] The Kalman filter confidence score for sensor i is represented by the following: After each update step, the Kalman filter outputs the state estimate and its covariance matrix. Each sensor runs a sub-filter separately, and its local confidence score is extracted as the Kalman filter confidence score.
[0078] The failure probability prediction value of sensor i is represented by a neural network based on the statistical characteristics of recent sensor readings and a sliding window data. The real-time updated environmental map is input into the neural network to obtain the failure probability prediction value of sensor i.
[0079] By using a dynamic weight allocation algorithm to calibrate the contribution of each sensor online, dynamically adjust the fusion weight of heterogeneous sensors, and isolate failed sensors, the traditional fusion strategy is unable to cope with the problems of sensor spatiotemporal asynchrony, measurement noise differences, and sensing failure under extreme conditions, thereby ensuring the beneficial effect of stable sensing under extreme conditions.
[0080] The unknown environment risk prediction module predicts potential collision risks and sensor failure risks based on multimodal data streams and historical navigation data, and obtains a comprehensive risk assessment value.
[0081] The unknown environment risk prediction module contains a spatiotemporal graph neural network and calculates a comprehensive risk assessment value based on multimodal data streams and historical data. The calculation formula is as follows:
[0082] ;
[0083] In the formula, This represents the overall risk assessment value. This represents the total number of predicted scenarios, that is, the number of all possible future developments considered by the system. This represents the probability of the k-th scenario occurring, with a value ranging from 0 to 1. This represents the risk value assessment for the k-th scenario. This represents the risk sensitivity coefficient, used to adjust the system's sensitivity to risk. This represents the risk attenuation factor.
[0084] ;
[0085] The probability of the kth scenario occurring and risk value assessment Both methods utilize historical data to train a spatiotemporal graph neural network prediction model, and input real-time multimodal data streams into the spatiotemporal graph neural network prediction model to obtain the probability of occurrence and risk value assessment of the k-th scenario at time t.
[0086] The adaptive navigation decision module dynamically adjusts the navigation strategy based on the constantly updated environmental map and comprehensive risk assessment value:
[0087] After dynamic point cloud filtering, the adaptive navigation decision module adjusts the navigation decision based on the updated environmental map and comprehensive risk assessment value. Dynamically adjust navigation strategy:
[0088] when When necessary, adjust the performance-first strategy;
[0089] when At that time, a conservative traffic strategy should be adjusted.
[0090] when When necessary, proactive safety strategies should be adjusted, and the driver or remote operator should be requested to take over.
[0091] Generate the optimal path and navigation strategy instructions based on the corresponding navigation strategy.
[0092] After dynamic point cloud filtering, the adaptive navigation decision module adjusts the navigation decision based on the updated environmental map and comprehensive risk assessment value. Dynamically adjust navigation strategy when When the system determines that the environmental risk is extremely low, it adopts globally optimal path planning, allowing the path curvature to approach the kinematic limit to shorten the travel distance; it increases the cruising speed to the preset upper limit; and it temporarily relaxes the safety margin for static obstacles to obtain better passage space. When the system determines that the environmental risk is moderate but there are uncertain dynamic factors, it will adjust the conservative traffic strategy, activate local replanning that considers the predicted trajectory of dynamic obstacles, reduce the cruising speed to 80% of the safe speed limit, increase the interaction distance with all obstacles, and prepare to execute courtesy yielding rules (such as slowing down to yield to pedestrians crossing the road). When the system determines that the environmental risk is extremely high and there is an imminent threat of collision or system failure, it immediately triggers an emergency avoidance maneuver (such as emergency braking or aggressive obstacle avoidance); it abandons global path tracking and prioritizes instantaneous safety; it maintains control at the highest decision-making level of the system and simultaneously sends the highest level alarm to the path planning control module and navigation status visualization module, requesting the driver or remote operator to take over.
[0093] By calculating the comprehensive risk assessment value By dynamically adjusting the hierarchical navigation strategy, the navigation decision-making and environmental risks are matched in real time, solving the problem that traditional navigation strategies are fixed and difficult to adapt to the uncertainty of unknown environments.
[0094] The operation error self-correction module calculates the confidence level of loop closure detection. The calculation formula is as follows:
[0095] ;
[0096] In the formula, This indicates the confidence level of the loop closure detection. ∈(0,1), Represents ORB feature similarity, calculated by brute-force matching of ORB feature points of the current frame and candidate closed-loop frames, and the ratio of the number of successful matches to the total number of features, ranging from [0,1]. The pose consistency score is represented by the PnP algorithm, which is used to solve the relative pose of the candidate closed-loop frame and evaluate its deviation from the odometry pose prediction value. The smaller the deviation, the higher the score. This represents the time difference between the current moment and the candidate closed-loop frame moment. The longer the time, the lower the confidence level of the closed loop. , , These represent the penalty weight coefficients for feature similarity, pose consistency score, and temporal difference, respectively. Represents the sigmoid function;
[0097] The ORB feature similarity, derived from the panoramic image sequence output by the panoramic vision perception unit, is calculated as follows: ORB feature descriptors are extracted from the current frame image, and brute-force matching is performed with the ORB features of candidate closed-loop frames. The Hamming distance similarity score is then calculated and normalized. The pose consistency score, derived from real-time pose data output by the high-precision IMU / GNSS integrated navigation module, is calculated as follows: the Euclidean distance difference and angle difference between the current frame pose and the candidate closed-loop frame pose are calculated, and then mapped to a consistency probability score using a Gaussian kernel function. The time difference between the current time and the candidate closed-loop frame time comes from the hardware timestamp synchronization mechanism. The specific calculation process is as follows: the UTC timestamp of the current frame minus the historical timestamp stored in the candidate closed-loop frame, in seconds. This is used to penalize the cumulative error uncertainty caused by long time intervals. The longer the time, the better.
[0098] when When the value is greater than 0.85, global pose graph optimization is automatically triggered, suppressing long-term drift error to within 0.5%.
[0099] Confidence level by loop closure detection The adaptive error correction dynamically corrects the cumulative drift of the visual odometry, solving the problem of error divergence in traditional long-term navigation tasks and achieving the effect of improving long-cycle navigation accuracy and reducing path redundancy.
[0100] The navigation control system also includes: a navigation status visualization module and a system operation and maintenance management module;
[0101] The navigation status visualization module presents the heterogeneous sensor array interface module, the time-series environment dynamic modeling module, the multimodal perception optimization module, the unknown environment risk prediction module, the adaptive navigation decision module, the operation error self-correction module, and the dynamic path planning control module in a visual interface, while also supporting manual monitoring and intervention.
[0102] The system operation and maintenance management module is responsible for equipment status monitoring, data storage and backup, algorithm model iteration and update, and access control.
[0103] This invention solves the core problems of traditional methods, such as poor adaptability in dynamic scenarios, inefficient fusion of heterogeneous sensors, and navigation inaccuracy caused by accumulated errors in long-term tasks, through a closed-loop linkage modular architecture, accurate calculation formulas, and efficient hardware adaptation. It can be widely applied to navigation scenarios in complex and unknown environments for autonomous robots and unmanned vehicles.
[0104] Example 2:
[0105] A deep learning-based adaptive navigation control method for unknown environments, wherein the heterogeneous sensor array interface integrates multiple types of sensor devices to collect spatial characteristics, dynamic target information and its own operating status data of the unknown environment in real time, and obtain multimodal data streams;
[0106] The heterogeneous sensor array interface serves as the system's data input front end, integrating a solid-state lidar array, a 4D millimeter-wave radar module, a panoramic visual perception unit, a high-precision IMU / GNSS integrated navigation module, and environmental meteorological sensors. Through hardware timestamp synchronization and a flexible sampling interface, it collects point clouds, images, millimeter-wave echoes, inertial data, and environmental parameters in real time, and outputs a standardized multimodal data stream with spatiotemporal alignment.
[0107] Based on multimodal data streams, the current environmental structure and dynamic elements are distinguished to construct a real-time updated environmental map;
[0108] The temporal environment dynamic modeling module performs temporal alignment preprocessing and dynamic feature extraction on multimodal data streams. Based on dynamic SLAM technology, it separates static environmental structures and dynamic obstacles in real time, performs dynamic point cloud filtering, and constructs a real-time updated environmental map with an update frequency of ≥10Hz.
[0109] The calculation formula for dynamic point cloud filtering in the temporal environment dynamic modeling module is as follows:
[0110] ;
[0111] In the formula, This represents the dynamic point probability mask for the current frame t. The number of points is represented by the number of points, and each element represents the probability that the corresponding 3D point belongs to a dynamic target. This represents a dynamic target segmentation network for 3D point clouds, trained end-to-end using multimodal input data. This represents the laser point cloud of the current frame. Represents the visual image of the current frame. This represents aggregated historical frame data. This represents the number of historical map points. This represents the learnable parameters of the network.
[0112] Based on real-time updated environmental maps, multimodal data streams, and historical navigation data, the contribution of each sensor is calibrated online, and sensors with a contribution below the set value are isolated or degraded.
[0113] The multimodal perception optimization module calibrates the contribution of each sensor online based on real-time updated environmental maps, multimodal data streams, and historical navigation data. Its fusion weight calculation formula is as follows:
[0114] ;
[0115] In the formula, Indicates the system One sensor (from) arrive Sum the combined scores of ) Indicates the number of sensors; Let represent the final fusion weight of the i-th sensor at time t. This represents the quality score of sensor i as evaluated by the deep Q-network. This represents the Kalman filter confidence level of sensor i. This represents the predicted failure probability value of sensor i. Indicates the probability of failure compensation. This indicates the influence of controlling for empirical quality scores. This indicates the influence of controlling the real-time statistical confidence level. This indicates the influence of controlling forward-looking failure prediction. Represents an exponential function;
[0116] Based on the real-time updated environmental map and comprehensive risk assessment values, the navigation strategy is dynamically adjusted to generate the optimal path and navigation strategy instructions.
[0117] The multimodal perception optimization module can identify solid-state LiDAR arrays, 4D millimeter-wave radar modules, panoramic vision perception units, or environmental and meteorological sensors. When this happens, the module automatically initiates sensor failure isolation and system degradation protection, and dynamically switches between perception control and navigation mode.
[0118] The multimodal perception optimization module identified a high-precision IMU / GNSS integrated navigation module or two or more sensors. If necessary, request the driver or remote operator to take over;
[0119] The quality score of sensor i as evaluated by the deep Q network is as follows: Based on sensor observation data, the deep Q network is used as a reinforcement learning agent. The current multi-sensor observation and system state are used as the state input, and the weight allocation strategy of each sensor is used as the action. The reward function is constructed with collision avoidance, improved positioning accuracy, trajectory smoothness and energy consumption reduction as navigation performance indicators. The "effectiveness" or "reliability" of different sensors in the current environment is scored to obtain the sensor quality score evaluated by the deep Q network.
[0120] The Kalman filter confidence score for sensor i is represented by the following: After each update step, the Kalman filter outputs the state estimate and its covariance matrix. Each sensor runs a sub-filter separately, and its local confidence score is extracted as the Kalman filter confidence score.
[0121] The failure probability prediction value of sensor i is represented by a neural network based on the statistical characteristics of recent sensor readings and a sliding window data. The real-time updated environmental map is input into the neural network to obtain the failure probability prediction value of sensor i.
[0122] Based on multimodal data streams and historical navigation data, potential collision risks and sensor failure risks are predicted, and a comprehensive risk assessment value is obtained.
[0123] The unknown environment risk prediction module predicts potential collision risks and sensor failure risks based on multimodal data streams and historical navigation data, and obtains a comprehensive risk assessment value.
[0124] The unknown environment risk prediction module contains a spatiotemporal graph neural network and calculates a comprehensive risk assessment value based on multimodal data streams and historical data. The calculation formula is as follows:
[0125] ;
[0126] In the formula, This represents the overall risk assessment value. This represents the total number of predicted scenarios, that is, the number of all possible future developments considered by the system. This represents the probability of the k-th scenario occurring, with a value ranging from 0 to 1. This represents the risk value assessment for the k-th scenario. This represents the risk sensitivity coefficient, used to adjust the system's sensitivity to risk. This represents the risk attenuation factor.
[0127] The probability of the kth scenario occurring and risk value assessment Both methods utilize historical data to train a spatiotemporal graph neural network prediction model, and input real-time multimodal data streams into the spatiotemporal graph neural network prediction model to obtain the probability of occurrence and risk value assessment of the k-th scenario at time t.
[0128] It receives instructions on the optimal path and navigation strategy, adjusts the execution mechanism in real time, and achieves dynamic path correction and precise execution.
[0129] The adaptive navigation decision module dynamically adjusts the navigation strategy based on the constantly updated environmental map and comprehensive risk assessment value:
[0130] After dynamic point cloud filtering, the adaptive navigation decision module adjusts the navigation decision based on the updated environmental map and comprehensive risk assessment value. Dynamically adjust navigation strategy:
[0131] when When necessary, adjust the performance-first strategy;
[0132] when At that time, a conservative traffic strategy should be adjusted.
[0133] when When necessary, proactive safety strategies should be adjusted, and the driver or remote operator should be requested to take over.
[0134] Generate the optimal path and navigation strategy instructions based on the corresponding navigation strategy.
[0135] The self-correction module for operational errors corrects the long-term drift errors of each sensor in real time through loop closure detection.
[0136] The operation error self-correction module calculates the confidence level of loop closure detection. The calculation formula is as follows:
[0137] ;
[0138] In the formula, This indicates the confidence level of the loop closure detection. ∈(0,1), Represents ORB feature similarity, calculated by brute-force matching of ORB feature points of the current frame and candidate closed-loop frames, and the ratio of the number of successful matches to the total number of features, ranging from [0,1]. The pose consistency score is represented by the PnP algorithm, which is used to solve the relative pose of the candidate closed-loop frame and evaluate its deviation from the odometry pose prediction value. The smaller the deviation, the higher the score. This represents the time difference between the current moment and the candidate closed-loop frame moment. The longer the time, the lower the confidence level of the closed loop. , , These represent the penalty weight coefficients for feature similarity, pose consistency score, and temporal difference, respectively. Represents the sigmoid function;
[0139] when When the value is greater than 0.85, global pose graph optimization is automatically triggered, suppressing long-term drift error to within 0.5%.
[0140] Navigation status visualization: The heterogeneous sensor array interface module, the time-series environment dynamic modeling module, the multimodal perception optimization module, the unknown environment risk prediction module, the adaptive navigation decision module, the operation error self-correction module, and the dynamic path planning control module are presented in a visual interface, while also supporting manual monitoring and intervention;
[0141] System operation and maintenance management: responsible for equipment status monitoring, data storage and backup, algorithm model iteration and update, and access control.
[0142] Example 3:
[0143] See Figure 2 A deep learning-based adaptive navigation control device for unknown environments includes a memory and a processor. The memory is used to store computer program code and transmit the computer program code to the processor.
[0144] The processor is configured to execute the aforementioned process of the deep learning-based adaptive navigation control system for unknown environments according to the instructions in the computer program code.
Claims
1. A deep learning-based adaptive navigation control system for unknown environments, characterized in that, Specifically, it includes: Heterogeneous sensor array interface, time-series environment dynamic modeling module, multimodal perception optimization module, unknown environment risk prediction module, adaptive navigation decision module, operation error self-correction module and dynamic path planning control module; The heterogeneous sensor array interface integrates multiple types of sensor devices to collect spatial characteristics of the unknown environment, dynamic target information, and its own operating status data in real time, thereby acquiring multimodal data streams. The time-series environment dynamic modeling module distinguishes the current environment structure and dynamic elements based on multimodal data streams, and constructs a real-time updated environment map; The multimodal perception optimization module calibrates the contribution of each sensor online based on real-time updated environmental maps, multimodal data streams, and historical navigation data, and isolates or downgrades the operation of sensors whose contribution is lower than a set value. The unknown environment risk prediction module predicts potential collision risks and sensor failure risks based on multimodal data streams and historical navigation data, and obtains a comprehensive risk assessment value. The adaptive navigation decision module dynamically adjusts the navigation strategy based on the real-time updated environmental map and comprehensive risk assessment value, and generates the optimal path and navigation strategy instructions. The dynamic path planning and control module receives the optimal path and navigation strategy instructions, and adjusts the execution mechanism in real time to achieve dynamic correction and precise execution of the path; The self-correction module for operational errors corrects the long-term drift errors of each sensor in real time through loop closure detection.
2. The deep learning-based adaptive navigation control system for unknown environments according to claim 1, characterized in that: The heterogeneous sensor array interface serves as the system's data input front end, integrating a solid-state lidar array, a 4D millimeter-wave radar module, a panoramic visual perception unit, a high-precision IMU / GNSS integrated navigation module, and environmental meteorological sensors. Through hardware timestamp synchronization and a flexible sampling interface, it collects point clouds, images, millimeter-wave echoes, inertial data, and environmental parameters in real time, and outputs a standardized multimodal data stream with spatiotemporal alignment.
3. The deep learning-based adaptive navigation control system for unknown environments according to claim 1, characterized in that: The temporal environment dynamic modeling module performs temporal alignment preprocessing and dynamic feature extraction on multimodal data streams. Based on dynamic SLAM technology, it separates static environmental structures and dynamic obstacles in real time, performs dynamic point cloud filtering, and constructs a real-time updated environmental map with an update frequency of ≥10Hz. The calculation formula for dynamic point cloud filtering in the temporal environment dynamic modeling module is as follows: ； In the formula, This represents the dynamic point probability mask for the current frame t. The number of points is represented by the number of points, and each element represents the probability that the corresponding 3D point belongs to a dynamic target. This represents a dynamic target segmentation network for 3D point clouds, trained end-to-end using multimodal input data. This represents the laser point cloud of the current frame. Represents the visual image of the current frame. This represents aggregated historical frame data. This represents the number of historical map points. This represents the learnable parameters of the network.
4. The deep learning-based adaptive navigation control system for unknown environments according to claim 1, characterized in that: The multimodal perception optimization module calibrates the contribution of each sensor online based on real-time updated environmental maps, multimodal data streams, and historical navigation data. Its fusion weight calculation formula is as follows: ； In the formula, Indicates the system The combined scores of the individual sensors are summed. Indicates the number of sensors; Let represent the final fusion weight of the i-th sensor at time t. This represents the quality score of sensor i as evaluated by the deep Q-network. This represents the Kalman filter confidence level of sensor i. This represents the predicted failure probability value of sensor i. Indicates the probability of failure compensation. This indicates the influence of controlling for empirical quality scores. This indicates the influence of controlling the real-time statistical confidence level. This indicates the influence of controlling forward-looking failure prediction. Represents an exponential function; The multimodal perception optimization module can identify solid-state LiDAR arrays, 4D millimeter-wave radar modules, panoramic vision perception units, or environmental and meteorological sensors. When this happens, the module automatically initiates sensor failure isolation and system degradation protection, and dynamically switches between perception control and navigation mode. The multimodal perception optimization module identified a high-precision IMU / GNSS integrated navigation module or two or more sensors. If necessary, request the driver or remote operator to take over.
5. The deep learning-based adaptive navigation control system for unknown environments according to claim 1, characterized in that: The unknown environment risk prediction module predicts potential collision risks and sensor failure risks based on multimodal data streams and historical navigation data, and obtains a comprehensive risk assessment value. The unknown environment risk prediction module contains a spatiotemporal graph neural network and calculates a comprehensive risk assessment value based on multimodal data streams and historical data. The calculation formula is as follows: ； In the formula, This represents the overall risk assessment value. This represents the total number of predicted scenarios, that is, the number of all possible future developments considered by the system. This represents the probability of the k-th scenario occurring, with a value ranging from 0 to 1. This represents the risk value assessment for the k-th scenario. This represents the risk sensitivity coefficient, used to adjust the system's sensitivity to risk. This represents the risk attenuation factor.
6. The deep learning-based adaptive navigation control system for unknown environments according to claim 1, characterized in that: The adaptive navigation decision module dynamically adjusts the navigation strategy based on the constantly updated environmental map and comprehensive risk assessment value: After dynamic point cloud filtering, the adaptive navigation decision module adjusts the navigation decision based on the updated environmental map and comprehensive risk assessment value. Dynamically adjust navigation strategy: when When necessary, adjust the performance-first strategy; when At that time, a conservative traffic strategy should be adjusted. when When necessary, proactive safety strategies should be adjusted, and the driver or remote operator should be requested to take over. Generate the optimal path and navigation strategy instructions based on the corresponding navigation strategy.
7. The deep learning-based adaptive navigation control system for unknown environments according to claim 1, characterized in that: The operation error self-correction module calculates the confidence level of loop closure detection. The calculation formula is as follows: ； In the formula, This indicates the confidence level of the loop closure detection. ∈(0,1), Represents ORB feature similarity, calculated by brute-force matching of ORB feature points of the current frame and candidate closed-loop frames, and the ratio of the number of successful matches to the total number of features, ranging from [0,1]. The pose consistency score is represented by the PnP algorithm, which is used to solve the relative pose of the candidate closed-loop frame and evaluate its deviation from the odometry pose prediction value. The smaller the deviation, the higher the score. This represents the time difference between the current moment and the candidate closed-loop frame moment. The longer the time, the lower the confidence level of the closed loop. , , These represent the penalty weight coefficients for feature similarity, pose consistency score, and temporal difference, respectively. Represents the sigmoid function; when When the value is greater than 0.85, global pose graph optimization is automatically triggered, suppressing long-term drift error to within 0.5%.
8. The deep learning-based adaptive navigation control system for unknown environments according to claim 1, characterized in that: The navigation control system also includes: a navigation status visualization module and a system operation and maintenance management module; The navigation status visualization module presents the heterogeneous sensor array interface module, the time-series environment dynamic modeling module, the multimodal perception optimization module, the unknown environment risk prediction module, the adaptive navigation decision module, the operation error self-correction module, and the dynamic path planning control module in a visual interface, while also supporting manual monitoring and intervention. The system operation and maintenance management module is responsible for equipment status monitoring, data storage and backup, algorithm model iteration and update, and access control.
9. A deep learning-based adaptive navigation control method for unknown environments, characterized in that: The deep learning-based adaptive navigation control method for unknown environments is implemented based on the deep learning-based adaptive navigation control system for unknown environments as described in any one of claims 1-8. The heterogeneous sensor array interface integrates multiple types of sensor devices to collect spatial characteristics of the unknown environment, dynamic target information, and its own operating status data in real time, thereby acquiring multimodal data streams. Based on multimodal data streams, the current environmental structure and dynamic elements are distinguished to construct a real-time updated environmental map; Based on real-time updated environmental maps, multimodal data streams, and historical navigation data, the contribution of each sensor is calibrated online, and sensors with a contribution below the set value are isolated or degraded. Based on multimodal data streams and historical navigation data, potential collision risks and sensor failure risks are predicted, and a comprehensive risk assessment value is obtained. Based on the real-time updated environmental map and comprehensive risk assessment values, the navigation strategy is dynamically adjusted to generate the optimal path and navigation strategy instructions. It receives instructions on the optimal path and navigation strategy, adjusts the execution mechanism in real time, and achieves dynamic path correction and precise execution. The self-correction module for operational errors corrects the long-term drift errors of each sensor in real time through loop closure detection.
10. A deep learning-based adaptive navigation control device for unknown environments, characterized in that: The deep learning-based adaptive navigation control device for unknown environments includes a memory and a processor. The memory is used to store computer program code and transmit the computer program code to the processor. The processor is configured to execute the process of the deep learning-based adaptive navigation control system for unknown environments as described in any one of claims 1 to 8, according to instructions in the computer program code.