Robot autonomous navigation system based on multi-modal sensor fusion detection

Abstract

The invention belongs to the technical field of robots, and provides a robot autonomous navigation system based on multi-modal sensor fusion detection, and the system comprises a plurality of sensors with different modals, which are used for obtaining environment information; the data fusion module is used for receiving data from a plurality of sensors and performing data fusion processing by adopting a preset sensing model; the autonomous navigation module is used for carrying out path planning and navigation decision making based on the fused environment perception information; the feedback adjusting module is used for adjusting the fusion strategy and the navigation strategy according to real-time feedback in the navigation process; according to the system, by integrating a plurality of sensors with different modalities, such as a laser radar, a visual camera and an inertial sensor, richer environment information can be obtained, and a preset sensing model is adopted in the data fusion module to perform data fusion processing, so that the navigation precision and the adaptive capacity of the robot in a complex environment are improved.

Description

Technical Field

[0001] The invention belongs to the technical field of robots, and in particular to a robot autonomous navigation system based on multi-modal sensor fusion detection. Background Art

[0002] With the continuous development of robotics technology, autonomous navigation has become an important research direction in the field of robotics. However, most of the existing robotic autonomous navigation systems rely on a single sensor, such as lidar or visual camera, which leads to the system often having problems such as insufficient perception information, low navigation accuracy, and limited obstacle recognition and tracking capabilities when facing complex environments.

[0003] To this end, those skilled in the art have proposed a robot autonomous navigation system based on multimodal sensor fusion detection to solve the problems raised by the background technology. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a robot autonomous navigation system based on multimodal sensor fusion detection to solve the problem in the prior art that when the sensor malfunctions or fails, the existing system is often unable to effectively perform data fusion and compensation, thereby affecting the robot's autonomous navigation capability.

[0005] A robot autonomous navigation system based on multimodal sensor fusion detection, comprising:

[0006] Multiple sensors of different modes are used to obtain environmental information; the multiple sensors of different modes include laser radar, visual camera, inertial sensor, etc.;

[0007] The data fusion module is used to receive data from multiple sensors and use a preset perception model to perform data fusion processing to obtain comprehensive environmental perception information;

[0008] An autonomous navigation module performs path planning and navigation decisions based on the fused environmental perception information; the autonomous navigation module also includes an obstacle detection and tracking unit for fusing data from a lidar, camera, and millimeter-wave radar to accurately identify and track obstacles on the road;

[0009] The feedback adjustment module is used to adjust the fusion strategy and navigation strategy according to the real-time feedback during the navigation process.

[0010] Preferably, in the data fusion module, before the data fusion process is performed, the sensor data is decomposed at multiple scales using wavelet transform to improve the resolution and accuracy of the data.

[0011] Preferably, the preset perception model is trained in the following manner: obtaining non-failed sensor data obtained by multiple sensors when they are not failed, randomly selecting one or more non-failed sensor data and converting them into failed sensor data obtained when the corresponding sensor fails, and training with failed sensor data and non-failed sensor data that are not randomly selected.

[0012] Preferably, in the data fusion module, a dynamic weight allocation based on variance is introduced to improve the accuracy of the fused data;

[0013] At the same time, a factor graph optimization algorithm is introduced to perform global optimization on multimodal sensor data, thereby obtaining more accurate and consistent data fusion results.

[0014] Preferably, in the data fusion module, the attention mechanism is used to dynamically adjust the importance of different modal features.

[0015] Preferably, in the autonomous navigation module, a deep reinforcement learning algorithm is used to train the robot to perform optimal path planning.

[0016] Preferably, in the autonomous navigation module, the navigation decision is made using a Monte Carlo tree search algorithm for more efficient decision making.

[0017] Preferably, the feedback regulation module can dynamically adjust the use and fusion strategy of the sensor according to environmental changes to ensure efficient and stable operation in various scenarios, and its dynamic adjustment adopts an adaptive weight adjustment algorithm.

[0018] A processor is configured to execute the above-mentioned robot autonomous navigation system based on multi-modal sensor fusion detection.

[0019] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned robot autonomous navigation system based on multimodal sensor fusion detection.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] 1. The present invention can obtain richer environmental information by integrating multiple sensors of different modalities, such as lidar, visual camera, inertial sensor, etc.; these data are processed in the data fusion module to obtain comprehensive environmental perception information, thereby improving the navigation accuracy and adaptability of the robot in complex environments.

[0022] 2. The obstacle detection and tracking unit in the autonomous navigation module of the present invention integrates data from lidar, camera and millimeter-wave radar, which can accurately identify and track obstacles on the road; it helps the robot to avoid obstacles in time during driving to ensure safety.

[0023] 3. In the data fusion module, the present invention adopts technical means such as wavelet transform, variance-based dynamic weight allocation, factor graph optimization algorithm and attention mechanism to improve the accuracy and stability of fused data; these technologies can dynamically adjust the fusion strategy according to the characteristics of different sensors and real-time feedback, so as to obtain more accurate and consistent data fusion results.

[0024] 4. The autonomous navigation module of the present invention uses deep reinforcement learning algorithm and Monte Carlo tree search algorithm for path planning and navigation decision-making, which can formulate more efficient and accurate navigation strategies; it helps the robot to quickly find the optimal path in a complex environment and improve navigation efficiency.

[0025] 5. The feedback regulation module of the present invention can dynamically adjust the use and fusion strategy of sensors according to environmental changes to ensure efficient and stable operation in various situations; it helps to improve the adaptability and stability of the system, so that it can perform good navigation performance in different environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a framework diagram of the robot autonomous navigation system based on multimodal sensor fusion detection of the present invention. DETAILED DESCRIPTION

[0027] The following embodiments of the present invention are described in further detail in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0028] Embodiment: The present invention provides a robot autonomous navigation system based on multimodal sensor fusion detection, such as Figure 1 As shown, it includes multiple sensors of different modes, a data fusion module, an autonomous navigation module and a feedback adjustment module, and the multiple sensors of different modes, the data fusion module, the autonomous navigation module and the feedback adjustment module are electrically connected in sequence:

[0029] Multiple sensors of different modes are used to obtain environmental information; the multiple sensors of different modes include laser radar, visual camera, inertial sensor, etc.;

[0030] The data fusion module is used to receive data from multiple sensors and use a preset perception model to perform data fusion processing to obtain comprehensive environmental perception information;

[0031] An autonomous navigation module performs path planning and navigation decisions based on the fused environmental perception information; the autonomous navigation module also includes an obstacle detection and tracking unit for fusing data from a lidar, camera, and millimeter-wave radar to accurately identify and track obstacles on the road;

[0032] The feedback adjustment module is used to adjust the fusion strategy and navigation strategy according to the real-time feedback during the navigation process.

[0033] From the above, we can see that by integrating multiple sensors of different modalities such as lidar, visual cameras, inertial sensors, etc., it is possible to obtain richer environmental information, and use the preset perception model in the data fusion module for data fusion processing, thereby improving the robot's navigation accuracy and adaptability in complex environments; at the same time, the obstacle detection and tracking unit in the autonomous navigation module can accurately identify and track obstacles on the road to ensure the safety of the robot during driving; in addition, the system can also dynamically adjust the fusion strategy and navigation strategy according to real-time feedback during the navigation process, further improving the stability and adaptability of the system, so that it can perform good navigation performance in different environments.

[0034] Furthermore, in the data fusion module, before the data fusion process is performed, the sensor data is decomposed at multiple scales using wavelet transform to improve the resolution and accuracy of the data. The formula of the wavelet transform is as follows:

[0035]

[0036] Among them, F(x,y) is the fused data, φ j,k,l (x,y) is the wavelet coefficient at scale j and position (k,l), w j (k,l) is the corresponding fusion weight.

[0037] From the above, we can see that in the data fusion module, using wavelet transform to decompose sensor data at multiple scales can significantly improve the resolution and accuracy of the data; through wavelet transform, the system can finely capture and analyze the characteristics of sensor data at different scales, thereby fusing more accurate and comprehensive environmental perception information, providing strong support for subsequent autonomous navigation.

[0038] Furthermore, the preset perception model is trained in the following manner: obtaining non-failed sensor data obtained by multiple sensors when they are not failed, randomly selecting one or more non-failed sensor data and converting them into failed sensor data obtained when the corresponding sensor fails, and training with failed sensor data and non-failed sensor data that are not randomly selected.

[0039] From the above, we can see that by using the data of both intact and failed sensors for training, the model's robustness to sensor failures can be enhanced; in this way, the model can learn how to effectively use the data of other intact sensors for compensation in the event of a sensor failure, thereby ensuring that when a sensor fails, the system can still perform accurate data fusion and navigation decisions, thereby improving the reliability and stability of the system.

[0040] Furthermore, in the data fusion module, a variance-based dynamic weight allocation is introduced to improve the accuracy of the fused data. The formula is as follows:

[0041]

[0042] Among them, w i is the weight of the ith sensor, σ i is the measurement variance of the i-th sensor;

[0043] At the same time, a factor graph optimization algorithm is introduced to perform global optimization on the multimodal sensor data, so as to obtain more accurate and consistent data fusion results. The formula of the factor graph optimization algorithm is as follows:

[0044]

[0045] Where X represents the set of variables to be optimized (such as sensor position, posture, etc.), f i (X) is the error function associated with the i-th observation or constraint.

[0046] From the above, we can see that in the data fusion module, the introduction of variance-based dynamic weight allocation and factor graph optimization algorithm can significantly improve the accuracy and consistency of fused data; by dynamically adjusting the weights of each sensor, the system can optimize the data fusion strategy in real time according to the measurement variance of each sensor, thereby ensuring that the fusion result is more accurate; at the same time, the application of the factor graph optimization algorithm further globally optimizes the multimodal sensor data, effectively eliminating the inconsistency between the data, and ultimately obtaining more accurate and reliable data fusion results, providing solid data support for the robot's autonomous navigation.

[0047] Furthermore, in the data fusion module, the attention mechanism is used to dynamically adjust the importance of different modal features. The formula is as follows:

[0048]

[0049] Among them, e i is the attention score of the i-th modality feature, α i is the attention weight.

[0050] From the above, we can see that in the data fusion module, using the attention mechanism to dynamically adjust the importance of different modal features can enhance the system's ability to capture key information and improve the efficiency of data fusion; by calculating the attention score of each modal feature, the system can automatically identify and focus on those information that have an important impact on environmental perception and navigation decisions, thereby achieving effective integration of different modal features; this dynamic adjustment mechanism enables the system to respond more flexibly to complex and changing environments, ensure the accuracy and real-time performance of data fusion, and thus improve the robot's autonomous navigation performance.

[0051] Furthermore, in the autonomous navigation module, a deep reinforcement learning algorithm is used to train the robot to perform optimal path planning. The formula of the deep reinforcement learning algorithm is as follows:

[0052]

[0053] Among them, π * represents the optimal strategy, π is a strategy in the strategy space, E π represents the expectation under strategy π, γ is the discount factor, r(s t ,a t ) is in state s t Take action a t The rewards received.

[0054] From the above, we can see that in the autonomous navigation module, using deep reinforcement learning algorithms to train robots to perform optimal path planning can significantly improve the robot's navigation efficiency and decision-making accuracy in complex environments; through deep reinforcement learning, the robot can learn the strategy of taking optimal actions under different states, so that it can quickly find and execute the optimal path planning during the actual navigation process; this learning mechanism enables the robot to adapt to changing environmental conditions, flexibly respond to various obstacles and uncertain factors, and ensure the smooth completion of navigation tasks.

[0055] Furthermore, in the autonomous navigation module, the navigation decision is to make a more efficient decision using the Monte Carlo tree search algorithm, and the formula of the Monte Carlo tree search algorithm is as follows:

[0056]

[0057] Among them, UCT(s,a) represents the upper confidence bound (Upper ConfidenceBound for Trees) value of selecting action a in state s, and a * is the optimal action to choose.

[0058] From the above, we can see that in the autonomous navigation module, the use of the Monte Carlo tree search algorithm for navigation decisions can significantly improve the efficiency and accuracy of decision-making; the Monte Carlo tree search algorithm can evaluate the pros and cons of different decision-making plans by simulating possible future paths and results, thereby selecting the optimal action; the algorithm can quickly find an approximate optimal solution within a limited time, and is particularly suitable for complex and uncertain navigation environments; through the Monte Carlo tree search algorithm, the robot can quickly make efficient and accurate decisions when faced with multiple possible navigation paths, ensuring the smooth progress of the navigation task.

[0059] Furthermore, the feedback adjustment module can dynamically adjust the use and fusion strategy of the sensor according to environmental changes to ensure efficient and stable operation in various scenarios. The dynamic adjustment adopts an adaptive weight adjustment algorithm. The formula of the adaptive weight adjustment algorithm is as follows:

[0060]

[0061] Among them, Δw i is the weight adjustment calculated based on the feedback.

[0062] From the above, we can see that in the feedback regulation module, the adaptive weight adjustment algorithm is used to dynamically adjust the use and fusion strategy of sensors according to environmental changes, which can ensure that the robot can maintain efficient and stable operation in various situations; by real-time monitoring of feedback information during the navigation process, the system can flexibly adjust the weight and fusion strategy of each sensor to adapt to different environmental conditions and navigation requirements; this dynamic adjustment mechanism enables the robot to better cope with complex and changing environmental challenges, maintain continuous stability of navigation performance, and thus improve the overall autonomous navigation capability and adaptability.

[0063] Working principle: The system obtains environmental information by integrating multiple sensors of different modes, and uses the data fusion module to process the sensor data to obtain comprehensive environmental perception information. The autonomous navigation module makes path planning and navigation decisions based on this information, while the feedback adjustment module dynamically adjusts the use of sensors and fusion strategies according to real-time feedback during the navigation process, ensuring that the system can maintain efficient and stable operation in various situations.

[0064] Furthermore, the robot autonomous navigation system based on multimodal sensor fusion detection of the embodiment is compared with the robot autonomous navigation system in the prior art (comparative example), and the following table is obtained:

[0065]

[0066]

[0067] It can be seen from the above table that the robot autonomous navigation system based on multimodal sensor fusion detection of this embodiment is superior to the robot autonomous navigation system in the prior art in terms of sensor usage, data fusion, obstacle detection and tracking, path planning and navigation decision-making, feedback adjustment, navigation accuracy and adaptability, and the system stability and reliability are also enhanced.

[0068] The embodiment of the present application provides an electronic device, which is applicable to the above-mentioned robot autonomous navigation system based on multimodal sensor fusion detection, including:

[0069] Memory, used to protect computer programs and data;

[0070] Processor, used to run system programs.

[0071] An embodiment of the present application provides a computer storage medium, which is suitable for the above-mentioned robot autonomous navigation system based on multimodal sensor fusion detection, and performs hierarchical confidentiality management on the above-mentioned system and data in accordance with confidentiality management requirements.

[0072] Those skilled in the art will appreciate that the embodiments of the present application may be provided as a system or a computer program product. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0073] The present application is described with reference to the flowcharts and / or block diagrams of the devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0074] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0075] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0076] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0077] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0078] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0079] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, commodity or device including the elements.

[0080] The embodiments of the present invention are provided for the purpose of illustration and description. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations of the present invention. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A robot autonomous navigation system based on multimodal sensor fusion detection, characterized in that: include: Multiple sensors of different modalities to obtain environmental information; A data fusion module is used to receive data from multiple sensors and perform data fusion processing using a preset perception model; The autonomous navigation module performs path planning and navigation decisions based on the fused environmental perception information; The feedback adjustment module is used to adjust the fusion strategy and navigation strategy according to the real-time feedback during the navigation process.

2. A robot autonomous navigation system based on multimodal sensor fusion detection as claimed in claim 1, characterized in that: In the data fusion module, wavelet transform is used to decompose the sensor data at multiple scales before data fusion processing.

3. The robot autonomous navigation system based on multimodal sensor fusion detection as claimed in claim 1, characterized in that: The preset perception model is trained in the following manner: obtaining non-failed sensor data obtained by multiple sensors when they are not failed, randomly selecting one or more non-failed sensor data and converting them into failed sensor data obtained when the corresponding sensor fails, and training with failed sensor data and non-failed sensor data that are not randomly selected.

4. The robot autonomous navigation system based on multimodal sensor fusion detection as claimed in claim 1, characterized in that: In the data fusion module, the variance-based dynamic weight allocation is introduced; At the same time, a factor graph optimization algorithm is introduced to perform global optimization on multimodal sensor data, thereby obtaining more accurate and consistent data fusion results.

5. The robot autonomous navigation system based on multimodal sensor fusion detection as claimed in claim 1, characterized in that: In the data fusion module, the attention mechanism is used to dynamically adjust the importance of different modal features.

6. The robot autonomous navigation system based on multimodal sensor fusion detection as claimed in claim 1, characterized in that: In the autonomous navigation module, deep reinforcement learning algorithms are used to train the robot to perform optimal path planning.

7. The robot autonomous navigation system based on multimodal sensor fusion detection as claimed in claim 1, characterized in that: In the autonomous navigation module, the navigation decision is made using the Monte Carlo tree search algorithm for more efficient decision making.

8. The robot autonomous navigation system based on multimodal sensor fusion detection as claimed in claim 1, characterized in that: The feedback regulation module can dynamically adjust the use and fusion strategy of sensors according to environmental changes to ensure efficient and stable operation in various scenarios. Its dynamic adjustment adopts an adaptive weight adjustment algorithm.

9. A processor, characterized in that: A robot autonomous navigation system configured to execute a multimodal sensor fusion detection-based system according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the robot autonomous navigation system based on multimodal sensor fusion detection as described in any one of claims 1 to 8 is implemented.

Images