A robot navigation method and robot control device fusing multi-modal perception

Abstract

The application provides a robot navigation method and a robot control device fusing multi-modal perception, and relates to the technical field of robot intelligent navigation. After laser radar data and RGB-D video stream data collected by a target robot for a current working environment are acquired, a multi-modal environment feature diffusion fusion mechanism is used to complementarily represent multi-modal environment information of the laser radar data and the RGB-D video stream data, so as to improve the instant understanding ability of the robot working environment. Then, a sub-target recommendation algorithm is combined to provide effective action decision guidance for a robot action decision network, so as to reduce the search probability of invalid actions and optimize the rationality of robot navigation decision, thereby effectively improving the environmental adaptability, navigation success rate and navigation performance stability of the robot in a dynamic working scene, and meeting the efficient navigation requirements of the robot in a dynamic working environment.

Description

Technical Field

[0001] This application relates to the field of robot intelligent navigation technology, and more specifically, to a robot navigation method and robot control device that integrates multimodal perception. Background Technology

[0002] With the continuous development of science and technology, the application of robotics technology is becoming more and more widespread in various fields. Mobile robots are usually required to autonomously navigate to a designated location within a specified work environment to perform specific tasks (such as food delivery or delivery tasks). However, during the robot navigation process, dynamic obstacles (such as pedestrians, other mobile robots, temporary obstacles that have moved or been added) often suddenly appear in the corresponding work environment, interfering with the robot's movement and making it difficult for the corresponding mobile robot to quickly navigate to the designated location.

[0003] Currently, the mainstream robot navigation solutions in the industry employ map-based navigation methods. These primarily rely on laser-based SLAM (Simultaneous Localization and Mapping) algorithms (such as GMAPping and Cartographer) to build maps of the working environment. This is combined with AMCL (Adaptive Monte Carlo Localization) algorithms for robot localization. Then, during global path planning using classic path planning algorithms such as A* and Dijkstra, obstacle avoidance optimization is performed using DWA (Dynamic Window Approach) or MPC (Model Predictive Control) algorithms to ensure the robot can autonomously navigate to its designated location. However, it's worth noting that while this robot navigation solution performs stably in static working environments, its heavy reliance on pre-built maps makes it difficult to update the map quickly when facing complex dynamic environments. This can easily lead to localization drift, inaccurate global path planning, or frequent triggering of recovery mechanisms, impacting overall robot navigation efficiency and ultimately failing to meet the high-efficiency navigation requirements of robots in dynamic working environments. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a robot navigation method and robot control device that integrates multimodal perception. It can achieve complementary representation of multimodal environmental information through a multimodal environmental feature diffusion fusion mechanism, thereby improving the real-time understanding of the robot's working environment. Combined with a sub-target recommendation algorithm, it provides effective action decision guidance for the robot's action decision network, reduces the probability of searching for invalid actions, and optimizes the rationality of robot navigation decisions. This effectively improves the robot's environmental adaptability, navigation success rate, and navigation performance stability in dynamic working scenarios, meeting the high-efficiency navigation requirements of robots in dynamic working environments.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, this application provides a robot navigation method that integrates multimodal perception, the method comprising: Acquire the LiDAR data and RGB-D video stream data currently collected by the target robot in the current working environment; Pedestrian detection and tracking are performed on the RGB-D video stream data to obtain a two-dimensional pedestrian motion feature map, and the lidar data is processed into a two-dimensional rasterization to obtain a two-dimensional environmental occupancy feature map. The two-dimensional pedestrian motion feature map and the two-dimensional environmental occupancy feature map are fused by feature diffusion to obtain a fused diffusion feature map that is adapted to the current working environment; Based on the fused diffusion feature map and the target navigation position of the target robot, navigation sub-targets are recommended to obtain the optimal sub-target position of the target robot as it moves toward the target navigation position. The pre-stored action decision network is invoked to make robot action decisions based on the optimal sub-target position, the lidar data, and the fused diffusion feature map, so as to obtain the target robot's current expected movement action and drive the target robot to move towards the target navigation position.

[0006] In an optional implementation, the two-dimensional pedestrian motion feature map includes a two-dimensional lateral pedestrian velocity feature map and a two-dimensional longitudinal pedestrian velocity feature map. The step of performing feature diffusion fusion on the two-dimensional pedestrian motion feature map and the two-dimensional environmental occupancy feature map to obtain a fused diffusion feature map adapted to the current working environment includes: A two-dimensional horizontal diffusion convolution kernel is called to perform feature diffusion convolution operation on the two-dimensional horizontal pedestrian speed feature map to obtain the corresponding two-dimensional horizontal pedestrian speed diffusion feature map. A two-dimensional vertical diffusion convolution kernel is called to perform feature diffusion convolution operation on the two-dimensional vertical pedestrian speed feature map to obtain the corresponding two-dimensional vertical pedestrian speed diffusion feature map. The two-dimensional horizontal pedestrian velocity diffusion feature map and the two-dimensional vertical pedestrian velocity diffusion feature map are superimposed to obtain the corresponding two-dimensional pedestrian motion diffusion feature map. The occupancy feature map is subjected to feature diffusion convolution operation by calling the occupancy feature diffusion convolution kernel to obtain the corresponding two-dimensional environment occupancy diffusion feature map; Based on the preset feature map superposition weights of the two-dimensional pedestrian movement diffusion feature map and the two-dimensional environment occupancy diffusion feature map, the two-dimensional pedestrian movement diffusion feature map and the two-dimensional environment occupancy diffusion feature map are weighted and fused to obtain the fused diffusion feature map.

[0007] In an optional implementation, the two-dimensional lateral diffusion convolution kernel includes a positive lateral diffusion convolution kernel and a negative lateral diffusion convolution kernel, and the two-dimensional longitudinal diffusion convolution kernel includes a positive longitudinal diffusion convolution kernel and a negative longitudinal diffusion convolution kernel. The step of performing feature diffusion convolution operations on each uniaxial pedestrian velocity feature map in the two-dimensional lateral pedestrian velocity feature map and the two-dimensional longitudinal pedestrian velocity feature map to obtain the corresponding uniaxial pedestrian velocity diffusion feature map includes: Negative and positive feature filtering are performed on the single-axis pedestrian velocity feature map to obtain non-negative and non-positive velocity feature maps that match the single-axis pedestrian velocity feature map. A positive axial diffusion convolution kernel is used to perform feature convolution operation on the non-negative velocity feature map to obtain the corresponding first velocity diffusion feature map, and a negative axial diffusion convolution kernel is used to perform feature convolution operation on the non-positive velocity feature map to obtain the corresponding second velocity diffusion feature map; wherein, the positive axial diffusion convolution kernel and the negative axial diffusion convolution kernel corresponding to the two-dimensional horizontal pedestrian velocity feature map are positive horizontal diffusion convolution kernel and negative horizontal diffusion convolution kernel, respectively, and the positive axial diffusion convolution kernel and the negative axial diffusion convolution kernel corresponding to the two-dimensional vertical pedestrian velocity feature map are positive vertical diffusion convolution kernel and negative vertical diffusion convolution kernel, respectively; The first velocity diffusion feature map and the second velocity diffusion feature map are superimposed to obtain the corresponding single-axis pedestrian velocity diffusion feature map; wherein, the single-axis pedestrian velocity diffusion feature map corresponding to the two-dimensional horizontal pedestrian velocity feature map is a two-dimensional horizontal pedestrian velocity diffusion feature map, and the single-axis pedestrian velocity diffusion feature map corresponding to the two-dimensional vertical pedestrian velocity feature map is a two-dimensional vertical pedestrian velocity diffusion feature map.

[0008] In an optional implementation, the step of recommending navigation sub-targets based on the fused diffusion feature map and the target robot's target navigation position to obtain the optimal sub-target position of the target robot as it moves towards the target navigation position includes: The current moving position and target navigation position of the target robot are mapped onto the two-dimensional coordinate system where the fused diffusion feature map is located to obtain the corresponding starting search position and target search position; Based on the initial search position, a breadth-first search algorithm is used to search for passable positions in the fused diffusion feature map to obtain multiple candidate passable positions that meet the preset search conditions. For each of the multiple candidate travel locations, a comprehensive path suitability score is calculated based on the target search location and the neighborhood feature distribution data of the candidate travel location in the fused diffusion feature map. The comprehensive path suitability scores of the multiple candidate passage locations are sorted in descending order, and the physical mapping position of the candidate passage location with the largest comprehensive path suitability score in the current working environment is selected as the optimal sub-target location.

[0009] In an optional implementation, for each candidate travel location, the step of calculating the comprehensive path suitability score of the candidate travel location based on the target search location and the neighborhood feature distribution data of the candidate travel location at the fused diffusion feature map includes: Based on the neighborhood feature distribution data of the candidate passage location, calculate the neighborhood passage suitability score of the candidate passage location at the fused diffusion feature map; Calculate the Euclidean distance between the target search location and the candidate travel location to obtain the global navigation suitability score of the candidate travel location at the fused diffusion feature map; wherein, the global navigation suitability score is the negative of the corresponding Euclidean distance; Based on the preset score influence weights of the neighborhood accessibility score and the global navigation suitability score, a weighted summation operation is performed on the neighborhood accessibility score and the global navigation suitability score to obtain the comprehensive accessibility score of the candidate access location.

[0010] In an optional implementation, the neighborhood accessibility score of any candidate access location at the fused diffusion feature map is calculated using the following formula: ; in, The x-coordinate value used to represent the candidate passage location in the fused diffusion feature map. The vertical coordinate value used to represent the candidate passage location in the fusion diffusion feature map. The neighborhood accessibility score used to represent the candidate access location Used to represent odd-numbered radii in a neighborhood. Used to represent the floor function. Used to indicate the corresponding x-coordinate And the ordinate is The target feature point is located at the actual feature value in the fusion diffusion feature map.

[0011] In an optional implementation, the method further includes: The action evaluation network, which belongs to the same Actor-Critic reinforcement learning architecture as the action decision network, is invoked to predict environmental features based on the expected movement action and the fused diffusion feature map, thereby obtaining a predicted environmental feature map under the execution of the expected movement action. The action evaluation network is invoked to evaluate the action value of the expected movement action based on the fused diffusion feature map and the predicted environment feature map, so as to obtain the actual action value of the expected movement action. The action decision network is optimized according to the actual action value of the desired movement.

[0012] In an optional implementation, before performing the step of recommending navigation sub-targets based on the fused diffusion feature map and the target robot's target navigation position, the method further includes: Based on the fused diffusion feature map, a forward passage risk assessment is performed on the current working environment to obtain the corresponding forward risk assessment value; Detect whether the forward risk assessment value exceeds a preset risk assessment threshold; If the forward risk assessment value is not detected to exceed the preset risk assessment threshold, the step of recommending navigation sub-targets based on the fused diffusion feature map and the target robot's target navigation position is executed; otherwise, the current forward direction of the target robot is adjusted, and the process jumps to the step of obtaining the LiDAR data and RGB-D video stream data collected by the target robot for the current working environment to continue execution.

[0013] In an optional implementation, the forward traffic risk assessment process based on the fused diffusion feature map is represented by the following formula: ; in, Used to represent the fusion diffusion feature map Used to represent the forward risk assessment value determined based on the fusion diffusion feature map. Used to represent the minimum safe column count threshold. The total number of feature columns used to represent the fusion diffusion feature map This is used to represent the total number of feature rows in the fused diffusion feature map. Used to indicate the height of the access risk assessment area. The first one used to represent the fusion diffusion feature map Line 1 The actual feature value at the feature point location of the column. Used to represent the floor function.

[0014] Secondly, this application provides a robot control device, which is communicatively connected to a target robot, the target robot being equipped with an RGB-D camera and a lidar; The robot control device includes a processor and a memory. The memory stores a computer program that can be executed by the processor. The processor can execute the computer program to implement the robot navigation method that integrates multimodal perception as described in any of the foregoing embodiments.

[0015] In this case, the beneficial effects of the embodiments of this application may include the following: After acquiring LiDAR data and RGB-D video stream data collected by the target robot in the current working environment, this application performs pedestrian detection and tracking on the RGB-D video stream data to obtain a two-dimensional pedestrian motion feature map, and performs two-dimensional rasterization processing on the LiDAR data to obtain a two-dimensional environment occupancy feature map. Then, feature diffusion fusion is performed on the two-dimensional pedestrian motion feature map and the two-dimensional environment occupancy feature map to achieve complementary representation of multimodal environmental information, resulting in a fused diffusion feature map adapted to the current working environment, improving the real-time understanding of the robot's working environment. Then, based on the fused diffusion feature map and the target robot's target navigation position, navigation sub-target recommendation is performed to obtain the optimal sub-target position of the target robot in the process of moving to the target navigation position. Then, the action decision network is called to make robot action decisions based on the optimal sub-target position, LiDAR data, and fused diffusion feature map, so as to provide effective action decision guidance for the robot action decision network through complementary representation of multimodal environmental information and sub-target recommendation algorithm, reduce the search probability of invalid actions, and thus quickly obtain the expected movement action that the target robot needs to perform and that has strong rationality, to drive the target robot to move to the target navigation position. Therefore, this application effectively improves the robot's environmental adaptability, navigation success rate, and navigation performance stability in dynamic operation scenarios by organically combining a multimodal environmental feature diffusion fusion mechanism and a sub-target recommendation algorithm, and can meet the robot's high-efficiency navigation requirements in dynamic operation environments.

[0016] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the composition of the robot control device provided in the embodiments of this application; Figure 2 One of the flowcharts of the robot navigation method integrating multimodal perception provided in the embodiments of this application; Figure 3 for Figure 2 A flowchart illustrating the sub-steps included in step S230; Figure 4 This is a schematic diagram of feature diffusion fusion between a two-dimensional pedestrian motion feature map and a two-dimensional environmental occupancy feature map provided in an embodiment of this application. Figure 5 for Figure 2 A flowchart illustrating the sub-steps included in step S240; Figure 6 A second schematic flowchart illustrating the robot navigation method integrating multimodal perception provided in this application embodiment; Figure 7 This is the third flowchart illustrating the robot navigation method integrating multimodal perception provided in the embodiments of this application.

[0019] Icons: 10-Robot control equipment; 11-Memory; 12-Processor; 13-Communication unit. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0021] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0022] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0023] In the description of this application, it should be understood that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product is in use, or the orientation or positional relationship commonly understood by those skilled in the art. They are used only for the convenience of describing this application and simplifying the description, and are not intended to indicate or imply that the equipment or component referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0024] In the description of this application, it should also be noted that, unless otherwise expressly specified and limited, the terms "set up," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0025] Furthermore, it is understood in the description of this application that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Those skilled in the art will understand the specific meaning of the above terms in this application based on the specific circumstances.

[0026] Through diligent research, the applicant discovered that existing robot navigation solutions already employ machine learning-based navigation methods, which can directly learn end-to-end navigation strategies from sensor data using Deep Reinforcement Learning (DRL) or Imitation Learning (IL) techniques. While machine learning navigation schemes based on single sensors (such as LiDAR or RGB cameras) achieve autonomous navigation strategy learning without relying on pre-built maps through a data-driven approach, their single-modal perception mechanism severely limits their environmental understanding capabilities. This is particularly problematic in dynamic environments where they struggle to accurately perceive and model pedestrian movement. (For example, while LiDAR-based machine learning navigation schemes can detect obstacle locations, LiDAR primarily provides geometric information and cannot effectively distinguish between dynamic pedestrians and static obstacles. Consequently, the corresponding navigation strategy treats pedestrians as ordinary static obstacles, ignoring their movement trends, which can lead to unreasonable path planning or increased collision risks. Conversely, while pure vision-based (i.e., RGB camera) machine learning navigation schemes can identify pedestrians, their field of view is limited to the installation angle of the RGB camera lens, making it difficult to fully perceive environmental changes. Significant blind spots in visual perception affect the robot's autonomous obstacle avoidance decisions.) Machine learning navigation schemes based on multiple sensors (such as LiDAR and RGB-D cameras) utilize CNNs (Convolutional Neural Networks). Multimodal feature splicing and fusion are performed on the network, and navigation path exploration is carried out in the high-dimensional state space through random sampling strategy. However, due to the limitation of the feature fusion mechanism, the complementarity of different modal information is not fully utilized, and the spatial consistency between multiple sensors cannot be effectively modeled. As a result, the robot navigation strategy still lacks stability in some key scenarios (e.g., narrow passages or dynamic obstacle interference). At the same time, because the navigation path exploration is carried out by random sampling strategy, there is a lack of clear navigation guidance mechanism (information), which leads to a large redundancy in the navigation path exploration process (i.e., a lot of time and computing resources are consumed to explore invalid movement navigation actions). The overall navigation performance is not stable and cannot meet the robot's efficient navigation requirements in dynamic working environments.

[0027] In this context, to address the aforementioned issues, this application provides a robot navigation method and robot control device that integrates multimodal perception. This method utilizes a multimodal environmental feature diffusion fusion mechanism to achieve complementary representation of multimodal environmental information (i.e., fully leveraging the complementarity between multimodal environmental information), thereby enhancing the robot's real-time understanding of its operating environment. This ensures that the robot's autonomous navigation operations can effectively adapt to dynamic operating environments. Furthermore, by combining a sub-target recommendation algorithm with the robot's action decision network, it provides effective action decision guidance, reducing the probability of searching for invalid actions and optimizing the rationality of robot navigation decisions. This effectively improves the robot's environmental adaptability, navigation success rate, and navigation performance stability in dynamic operating scenarios, meeting the high-efficiency navigation requirements of robots in dynamic operating environments.

[0028] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0029] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the robot control device 10 provided in this application embodiment. In this application embodiment, the robot control device 10 can communicate with the controlled mobile robot (i.e., the target robot), and based on the lidar data and RGB-D video stream data (including multiple consecutive frames of RGB-D environmental images) collected by the target robot in its own forward direction for the current working environment, through the organic combination of a multimodal environmental feature diffusion fusion mechanism and a sub-target recommendation algorithm, effectively improves the target robot's environmental adaptability, navigation success rate, and navigation performance stability in dynamic working scenarios, enabling the target robot to effectively meet the robot's efficient navigation requirements in dynamic working environments under the navigation scheduling function of the robot control device 10. The target robot is equipped with an RGB-D camera and a lidar. The RGB-D camera and the lidar are used for multimodal environmental perception of the actual working environment that the target robot needs to face in its own forward direction, wherein the RGB-D camera is used to collect RGB-D environmental images.

[0030] In this embodiment, the robot control device 10 may be a computer device independent of the target robot. The computer device may be, but is not limited to, a personal computer, a laptop, a tablet computer, a server, etc.; the robot control device 10 may also be integrated with the target robot.

[0031] In this embodiment, the robot control device 10 may include a memory 11, a processor 12, and a communication unit 13. The memory 11, the processor 12, and the communication unit 13 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines.

[0032] In this embodiment, the memory 11 may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory 11 is used to store computer programs, and the processor 12 can execute the computer programs accordingly after receiving execution instructions.

[0033] Meanwhile, the memory 11 is also used to store action evaluation networks (corresponding to the Critic network) and action decision networks (corresponding to the Actor network) belonging to the same Actor-Critic reinforcement learning architecture. The action decision network is used to generate robot actions (i.e., robot navigation strategies) corresponding to the robot's autonomous navigation function. The action evaluation network is used to evaluate the actual value of the robot actions generated by the action decision network, and to optimize the action decision network based on the evaluated actual value, so as to improve the stability and generalization of the robot navigation strategies generated by the action decision network.

[0034] In this embodiment, the processor 12 can be an integrated circuit chip with signal processing capabilities. The processor 12 can be a general-purpose processor, including at least one of a central processing unit (CPU), graphics processing unit (GPU), network processor (NP), digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0035] In this embodiment, the communication unit 13 is used to establish a communication connection between the robot control device 10 and other electronic devices via a network, and to send and receive data via the network, wherein the network includes wired communication networks and wireless communication networks. For example, the robot control device 10 can obtain the target navigation location that the target robot needs to go to through the communication unit 13, and drive the target robot to move to the target navigation location.

[0036] In this embodiment, the robot control device 10 can pre-store a specific computer program related to the robot navigation function of fusing multimodal perception in the memory 11. By driving the processor 12 to execute the specific computer program, the device utilizes the multimodal environmental feature diffusion fusion mechanism to achieve complementary representation of multimodal environmental information, thereby improving the robot's real-time understanding of the working environment. Combined with the sub-target recommendation algorithm, the device provides effective action decision guidance for the robot action decision network, reduces the probability of searching for invalid actions, and optimizes the rationality of robot navigation decisions. This effectively improves the robot's environmental adaptability, navigation success rate, and navigation performance stability in dynamic working scenarios, meeting the robot's efficient navigation requirements in dynamic working environments.

[0037] Understandable, Figure 1 The block diagram shown is only a schematic diagram of one composition of the robot control device 10. The robot control device 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown. Figure 1 The components shown can be implemented using hardware, software, or a combination thereof.

[0038] In this application, to ensure that the robot control device 10 can effectively improve the robot's environmental adaptability, navigation success rate, and navigation performance stability in dynamic work scenarios through the organic combination of multimodal environmental feature diffusion fusion mechanism and sub-target recommendation algorithm, and meet the robot's efficient navigation requirements in dynamic work environments, this application provides a robot navigation method that integrates multimodal perception to achieve the aforementioned objectives. The robot navigation method that integrates multimodal perception provided in this application will be described in detail below.

[0039] Please refer to Figure 2 , Figure 2 This is one of the flowcharts illustrating the robot navigation method integrating multimodal perception provided in this application embodiment. In this application embodiment, Figure 2 The robot navigation method shown may include steps S210 to S250.

[0040] Step S210: Obtain the LiDAR data and RGB-D video stream data collected by the target robot in the current working environment.

[0041] In this embodiment, the lidar data and RGB-D video stream data acquired by the robot control device 10 at any time are matched with the working environment that the target robot needs to face in its own forward direction, and the acquisition timing of the lidar data and the RGB-D video stream data is consistent.

[0042] Step S220: Perform pedestrian detection and tracking on the RGB-D video stream data to obtain a two-dimensional pedestrian motion feature map, and perform two-dimensional rasterization processing on the LiDAR data to obtain a two-dimensional environmental occupancy feature map.

[0043] In this embodiment, the RGB-D video stream data includes multiple consecutive frames of RGB-D environmental images. The robot control device 10 can input these multiple consecutive frames of RGB-D environmental images into the YOLO (You Only Look Once) network to perform pedestrian detection on each frame of the RGB-D environmental image. This yields the bounding box coordinates and pedestrian ID confidence scores for each pedestrian in each frame of the RGB-D environmental image. Combined with depth information projection, the actual position information of the corresponding pedestrian within the perception field of view of the RGB-D camera is obtained. Then, the robot control device 10 uses DeepSORT based on the actual position information of each pedestrian within the perception field of view of the RGB-D camera. The algorithm performs pedestrian tracking on the continuous multi-frame RGB-D environmental images to determine the two-dimensional velocity information of each pedestrian in the horizontal plane (including the velocity information of the corresponding pedestrian on the two-dimensional horizontal axis and the velocity information of the corresponding pedestrian on the two-dimensional vertical axis). Then, by mapping the actual position information and two-dimensional velocity information of all pedestrians onto the horizontal plane, a corresponding two-dimensional pedestrian motion feature map is obtained. The two-dimensional pedestrian motion feature map records the position distribution information of all pedestrians in the two-dimensional horizontal plane and the two-dimensional velocity information of all pedestrians in the two-dimensional horizontal plane. Thus, the two-dimensional pedestrian motion feature map can be decomposed into a two-dimensional horizontal pedestrian velocity feature map and a two-dimensional vertical pedestrian velocity feature map: the two-dimensional horizontal pedestrian velocity feature map records the position distribution information of all pedestrians in the two-dimensional horizontal plane and the component velocity information of all pedestrians in the two-dimensional horizontal axis direction; the two-dimensional vertical pedestrian velocity feature map records the position distribution information of all pedestrians in the two-dimensional horizontal plane and the component velocity information of all pedestrians in the two-dimensional vertical axis direction.

[0044] In this embodiment, the lidar data consists of 180 sets of detection data, used to represent the detection distance information within a 180° observation range along the forward direction of the target robot. Each set of detection data includes the detection angle value and actual detection distance value of the corresponding lidar. The robot control device 10 can perform two-dimensional rasterization processing on the lidar data to convert these 180 sets of detection data into discrete grid points in a two-dimensional plane, obtaining a corresponding two-dimensional environmental occupancy feature map. The two-dimensional environmental occupancy feature map is used to describe the distribution of existing obstacles (including dynamic pedestrians, dynamic obstacles, static obstacles, etc.) in the current working environment within the two-dimensional detection plane.

[0045] Step S230: Perform feature diffusion fusion on the two-dimensional pedestrian motion feature map and the two-dimensional environmental occupancy feature map to obtain a fused diffusion feature map that is adapted to the current working environment.

[0046] In this embodiment, the robot control device 10 can utilize a multimodal environmental feature diffusion fusion mechanism to perform feature diffusion fusion on the lidar data and the RGB-D video stream data, so as to give full play to the complementarity between multimodal environmental information, improve the real-time understanding of the robot's working environment, and obtain a fused diffusion feature map that is adapted to the current working environment faced by the target robot in its own forward direction.

[0047] Optionally, please refer to the following: Figure 3 and Figure 4 ,in Figure 3 yes Figure 2 A flowchart illustrating the sub-steps included in step S230. Figure 4 This is a schematic diagram of feature diffusion fusion between a two-dimensional pedestrian motion feature map and a two-dimensional environment occupancy feature map provided in this application embodiment. In this application embodiment, step S230 may include sub-steps S231 to S234 to achieve complementary integration of data from different sensors, providing a more comprehensive environmental understanding for robot autonomous navigation. This enables the corresponding fused diffusion feature map to not only accurately reflect the pedestrian's movement trend but also effectively represent the obstacle occupancy status of the current working environment, providing crucial spatial-dynamic information support for subsequent navigation strategy decision-making.

[0048] Sub-step S231: Call the two-dimensional horizontal diffusion convolution kernel to perform feature diffusion convolution operation on the two-dimensional horizontal pedestrian speed feature map to obtain the corresponding two-dimensional horizontal pedestrian speed diffusion feature map, and call the two-dimensional vertical diffusion convolution kernel to perform feature diffusion convolution operation on the two-dimensional vertical pedestrian speed feature map to obtain the corresponding two-dimensional vertical pedestrian speed diffusion feature map.

[0049] In this embodiment, the two-dimensional lateral diffusion convolution kernel may include a positive lateral diffusion convolution kernel (i.e., Figure 4 The middle diffusing convolution kernel located in the upper right corner) and the negative lateral diffusing convolution kernel (i.e. Figure 4 The two-dimensional longitudinal diffusion convolution kernel (the diffusion convolution kernel adjacent to the upper right diffusion convolution kernel) may include a positive longitudinal diffusion convolution kernel (i.e., Figure 4 The middle and upper left diffuse convolution kernels) and the negative longitudinal diffuse convolution kernels (i.e. Figure 4The four diffusion convolution kernels (adjacent to the upper left diffusion convolution kernel) are used to simulate the movement trend of pedestrians in the current working environment by combining the two-dimensional pedestrian motion feature map. These four diffusion convolution kernels correspond to the feature diffusion directions of the two-dimensional horizontal axis (positive), the two-dimensional horizontal axis (negative), the two-dimensional vertical axis (positive), and the two-dimensional vertical axis (negative), respectively. The step of performing feature diffusion convolution operations on each single-axis pedestrian velocity feature map in the two-dimensional horizontal pedestrian velocity feature map and the two-dimensional vertical pedestrian velocity feature map to obtain the corresponding single-axis pedestrian velocity diffusion feature map includes: Negative and positive feature filtering are performed on the single-axis pedestrian velocity feature map to obtain non-negative and non-positive velocity feature maps that match the single-axis pedestrian velocity feature map. A positive axial diffusion convolution kernel is used to perform feature convolution operation on the non-negative velocity feature map to obtain the corresponding first velocity diffusion feature map, and a negative axial diffusion convolution kernel is used to perform feature convolution operation on the non-positive velocity feature map to obtain the corresponding second velocity diffusion feature map; wherein, the positive axial diffusion convolution kernel and the negative axial diffusion convolution kernel corresponding to the two-dimensional horizontal pedestrian velocity feature map are positive horizontal diffusion convolution kernel and negative horizontal diffusion convolution kernel, respectively, and the positive axial diffusion convolution kernel and the negative axial diffusion convolution kernel corresponding to the two-dimensional vertical pedestrian velocity feature map are positive vertical diffusion convolution kernel and negative vertical diffusion convolution kernel, respectively; The first velocity diffusion feature map and the second velocity diffusion feature map are superimposed to obtain the corresponding single-axis pedestrian velocity diffusion feature map; wherein, the single-axis pedestrian velocity diffusion feature map corresponding to the two-dimensional horizontal pedestrian velocity feature map is a two-dimensional horizontal pedestrian velocity diffusion feature map, and the single-axis pedestrian velocity diffusion feature map corresponding to the two-dimensional vertical pedestrian velocity feature map is a two-dimensional vertical pedestrian velocity diffusion feature map.

[0050] Sub-step S232 involves overlaying the two-dimensional horizontal pedestrian velocity diffusion feature map and the two-dimensional vertical pedestrian velocity diffusion feature map to obtain the corresponding two-dimensional pedestrian motion diffusion feature map.

[0051] In this embodiment, the two-dimensional pedestrian motion diffusion feature map can be adopted. Figure 4 "in "to indicate".

[0052] Sub-step S233: Call the occupancy feature diffusion convolution kernel to perform feature diffusion convolution operation on the two-dimensional environment occupancy feature map to obtain the corresponding two-dimensional environment occupancy diffusion feature map.

[0053] In this embodiment, the occupied feature diffusion convolution kernel is... Figure 4 The bottommost diffuse convolution kernel in the image, the two-dimensional environment occupancy diffuse feature map can be obtained using... Figure 4 "in "to indicate".

[0054] Sub-step S234: Based on the preset feature map superposition weights of the two-dimensional pedestrian motion diffusion feature map and the two-dimensional environment occupancy diffusion feature map, the two-dimensional pedestrian motion diffusion feature map and the two-dimensional environment occupancy diffusion feature map are weighted and fused to obtain a fused diffusion feature map.

[0055] In this embodiment, the preset feature map superposition weight of the two-dimensional pedestrian motion diffusion feature map is adopted. Figure 4 "in The preset feature map superposition weight of the two-dimensional environmental occupancy diffusion feature map is represented as follows: Figure 4 "in The fusion diffusion feature map is represented as follows: Figure 4 "in "to indicate".

[0056] Therefore, by executing the above sub-steps S231 to S234, this application can achieve complementary integration of data from different sensors, providing a more comprehensive understanding of the environment for robot autonomous navigation. This enables the corresponding fused diffusion feature map to not only accurately reflect the movement trend of pedestrians, but also effectively represent the obstacle occupancy status of the current working environment, providing key spatial-dynamic information support for subsequent navigation strategy decision-making.

[0057] Step S240: Based on the fused diffusion feature map and the target robot's target navigation position, recommend navigation sub-targets to obtain the optimal sub-target position of the target robot as it moves towards the target navigation position.

[0058] In this embodiment, when the robot control device 10 obtains the fusion diffusion feature map of the current working environment, it can use a sub-target recommendation algorithm to recommend sub-target positions based on the fusion diffusion feature map and the target navigation position, so as to determine the optimal sub-target position that is most likely to help the target robot effectively avoid obstacles and approach the target navigation position.

[0059] Alternatively, please refer to Figure 5 , Figure 5 yes Figure 2 The flowchart of step S240 includes the sub-steps. In this embodiment, step S240 may include sub-steps S241 to S244 to provide effective motion decision guidance to the robot motion decision network, improve the efficiency of the robot motion decision network in outputting reasonable navigation strategies, and reduce the probability of searching for invalid actions.

[0060] Sub-step S241: Map the current moving position and target navigation position of the target robot to the two-dimensional coordinate system where the fused diffusion feature map is located, respectively, to obtain the corresponding starting search position and target search position.

[0061] In this embodiment, the starting search position is the two-dimensional mapping position of the current moving position in the two-dimensional coordinate system of the fusion diffusion feature map, and the target search position is the two-dimensional mapping position of the target navigation position in the two-dimensional coordinate system of the fusion diffusion feature map.

[0062] Sub-step S242: Based on the initial search position, a breadth-first search algorithm is used to search for passable positions in the fused diffusion feature map to obtain multiple candidate passable positions that meet the preset search conditions.

[0063] The preset search conditions may include, but are not limited to: the actual search distance from the corresponding candidate passable position to the starting search position is less than or equal to the maximum preset search distance; the number of passable search positions traversed from the corresponding candidate passable position to the starting search position is less than or equal to the maximum preset search position number; and the total search time from the corresponding candidate passable position to the starting search position is less than or equal to the maximum preset search time.

[0064] Sub-step S243: For each candidate passable location among multiple candidate passable locations, calculate the comprehensive path suitability score of the candidate passable location based on the target search location and the neighborhood feature distribution data of the candidate passable location at the fusion diffusion feature map.

[0065] In this embodiment, the neighborhood feature distribution data of any candidate travel location includes the actual feature values ​​of multiple neighborhood feature points distributed around the corresponding candidate travel location in the fused diffusion feature map. Specifically, the step of calculating the comprehensive path suitability score for each candidate travel location based on the target search location and the neighborhood feature distribution data of that candidate travel location in the fused diffusion feature map includes: Based on the neighborhood feature distribution data of the candidate passage location, calculate the neighborhood passage suitability score of the candidate passage location at the fused diffusion feature map; Calculate the Euclidean distance between the target search location and the candidate travel location to obtain the global navigation suitability score of the candidate travel location at the fused diffusion feature map; wherein, the global navigation suitability score is the negative of the corresponding Euclidean distance; Based on the preset score influence weights of the neighborhood accessibility score and the global navigation suitability score, a weighted summation operation is performed on the neighborhood accessibility score and the global navigation suitability score to obtain the comprehensive accessibility score of the candidate access location.

[0066] In this process, the neighborhood mobility suitability score of any candidate passage location at the fused diffusion feature map is calculated using the following formula: ; in, The x-coordinate value used to represent the candidate passage location in the fused diffusion feature map. The vertical coordinate value used to represent the candidate passage location in the fusion diffusion feature map. The neighborhood accessibility score used to represent the candidate access location Used to represent odd-numbered radii in a neighborhood. Used to represent the floor function. Used to indicate the corresponding x-coordinate And the ordinate is The target feature point is located at the actual feature value in the fusion diffusion feature map.

[0067] Sub-step S244: Sort the comprehensive path suitability scores of multiple candidate passage locations in descending order, and select the physical mapping position of the candidate passage location with the largest comprehensive path suitability score in the current working environment as the optimal sub-target position.

[0068] Therefore, by executing the above sub-steps S241 to S244, this application can facilitate the provision of effective motion decision guidance to the robot motion decision network, improve the efficiency of the robot motion decision network in outputting reasonable navigation strategies, and reduce the probability of searching for invalid actions.

[0069] Step S250: The pre-stored action decision network is invoked to make robot action decisions based on the optimal sub-target position, LiDAR data and fused diffusion feature map, so as to obtain the target robot's current expected movement action and drive the target robot to move towards the target navigation position.

[0070] In this embodiment, the action decision network can be composed of a Vision Transformer network architecture and a Transformer network architecture. The robot control device 10 can, based on the LiDAR data and fused diffusion feature map of the current working environment, use the optimal sub-target position that is most likely to help the target robot effectively avoid obstacles and approach the target navigation position as action decision guidance information. It then calls the action decision network to make robot action decisions, thereby quickly obtaining the desired movement action that the target robot needs to execute and that has strong rationality. This drives the target robot to continue moving towards the target navigation position, reducing the probability of invalid action searches during the robot navigation strategy decision-making process, achieving a rationality optimization effect for robot navigation decisions, and ensuring that the corresponding target robot can have good environmental adaptability, navigation success rate, and navigation performance stability in dynamic working scenarios, meeting the high-efficiency navigation requirements of robots in dynamic working environments.

[0071] Therefore, by executing the above steps S210 to S250, this application can effectively improve the robot's environmental adaptability, navigation success rate, and navigation performance stability in dynamic work scenarios through the organic combination of multimodal environmental feature diffusion fusion mechanism and sub-target recommendation algorithm, thus meeting the robot's high-efficiency navigation requirements in dynamic work environments.

[0072] Alternatively, please refer to Figure 6 , Figure 6 This is the second flowchart illustrating the robot navigation method integrating multimodal perception provided in this application embodiment. In this application embodiment, with... Figure 2 The robot navigation method shown Figure 6 The robot navigation method shown may also include steps S260 to S280 between steps S230 and S240, so as to avoid the target robot directly entering the high obstacle density area, and can bypass the high obstacle density area and change the direction of travel to move to the target navigation position, thus avoiding unnecessary robot navigation time loss.

[0073] Step S260: Based on the fused diffusion feature map, perform a forward passage risk assessment on the current working environment to obtain the corresponding forward risk assessment value.

[0074] In this embodiment, after executing step S230, the robot control device 10 can analyze the fused diffusion feature map to assess the obstacle density in the adjacent area near the current movement position within the current working environment, thereby obtaining a corresponding forward risk assessment value. The larger the forward risk assessment value, the higher the obstacle density in the corresponding adjacent area. The forward passage risk assessment process based on the fused diffusion feature map is represented by the following formula: ; in, Used to represent the fusion diffusion feature map Used to represent the forward risk assessment value determined based on the fusion diffusion feature map. Used to represent the minimum safe column count threshold. The total number of feature columns used to represent the fusion diffusion feature map This is used to represent the total number of feature rows in the fused diffusion feature map. This is used to indicate the height of the corresponding traffic risk assessment area in the fusion diffusion feature map for the adjacent area. The first one used to represent the fusion diffusion feature map Line 1 The actual feature value at the feature point location of the column. Used to represent the floor function.

[0075] Step S270: Detect whether the risk assessment value ahead exceeds the preset risk assessment threshold.

[0076] In this embodiment, when the forward risk assessment value exceeds the preset risk assessment threshold, it indicates that the obstacle density in the adjacent area near the current movement position in the current working environment is too high, and the target robot should not directly enter the current working environment. At this time, the robot control device 10 will execute step S280. When the forward risk assessment value does not exceed the preset risk assessment threshold, it indicates that the obstacle density in the adjacent area near the current movement position in the current working environment is low, and the target robot can directly enter the current working environment. At this time, the robot control device 10 will continue to execute step S240.

[0077] Step S280: Adjust the current forward direction of the target robot.

[0078] In this embodiment, when the robot control device 10 detects that the forward risk assessment value exceeds the preset risk assessment threshold, it will adjust the current forward direction of the target robot accordingly, so that the target robot will subsequently bypass the adjacent area (i.e., the obstacle density area) and change its forward direction to move to the target navigation position, avoiding unnecessary robot navigation time loss. Then, the robot control device 10 will jump to the above step S210 to continue execution, so as to ensure that the target robot can move to the target navigation position as soon as possible. In one embodiment of this invention, the robot control device 10 can adjust the current forward direction of the target robot to allow it to return to a relatively open target movement area behind it, so that the target robot can quickly re-navigate to the target navigation position. Specifically, the robot control device 10 can temporarily use the target movement area as the target navigation position in steps S240 and S250, and execute steps S240 and S250 accordingly to guide the target robot back to the target movement area. Then, by resetting the target navigation position and continuing to execute step S210, the device ensures that the target robot can re-navigate from the target movement area to the originally set target navigation position.

[0079] Therefore, through the above steps S260 to S280, this application can avoid the target robot directly entering the high obstacle density area, and can bypass the high obstacle density area and change the direction of travel to move to the target navigation position, thus avoiding unnecessary robot navigation time loss.

[0080] Alternatively, please refer to Figure 7 , Figure 7 This is the third flowchart illustrating the robot navigation method integrating multimodal perception provided in this application embodiment. In this application embodiment, with... Figure 2 or Figure 6 Compared to the robot navigation methods shown, Figure 7 The robot navigation method shown may further include steps S310 to S330 after step S250, in order to introduce a prediction algorithm at the action evaluation network to effectively predict the working environment after the robot navigation strategy (i.e. the aforementioned expected movement action) is executed, so that the evaluation results given by the action evaluation network for the robot navigation strategy can more accurately reflect the real interaction between the target robot and the dynamic working environment, and ensure that the corresponding optimized action decision network has stronger stability and generalization in the robot navigation strategy generation function.

[0081] Step S310: Call the action evaluation network, which belongs to the same Actor-Critic reinforcement learning architecture as the action decision network, to predict the environment features based on the expected movement action and the fused diffusion feature map, and obtain the predicted environment feature map under the execution of the expected movement action.

[0082] In this embodiment, the motion evaluation network may incorporate a Stochastic Occupancy Grid Map Prediction (SOGMP) algorithm based on Variational Auto-Eencoder (VAE) to integrate functions such as robot motion compensation, dynamic object prediction, and future state uncertainty estimation to generate a preliminary observation of the working environment feature map (i.e., the predicted environment feature map) after the target robot has performed the desired movement.

[0083] Step S320: The action evaluation network is invoked to evaluate the action value of the expected movement action based on the fused diffusion feature map and the predicted environment feature map, so as to obtain the actual action value of the expected movement action.

[0084] In this embodiment, the motion evaluation network is composed of a Vision Transformer network architecture, a Transformer network architecture, and a Multi-Layer Perceptron (MLP) to ensure that the motion evaluation network can effectively evaluate the actual motion value that the expected movement motion brings to the dynamic working environment after execution. The actual motion value can be represented and output using a state-action value function.

[0085] Step S330: Optimize the action decision network according to the actual action value of the desired movement.

[0086] Therefore, by executing the above steps S310 to S330, this application can introduce a prediction algorithm into the action evaluation network to effectively predict the working environment after the robot navigation strategy (i.e., the aforementioned expected movement action) is executed. This allows the evaluation results given by the action evaluation network for the robot navigation strategy to more accurately reflect the real interaction between the target robot and the dynamic working environment, and ensures that the corresponding optimized action decision network has stronger stability and generalization in the robot navigation strategy generation function.

[0087] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0088] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part. If the various functions provided in this application are implemented in the form of software functional modules and sold or used as independent products, they can be stored in a storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause the robot control device 10 to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned readable storage medium includes: USB flash drive, mobile hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, and other media capable of storing program code.

[0089] The above descriptions are merely various embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A robot navigation method integrating multimodal perception, characterized in that, The method includes: Acquire LiDAR data and RGB-D video stream data collected by the target robot in the current working environment; Pedestrian detection and tracking are performed on the RGB-D video stream data to obtain a two-dimensional pedestrian motion feature map, and the LiDAR data is processed into a two-dimensional rasterization to obtain a two-dimensional environmental occupancy feature map. The two-dimensional pedestrian motion feature map and the two-dimensional environmental occupancy feature map are fused by feature diffusion to obtain a fused diffusion feature map that is adapted to the current working environment; Based on the fused diffusion feature map and the target navigation position of the target robot, navigation sub-targets are recommended to obtain the optimal sub-target position of the target robot as it moves toward the target navigation position. The pre-stored action decision network is invoked to make robot action decisions based on the optimal sub-target position, the lidar data, and the fused diffusion feature map, so as to obtain the target robot's current expected movement action and drive the target robot to move towards the target navigation position; The two-dimensional pedestrian motion feature map includes a two-dimensional lateral pedestrian velocity feature map and a two-dimensional longitudinal pedestrian velocity feature map. The step of performing feature diffusion fusion on the two-dimensional pedestrian motion feature map and the two-dimensional environmental occupancy feature map to obtain a fused diffusion feature map adapted to the current working environment includes: A two-dimensional horizontal diffusion convolution kernel is called to perform feature diffusion convolution operation on the two-dimensional horizontal pedestrian speed feature map to obtain the corresponding two-dimensional horizontal pedestrian speed diffusion feature map. A two-dimensional vertical diffusion convolution kernel is called to perform feature diffusion convolution operation on the two-dimensional vertical pedestrian speed feature map to obtain the corresponding two-dimensional vertical pedestrian speed diffusion feature map. The two-dimensional horizontal pedestrian velocity diffusion feature map and the two-dimensional vertical pedestrian velocity diffusion feature map are superimposed to obtain the corresponding two-dimensional pedestrian motion diffusion feature map. The occupancy feature map is subjected to feature diffusion convolution operation by calling the occupancy feature diffusion convolution kernel to obtain the corresponding two-dimensional environment occupancy diffusion feature map; Based on the preset feature map superposition weights of the two-dimensional pedestrian movement diffusion feature map and the two-dimensional environment occupancy diffusion feature map, the two-dimensional pedestrian movement diffusion feature map and the two-dimensional environment occupancy diffusion feature map are weighted and fused to obtain the fused diffusion feature map.

2. The method according to claim 1, characterized in that, The two-dimensional lateral diffusion convolution kernel includes a positive lateral diffusion convolution kernel and a negative lateral diffusion convolution kernel, and the two-dimensional longitudinal diffusion convolution kernel includes a positive longitudinal diffusion convolution kernel and a negative longitudinal diffusion convolution kernel. The step of performing feature diffusion convolution operations on each uniaxial pedestrian velocity feature map in the two-dimensional lateral pedestrian velocity feature map and the two-dimensional longitudinal pedestrian velocity feature map to obtain the corresponding uniaxial pedestrian velocity diffusion feature map includes: Negative and positive feature filtering are performed on the single-axis pedestrian velocity feature map to obtain non-negative and non-positive velocity feature maps that match the single-axis pedestrian velocity feature map. A positive axial diffusion convolution kernel is used to perform feature convolution operation on the non-negative velocity feature map to obtain the corresponding first velocity diffusion feature map, and a negative axial diffusion convolution kernel is used to perform feature convolution operation on the non-positive velocity feature map to obtain the corresponding second velocity diffusion feature map; wherein, the positive axial diffusion convolution kernel and the negative axial diffusion convolution kernel corresponding to the two-dimensional horizontal pedestrian velocity feature map are positive horizontal diffusion convolution kernel and negative horizontal diffusion convolution kernel, respectively, and the positive axial diffusion convolution kernel and the negative axial diffusion convolution kernel corresponding to the two-dimensional vertical pedestrian velocity feature map are positive vertical diffusion convolution kernel and negative vertical diffusion convolution kernel, respectively; The first velocity diffusion feature map and the second velocity diffusion feature map are superimposed to obtain the corresponding single-axis pedestrian velocity diffusion feature map; wherein, the single-axis pedestrian velocity diffusion feature map corresponding to the two-dimensional horizontal pedestrian velocity feature map is a two-dimensional horizontal pedestrian velocity diffusion feature map, and the single-axis pedestrian velocity diffusion feature map corresponding to the two-dimensional vertical pedestrian velocity feature map is a two-dimensional vertical pedestrian velocity diffusion feature map.

3. The method according to claim 1, characterized in that, The step of recommending navigation sub-targets based on the fused diffusion feature map and the target robot's target navigation position, to obtain the optimal sub-target position of the target robot as it moves towards the target navigation position, includes: The current moving position and target navigation position of the target robot are mapped onto the two-dimensional coordinate system where the fused diffusion feature map is located to obtain the corresponding starting search position and target search position; Based on the initial search position, a breadth-first search algorithm is used to search for passable positions in the fused diffusion feature map to obtain multiple candidate passable positions that meet the preset search conditions. For each of the multiple candidate travel locations, a comprehensive path suitability score is calculated based on the target search location and the neighborhood feature distribution data of the candidate travel location in the fused diffusion feature map. The comprehensive path suitability scores of the multiple candidate passage locations are sorted in descending order, and the physical mapping position of the candidate passage location with the largest comprehensive path suitability score in the current working environment is selected as the optimal sub-target location.

4. The method according to claim 3, characterized in that, For each candidate travel location, the step of calculating the comprehensive path suitability score of the candidate travel location based on the target search location and the neighborhood feature distribution data of the candidate travel location in the fused diffusion feature map includes: Based on the neighborhood feature distribution data of the candidate passage location, calculate the neighborhood passage suitability score of the candidate passage location at the fused diffusion feature map; Calculate the Euclidean distance between the target search location and the candidate travel location to obtain the global navigation suitability score of the candidate travel location at the fused diffusion feature map; wherein, the global navigation suitability score is the negative of the corresponding Euclidean distance; Based on the preset score influence weights of the neighborhood accessibility score and the global navigation suitability score, a weighted summation operation is performed on the neighborhood accessibility score and the global navigation suitability score to obtain the comprehensive accessibility score of the candidate access location.

5. The method according to claim 4, characterized in that, The neighborhood mobility suitability score of any candidate access location at the fused diffusion feature map is calculated using the following formula: ； in, The x-coordinate value used to represent the candidate passage location in the fused diffusion feature map. The vertical coordinate value used to represent the candidate passage location in the fusion diffusion feature map. The neighborhood accessibility score used to represent the candidate access location Used to represent odd-numbered radii in a neighborhood. Used to represent the floor function. Used to indicate the corresponding x-coordinate And the ordinate is The target feature point is located at the actual feature value in the fusion diffusion feature map.

6. The method according to claim 1, characterized in that, The method further includes: The action evaluation network, which belongs to the same Actor-Critic reinforcement learning architecture as the action decision network, is invoked to predict environmental features based on the expected movement action and the fused diffusion feature map, thereby obtaining a predicted environmental feature map under the execution of the expected movement action. The action evaluation network is invoked to evaluate the action value of the expected movement action based on the fused diffusion feature map and the predicted environment feature map, so as to obtain the actual action value of the expected movement action. The action decision network is optimized according to the actual action value of the desired movement.

7. The method according to any one of claims 1-6, characterized in that, Before performing the step of recommending navigation sub-targets based on the fused diffusion feature map and the target navigation position of the target robot, the method further includes: Based on the fused diffusion feature map, a forward passage risk assessment is performed on the current working environment to obtain the corresponding forward risk assessment value; Detect whether the forward risk assessment value exceeds a preset risk assessment threshold; If the forward risk assessment value is not detected to exceed the preset risk assessment threshold, the step of recommending navigation sub-targets based on the fused diffusion feature map and the target robot's target navigation position is executed; otherwise, the current forward direction of the target robot is adjusted, and the process jumps to the step of obtaining the LiDAR data and RGB-D video stream data collected by the target robot for the current working environment to continue execution.

8. The method according to claim 7, characterized in that, The forward traffic risk assessment process based on the fused diffusion feature map is represented by the following formula: ； in, Used to represent the fusion diffusion feature map Used to represent the forward risk assessment value determined based on the fusion diffusion feature map. Used to represent the minimum safe column count threshold. The total number of feature columns used to represent the fusion diffusion feature map This is used to represent the total number of feature rows in the fused diffusion feature map. Used to indicate the height of the access risk assessment area. The first one used to represent the fusion diffusion feature map Line 1 The actual feature value at the feature point location of the column. Used to represent the floor function.

9. A robot control device, characterized in that, The robot control device is communicatively connected to the target robot, which is equipped with an RGB-D camera and a lidar. The robot control device includes a processor and a memory, the memory storing a computer program executable by the processor, and the processor executing the computer program to implement the robot navigation method with fused multimodal perception as described in any one of claims 1-8.

Images