Cloud robot SLAM (Simultaneous Localization and Mapping) system based on multi-modal environment attention regulation and control mechanism

Abstract

The invention discloses a cloud robot SLAM (Simultaneous Localization and Mapping) system based on a multi-modal environment attention regulation and control mechanism, which belongs to the technical field of cloud robot SLAM and comprises a cloud robot control module, a robot sensor sensing module and a cloud SLAM module, the cloud robot control module is connected with the cloud SLAM module through the embedded service scheduling module and the robot communication module; the cloud robot control module is connected with the robot sensor sensing module through the embedded service scheduling module and the intelligent acquisition and conversion module; and the cloud SLAM module dynamically schedules a data acquisition task of each sensor according to a calculation task demand by utilizing an environment attention regulation and control mechanism so as to realize a high-precision map building task under all weathers in a complex environment. According to the method, the sensing workload of the cloud robot in a complex environment is effectively reduced, unnecessary data transmission and processing are reduced, the problem of environmental limitation of a sensor can be solved, the precision of a map is improved, the robustness of a system is enhanced, and the method is suitable for a complex practical application scene.

Description

Technical Field

[0001] The present invention belongs to the technical field of cloud robot SLAM, and in particular relates to a cloud robot SLAM system based on a multimodal environment attention regulation mechanism. Background Art

[0002] As a cutting-edge direction of robot technology application, robot SLAM faces the contradiction between local resources and real-time complex computing requirements due to limitations in robot body size, energy consumption, cost, etc. With the rapid development of cloud computing and robot technology, cloud robot SLAM supports the efficient operation of robot tasks in the form of "resources as a service" in the background, and makes use of sufficient computing and storage resources in the cloud to make up for the lack of local resources of the robot.

[0003] In the prior art, the invention patent "CN202111498556.4" discloses a distributed SLAM robot control strategy based on cloud services. The robot first turns on its own data acquisition module, collects information in real time through its own data acquisition module, and establishes a communication connection with the cloud service end, which solves the data interaction problem between the SLAM robot and the cloud service end, and realizes the collaborative computing and work between the service robot and the cloud. The master's thesis of the National University of Defense Technology, "Research on Synchronous Positioning and Mapping Technology of Cloud Robots", proposes a SLAM service architecture based on cloud robots, which puts the computationally intensive data processing and data calculation stages of the SLAM process into the cloud. The robot actively applies for SLAM services during operation and continuously provides environmental perception data to the service, which can obtain the constructed map and its own positioning without worrying about how the SLAM process is implemented and whether there are sufficient computing resources.

[0004] However, there are still the following challenges in using existing cloud robot methods to perform simultaneous positioning and mapping tasks in complex dynamic environments: (1) There is redundant collection of sensor data. Redundant data collection of sensors increases the burden of data processing, especially in dynamic environments. When all sensors continuously collect data, system resources are seriously wasted. Most traditional SLAM systems use a fixed frequency to collect all sensor data, and fail to dynamically adjust according to the actual needs of the task and environment, which leads to low computing efficiency and high energy consumption of local computing resources; (2) Robot sensors involve a large number of sensors such as visual sensors, lidar, GPS, IMU, millimeter wave radar, etc., and the sensors have environmental limitations. The working environment lighting, weather, number, type, and distribution density of obstacles will directly affect the sensor's perception ability. LiDAR may be disturbed in fog, haze, rainy days, etc. In indoor environments with high obstacle density, LiDAR's line of sight will also be blocked, resulting in data loss or incompleteness. Especially in the face of glass or reflective target environments, LiDAR cannot accurately perceive and identify, and RGB-D camera performance may decline when the lighting changes greatly or there is occlusion. Although millimeter wave radar is suitable for severe weather environments, its point cloud is too sparse. (3) Although there are many existing SLAM methods based on sensor fusion, most of them rely on fixed sensor acquisition working modes and have certain environmental fusion limitations. In particular, when facing light and dark environment changes and dynamic obstacle occlusion scenes, the algorithm is difficult to respond flexibly, resulting in reduced mapping accuracy. Therefore, for cloud robot SLAM tasks in complex and dynamically changing scenes, how to further reduce cloud robot power consumption, reduce hardware integration costs, and improve mapping accuracy is of great significance to promoting the practical application of robots. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a cloud robot SLAM system based on a multimodal environment attention control mechanism, which performs dynamic data acquisition service scheduling according to task requirements to reduce local computing energy consumption, and realizes SLAM that adaptively fuses multi-source data such as vision, lidar, and millimeter waves under the action of the multimodal environment attention control mechanism to cope with high-precision mapping tasks in dynamic and complex environments.

[0006] The present invention provides the following technical solutions:

[0007] Provided is a cloud robot SLAM system based on a multimodal environment attention regulation mechanism, comprising: a cloud robot control module, a robot sensor perception module, an intelligent acquisition conversion module, an embedded service scheduling module, a robot communication module and a cloud SLAM module;

[0008] The cloud robot control module is connected to the cloud SLAM module through the embedded service scheduling module and the robot communication module;

[0009] The cloud robot control module is connected to the robot sensor perception module through the embedded service scheduling module and the intelligent acquisition conversion module;

[0010] The cloud SLAM module utilizes the environmental attention regulation mechanism, and under the joint action of the intelligent acquisition conversion module and the embedded service scheduling module, dynamically schedules the data acquisition tasks of each sensor to achieve high-precision map building tasks in complex environments around the clock.

[0011] Optionally, the robot sensor perception module includes a laser radar, a millimeter wave radar, a depth camera, a GPS, an encoder and an IMU.

[0012] Optionally, the environmental attention control mechanism is used to dynamically schedule the data collection tasks of each sensor under the joint action of the intelligent collection conversion module (3) and the embedded service scheduling module (4) to achieve a high-precision map construction task in a complex environment around the clock, specifically including:

[0013] Step SL1: uniformly map the heterogeneous multi-sensor data acquired by the intelligent acquisition conversion module to the same coordinate system;

[0014] Step SL2: According to the environmental attention control mechanism, the data collection tasks of each sensor are dynamically scheduled in real time, and the fusion weight of each sensor data is adjusted;

[0015] Step SL3: Based on the adjusted fusion weights, the sensor data are fused, and the Cartographer algorithm is used to estimate the robot's posture and simultaneously build a map of the surrounding environment, so as to ultimately achieve high-precision global map construction in complex environments.

[0016] Optionally, the step SL1 includes:

[0017] The collected lidar point cloud, millimeter-wave radar point cloud and depth color image data are internally aligned and externally aligned; the internal alignment includes correcting the focal length, principal point offset and distortion parameters of the depth camera; the external alignment includes: determining the constraint relationship between the lidar and the millimeter-wave radar, and between the lidar and the depth camera through the relative spatial position and direction between the depth camera, the lidar and the millimeter-wave radar, so that the data of each sensor is mapped to a unified coordinate system.

[0018] Optionally, the step SL2 includes:

[0019] Obtain environmental obstacle density based on the LiDAR point cloud; obtain ambient light intensity based on the depth camera’s color image data;

[0020] Based on the preset scheduling strategy, the data collection tasks of each sensor are dynamically scheduled in real time according to the density of environmental obstacles and the ambient light intensity, including the collection frequency of each sensor;

[0021] According to the density of environmental obstacles and the intensity of ambient light, adjust the weight adjustment parameters of each sensor and obtain the weight score of each sensor;

[0022] Based on the weight scores of each sensor, the Softmax function is used to perform normalization processing to obtain the probability weight of each sensor;

[0023] Based on the probability weight of each sensor, adjust the fusion weight of each sensor during pose estimation.

[0024] Optionally, the environmental obstacle density D is:

[0025]

[0026] Among them, N obstacles is the number of point clouds identified as obstacles by the LiDAR point cloud, N total is the number of environmental point clouds in the lidar point cloud;

[0027] The ambient light intensity L is:

[0028]

[0029] Among them, N totalpx is the total number of pixels of the color image acquired by the depth camera, ∑gray is the total gray value of the color image acquired by the depth camera;

[0030] The weight score ω of the i-th sensor i for:

[0031] ω i =α i L+β i ·D

[0032] Among them, α i is the weight adjustment parameter of L, β i is the weight adjustment parameter of D;

[0033] The probability weight of the i-th sensor is θ i for:

[0034]

[0035] Optionally, the intelligent acquisition and conversion module performs unified computation and integration on heterogeneous data from multiple sensors to achieve unified data acquisition at the physical layer; the embedded service scheduling module schedules data acquisition on demand and transmits control instructions issued by the cloud SLAM module to the cloud robot control module to drive the cloud robot.

[0036] Optionally, the intelligent acquisition conversion module includes a data integration interface layer and a data analysis conversion layer; the interfaces integrated in the data integration interface layer include USB, CAN, RS485 and RS232; the data integration interface layer is connected to each sensor of the robot sensor perception module;

[0037] The data analysis and conversion layer receives and analyzes the sensor data input by the data integration interface layer, and implements analysis and encapsulation of the perception information according to a predefined communication protocol.

[0038] Optionally, the embedded service scheduling module schedules data collection on demand, including:

[0039] Step D1: Divide the cloud robot into tasks, including sensing tasks, communication tasks and control tasks;

[0040] Step D2: assigning different priorities to various tasks, where the control task is given the highest priority, the communication task is second, and the sensing task is the lowest priority;

[0041] Step D3: Enable priority-based task scheduling, and the communication task immediately seizes CPU resources and starts execution;

[0042] Step D4: Waiting for the perception command of the cloud SLAM module. The sensing task stays in the waiting state before receiving the perception command.

[0043] Step D5: Read the instructions from the cloud SLAM module, parse them, and obtain perception instructions and control instructions;

[0044] Step D6: Determine whether there is a perception requirement: If yes, proceed to step D7; if there is no perception requirement and the cloud SLAM module only issues a control instruction, then return to step D4 and continue to wait for the perception instruction;

[0045] Step D7: According to the perception instructions of the cloud SLAM module, the corresponding sensing task is scheduled to be executed, and only the required specified sensor data is collected;

[0046] Step D8: parsing and processing the collected sensor raw data, and performing data conversion and integration according to a predefined communication protocol;

[0047] Step D9: Upload the packaged perception information to the cloud SLAM module, then return to step D4 and enter the next task cycle.

[0048] Optionally, the cloud robot control module includes a cloud robot microcontroller and a cloud robot driving submodule; the cloud robot microcontroller selects an STM32F405RGT6 chip as the main control chip, and the cloud robot driving submodule selects two DRV8870 chips as motor driving chips, which are respectively connected to the cloud robot microcontroller, receive PWM signals from the cloud robot microcontroller, and drive two motors to control the movement of the cloud robot.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] (1) The present invention not only has the advantages of existing cloud robot SLAM, but also overcomes the limitations of current technology use. It can achieve high-precision mapping tasks in complex environments around the clock under low-cost, high-efficiency control systems under constraints on computing, storage, and network resources, and complex dynamic environments.

[0051] (2) The present invention discloses a multimodal environmental attention control mechanism. Through comprehensive judgment on lighting conditions, environmental obstacle density, etc., the environmental attention activation layer is used to dynamically calculate the optimal environmental feature weight parameters of multiple sensors such as lidar, depth camera, and millimeter wave radar to cope with the dynamic changes in the robot's working environment and solve the problem of sensor environmental limitations. It can not only improve the accuracy of the map, but also enhance the robustness of the system and adapt to complex practical application scenarios.

[0052] (3) The present invention provides an embedded service scheduling module, which solves the problem of high energy consumption of sensors during cloud SLAM and realizes the selective collection of perception information. Compared with the full-speed collection strategy under the traditional fixed working mode of the sensor, the present invention combines the environmental attention control mechanism of the cloud SLAM module, only collects the perception information required by the cloud SLAM calculation in the current environment, and dynamically adjusts it according to the output results of the algorithm to avoid unnecessary collection tasks that waste system resources. This method can effectively reduce the perception workload of cloud robots in complex environments, reduce unnecessary data transmission and processing, improve system resource utilization, reduce energy consumption and reduce the demand for hardware performance.

[0053] (4) The present invention takes into account that the existing robot sensor perception module collects a variety of data types, and different interfaces require corresponding data converters, which are expensive and increase the local load of the robot. The present invention provides a cloud robot multi-sensor heterogeneous data unified computing integration service. By integrating multiple communication interfaces into the intelligent acquisition and conversion module, the sensors connected to the cloud robot can complete data acquisition and processing by themselves through the service. The collected sensor data will be converted using a unified cloud communication protocol and uploaded to the cloud by themselves, avoiding the need for expensive conversion equipment for some sensors, eliminating the need for redundant protocol conversion equipment, reducing costs and simplifying the system architecture, while reducing the local computing burden of the robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is an overall block diagram of the cloud robot SLAM system based on the multimodal environment attention control mechanism of the present invention;

[0055] Figure 2 It is a schematic diagram of the interface connection of the cloud robot microcontroller of the present invention;

[0056] Figure 3 It is a diagram of an on-demand scheduling data collection service method of an embedded service scheduling module of the present invention;

[0057] Figure 4 It is a flow chart of the cloud SLAM module of the present invention performing SLAM tasks based on the multimodal environment attention mechanism;

[0058] Figure 5 This is a comparison of the energy consumption of the robot system before and after optimization for different numbers of perception tasks of the same duration;

[0059] Figure 6 It is a comparison chart of the energy consumption of the robot system before and after optimization of different perception tasks of the same duration.

[0060] Marked in the figure are: 1 is the cloud robot control module, 2 is the robot sensor perception module, 3 is the intelligent acquisition and conversion module, 4 is the embedded service scheduling module, 5 is the robot communication module, 6 is the cloud SLAM module, 11 is the cloud robot microcontroller, 12 is the cloud robot driving submodule, 31 is the data integration interface layer, 32 is the data analysis and conversion layer, 61 is the cloud ROS service layer, and 62 is the cloud SLAM service layer. DETAILED DESCRIPTION

[0061] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0062] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatus.

[0063] like Figure 1 As shown, the present invention discloses a cloud robot SLAM system based on a multimodal environment attention regulation mechanism, including a cloud robot control module 1, a robot sensor perception module 2, an intelligent acquisition conversion module 3, an embedded service scheduling module 4, a robot communication module 5, and a cloud SLAM module 6.

[0064] like Figure 2 As shown, the cloud robot control module 1 includes a cloud robot microcontroller 11 and a cloud robot driving submodule 12. The cloud robot microcontroller 11 selects the STM32F405RGT6 chip (hereinafter referred to as the STM32 chip) as the main control chip. The A4, A5, A6, A7 pins and the B3, B4, B5, B6 pins of the chip are respectively connected to the IMU and encoder of the robot sensor perception module 2. The STM32 chip receives data through the SPI bus and the UART bus; the A11 pin and the A12 pin of the STM32 chip are connected to the intelligent acquisition conversion module 3. The CAN interface of the STM32 chip is connected to realize CAN bus communication; the A2 and A3 pins of the STM32 chip are connected to the USB interface of the intelligent acquisition conversion module 3 for USB data transmission; the B10 and B11 pins of the STM32 chip are respectively connected to the RS485 interface to establish RS485 bus communication; the C7 and C8 pins of the STM32 chip are respectively connected to the RS232 interface to build RS232 bus communication; the C10 and C11 pins of the STM32 chip are respectively connected to the self-organizing network interface of the robot communication module 5 to realize self-organizing network communication. The cloud robot driving submodule 12 selects two DRV8870 chips as motor driving chips, which are respectively connected to the C1, C9 pins and C6, C0 pins of the cloud robot microcontroller 11, receive PWM signals from the cloud robot microcontroller 11, and drive two motors to control the movement of the cloud robot.

[0065] The robot sensor perception module 2, including GPS, IMU, depth camera, lidar, millimeter wave radar and encoder, provides multi-dimensional spatiotemporal information such as visual images, depth distance, point cloud, etc., which is used for complex environment perception of cloud robot SLAM.

[0066] The intelligent acquisition and conversion module 3 includes a data integration interface layer 31 and a data analysis and conversion layer 32, thereby realizing unified computing and integration services for multi-sensor heterogeneous data.

[0067] The data integration interface layer 31 includes USB, CAN, RS232 and RS485 interfaces, supports various interface types in the existing robot sensor perception module 2, connects to the robot sensor perception module 2, and receives various sensor data. The data analysis and conversion layer 32 can realize the signal level conversion of sensor input data, extract and calculate the effective data frame information of the sensor, and complete the data integration service according to the customized conversion protocol, and finally realize the unified integration service of heterogeneous sensor information.

[0068] The specific method for realizing the unified computing and integration service of multi-sensor heterogeneous data is as follows:

[0069] Step S1: Integrate the USB, CAN, RS485, RS232 and other communication interfaces required by the robot sensor perception module 2 into the data integration interface layer 31 in the intelligent acquisition conversion module 3;

[0070] Step S2: receiving sensor data through the data integration interface layer 31, and then passing the data to the data analysis and conversion layer 32;

[0071] Step S3: The data analysis and conversion layer 32 receives and analyzes the sensor data input from the USB, CAN, RS485, and RS232 interfaces, and completes the packaging of the perception information according to a custom proprietary communication protocol.

[0072] Specifically, the data frame starts with fixed 0xEF and 0xFE; followed by the data length byte, which indicates the number of bytes in the data portion of the frame; followed by an identifier used to distinguish different categories of sensor data, 0x01 for lidar data, 0x02 for millimeter-wave radar data, 0x03 for depth camera data, 0x04 for GPS data, 0x05 for IMU data, and 0x06 for encoder data; followed by the data content of the frame, consisting of several four-byte floating-point data, the number of which is determined by the specific sensor output information; the end of the frame is a checksum used to ensure data integrity, which is calculated by the accumulation of all data characters in the data frame.

[0073] In particular, the millimeter-wave radar data based on CAN communication requires STM32 chip embedded computing analysis, the process is as follows:

[0074] Range=(CAN_Byte3×256+CAN_Byte4)×0.01

[0075] Azimuth=CAN_Byte5-90

[0076] Vrel=(CAN_Byte6×256+CAN_Byte7)×0.05-35

[0077] Among them, Range, Azimuth, and Vrel represent the relative distance, angle, and speed of the radar target, respectively. CAN_Byte3, CAN_Byte4, CAN_Byte5, CAN_Byte6, and CAN_Byte7 are the 3rd, 4th, 5th, 6th, and 7th bits of the CAN data frame, respectively. Then, Range, Azimuth, and Vrel are converted into four-byte binary representations for data packaging. The cloud protocol frame starts with fixed frame headers 0xEF and 0xFE, indicating the start of communication; followed by the data length field, which is set to 13 in this protocol, including the location information of three floating-point numbers; the identifier field is set to 0x02 to indicate that the frame data is the millimeter-wave radar information input by the CAN interface; the data content part includes Range, Azimuth, and Vrel, each coordinate is represented by a 4-byte floating-point number, Range information is stored in the 4th to 7th bytes, Azimuth from the 8th to the 11th bytes, and Vrel from the 12th to the 15th bytes; to ensure data integrity, the checksum is set to the accumulation of all data bytes in the frame and stored in the 16th byte of the frame.

[0078] Step S4: Establish data communication in a complex environment with the cloud SLAM module 6 through the robot communication module 5, call the cloud ROS service layer 61, and obtain the corresponding data service provided by the embedded service scheduling module 4 according to task requirements. The data service is uniformly released through the intelligent acquisition and conversion module 3.

[0079] The embedded service scheduling module 4 provides real-time scheduling of multimodal sensor data acquisition and conversion services and robot control services to achieve scheduling services required by upper-level tasks. This module establishes a connection with the cloud SLAM module 6 through the robot communication module 5, and can provide a stable and reliable network connection in complex and harsh environments.

[0080] Although the unified computing integration service of multi-sensor heterogeneous data has solved the problem of unified data collection at the physical layer, in the existing SLAM technology, the robot sensor data is always collected and uploaded at the configured frequency. In order to realize task-oriented data collection and sharing, a reasonable embedded service scheduling strategy is also required to schedule data collection services on demand to reduce the waste of computing and communication resources, and at the same time provide the perception information required for cloud SLAM on demand to ensure the generation of high-precision maps.

[0081] like Figure 3 As shown, the embedded service scheduling module 4 provides an on-demand scheduling data collection service, including:

[0082] Step D1: Divide and design cloud robot tasks based on real-time embedded systems. Tasks include sensing tasks, communication tasks, and control tasks. Sensing tasks provide data acquisition services for each sensor in the robot sensor perception module 2, communication tasks provide communication services between the cloud SLAM module and the embedded system, and control tasks provide robot motor drive services. Sensing tasks and control tasks are triggered by the perception instructions and control instructions issued by the cloud SLAM module. Sensing tasks are divided into independent multi-channel communication tasks based on the multiple interfaces integrated in the intelligent acquisition conversion module.

[0083] Step D2: Assign different priorities to various tasks, among which the control task is given the highest priority to ensure that it can take priority in task execution and avoid delayed execution due to interference from other tasks; the communication task is given a medium priority to ensure that the intelligent acquisition conversion module can respond to the perception needs of the cloud SLAM module in a timely manner and dynamically adjust the target sensor data acquisition without affecting the motion control of the cloud robot; the sensing task is given the lowest priority to avoid affecting the real-time performance of the control task and the communication task, and to ensure the stable operation of the cloud robot;

[0084] Step D3: The system starts priority-based task scheduling, and the communication task immediately seizes CPU resources and starts execution;

[0085] Step D4: The system waits for the perception command of the cloud SLAM module, and the sensing task stays in the waiting state before receiving the perception command;

[0086] Step D5: The system reads the instructions from the cloud SLAM module, parses them, and obtains perception instructions and control instructions;

[0087] Step D6: Determine whether there is a perception requirement: If yes, proceed to step D7; if there is no perception requirement and the cloud SLAM module only issues a control instruction, then return to step D4 and continue to wait for the perception instruction;

[0088] Step D7: According to the perception instructions of the cloud SLAM module, the corresponding sensing task is scheduled to be executed, and only the required specified sensor data is collected;

[0089] Step D8: Analyze and process the collected sensor raw data, and perform data conversion and integration according to the customized cloud communication protocol;

[0090] Step D9: Upload the packaged perception information to the cloud SLAM module, then return to step D4 and enter the next task cycle.

[0091] The robot communication module 5 is composed of a self-organizing network radio station, which can transmit the video image of the cloud robot through the network address and transmit control data in both directions. The transmission distance exceeds 1Km. The use of dual-antenna 2*2MIMO technology can ensure good transmission diffraction performance in complex dynamic scenes.

[0092] The cloud SLAM module 6 includes a cloud ROS service layer 61 and a cloud SLAM service layer 62 .

[0093] The cloud ROS service layer 61 provides a ROS environment for the cloud SLAM module 6, and can interact with the embedded service scheduling module 4, send task requirements and receive the required environmental perception data in real time. The cloud SLAM service 62 uses the multimodal environmental attention control mechanism to propose data service requirements and obtain perception information through the cloud ROS service layer 61, perform SLAM calculations, and finally achieve highly robust SLAM in extremely complex environments and generate high-precision maps.

[0094] The cloud SLAM module 6 can utilize the environmental attention control mechanism, and under the joint action of the intelligent acquisition conversion module 3 and the embedded service scheduling module 4, dynamically schedule data acquisition tasks to achieve high-precision map building tasks in complex environments around the clock, such as Figure 4 As shown, it specifically includes:

[0095] Step SL1: Align the cross-modal data of the radar vision to uniformly map the heterogeneous multi-sensor data obtained by the intelligent acquisition and conversion module to the same coordinate system.

[0096] Step SL2: Establish an environmental attention control mechanism to dynamically schedule the collection services of each sensor in real time and adjust the fusion weight of the sensor to cope with the dynamic changes of the robot's working environment.

[0097] Step SL3: Combine the fused perception data after the environmental attention regulation mechanism with the Cartographer algorithm to estimate the robot's posture in the environment and build a map of the surrounding environment at the same time, ultimately achieving high-precision global map construction in complex environments.

[0098] The cross-modal data alignment of RayVision is as follows: internal and external alignment is performed based on the collected LiDAR point cloud, millimeter-wave radar point cloud, and depth color image data.

[0099] Internal alignment involves obtaining the camera's focal length, principal point offset, and distortion parameters.

[0100] External alignment uses lidar as a reference. According to the relative spatial position and orientation of the depth camera, lidar and millimeter-wave radar, the constraint relationship between lidar and millimeter-wave radar, and lidar and depth camera is determined, so that the data of each sensor is mapped to a unified coordinate system to achieve spatial alignment.

[0101] Optionally, based on the depth camera perception information and filtering out ground or excessive redundant data, a method of point cloud projection is adopted to convert the depth image information into pseudo single-line lidar data, realizing the simulation of the depth camera point cloud to the lidar data; finally, through Iterative Closest Point (ICP) optimization, the pseudo single-line lidar point cloud is aligned with the lidar target point cloud to achieve the effect of minimizing the alignment error; due to the sparse point cloud of the millimeter-wave radar, only the ranging data of the lidar and the millimeter-wave radar needs to be aligned.

[0102] The environmental attention regulation mechanism includes environmental obstacle density judgment, environmental light intensity judgment, and an attention activation layer, thus realizing a dynamic environment adaptive fusion strategy.

[0103] To obtain the environmental obstacle density judgment, first calculate the obstacle density D based on the radar point cloud, and its formula is as follows:

[0104]

[0105] Among them, N obstacles represents the number of point clouds recognized as obstacles, and N total is the number of environmental point clouds. When 0 ≤ D ≤ 0.5, it indicates that the environmental obstacle density is low. At this time, increase the probability factors of the lidar and the camera, and appropriately reduce the probability factor of the millimeter-wave radar. When 0.5 < D ≤ 1, it indicates that the environmental obstacle density is high and the occlusion is serious. At this time, appropriately reduce the weight of the RGB-D camera to avoid perception errors caused by limited viewing angles and occlusions, and increase the probability weights of the lidar and the millimeter-wave radar.

[0106] For the environmental light intensity judgment, first calculate the environmental light intensity L based on the depth camera, and its formula is as follows:

[0107]

[0108] Among them, ∑gary represents the total gray value of the image, and N totalpx represents the total number of image pixels. When 0 ≤ L ≤ 0.3, it indicates that the environmental light is poor. At this time, the reliability of the camera decreases. When performing multi-modal data fusion, reduce the probability weight of the camera and increase the weight of the radar. When 0.3 < L < 0.7, it indicates that the environmental light is uniform and moderate. At this time, increase the weight of the camera. When 0.7 ≤ L ≤ 1, it indicates that the light intensity is too high. At this time, reduce the weight of the RGB-D camera and allocate more weights to other sensors such as the lidar and the millimeter-wave radar to ensure the accuracy of the fused data.

[0109] For the environmental attention activation layer, first, according to the collected environmental information, for the initial weight score ω of each sensor i iCalculate; the calculation formula for the initial probability weight score is as follows:

[0110] ω i =α i L+β i ·D

[0111] Among them, α i and β i It is a weight adjustment parameter, which is adjusted according to the working characteristics of each sensor in different environments.

[0112] After the initial weight score calculation is completed, the activation layer is normalized by the Softmax function to calculate the probability weight of each sensor θ i , the formula is as follows:

[0113]

[0114] Based on the probability weight of each sensor, the fusion weight of each sensor during pose estimation is adjusted. The sensor data is fused based on the adjusted fusion weight. The Cartographer algorithm is used to estimate the robot's pose and simultaneously build a map of the surrounding environment, ultimately achieving high-precision global map construction in complex environments.

[0115] like Figure 5 As shown, Figure 5 (a) is a comparison of the energy consumption of the robot system before and after optimization when performing a perception task; Figure 5 (b) is a comparison of the energy consumption of the robot system before and after optimization when performing two perception tasks; Figure 5 (c) is a comparison of the energy consumption of the robot system before and after optimization when performing three perception tasks; when the cloud SLAM module sends different numbers or types of sensor perception commands, the system records the power changes and measures the power consumption before and after optimization for the same duration. Figure 5 The comparison of the power consumption of the robot before and after optimization shows that the power consumption of the system before optimization always remains at a high level when performing 1, 2, and 3 perception tasks. This is because the unoptimized system continuously collects all sensor information and does not adjust with the changes in the demand for the cloud SLAM module. The optimized sensing tasks can selectively collect sensor data according to decision-making needs. Although the energy consumption increases with the increase in the number of tasks, it is still significantly lower than the energy consumption of the system before optimization.

[0116] Another set of experiments compared the power consumption of different types of tasks before and after optimization for the same duration, e.g. Figure 6 As shown, Figure 6 (a) Power consumption before and after optimization when performing voltage sensing tasks; Figure 6(b) Optimizing front and rear power consumption when performing radar sensing tasks; Figure 6 (c) Power consumption before and after optimization when performing encoder sensing tasks; the results show that when the system performs voltage sensing tasks, radar sensing tasks and encoder sensing tasks, the energy consumption of the robot system is lower than the level before optimization.

[0117] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by software plus a necessary general hardware platform.

[0118] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A cloud robot SLAM system based on a multimodal environment attention control mechanism, characterized in that: include: A cloud robot control module (1), a robot sensor perception module (2), an intelligent acquisition conversion module (3), an embedded service scheduling module (4), a robot communication module (5) and a cloud SLAM module (6); The cloud robot control module (1) is connected to the cloud SLAM module (6) via the embedded service scheduling module (4) and the robot communication module (5); The cloud robot control module (1) is connected to the robot sensor perception module (2) via the embedded service scheduling module (4) and the intelligent acquisition conversion module (3); The cloud SLAM module (6) utilizes the environmental attention control mechanism and, under the joint action of the intelligent acquisition conversion module (3) and the embedded service scheduling module (4), dynamically schedules the data acquisition tasks of each sensor to achieve high-precision map building tasks in complex environments around the clock.

2. The cloud robot SLAM system based on multimodal environment attention control mechanism according to claim 1, characterized in that: The robot sensor perception module (2) comprises a laser radar, a millimeter wave radar, a depth camera, a GPS, an encoder and an IMU.

3. The cloud robot SLAM system based on multimodal environment attention control mechanism according to claim 1, characterized in that: The environmental attention control mechanism is used to realize high-precision map building tasks in complex environments around the clock by dynamically scheduling the data collection tasks of each sensor under the joint action of the intelligent collection conversion module (3) and the embedded service scheduling module (4), specifically including: Step SL1: uniformly mapping the heterogeneous multi-sensor data acquired by the intelligent acquisition conversion module (3) to the same coordinate system; Step SL2: According to the environmental attention control mechanism, the data collection tasks of each sensor are dynamically scheduled in real time, and the fusion weight of each sensor data is adjusted; Step SL3: Based on the adjusted fusion weights, the sensor data are fused, and the Cartographer algorithm is used to estimate the robot's posture and simultaneously build a map of the surrounding environment, so as to ultimately achieve high-precision global map construction in complex environments.

4. The cloud robot SLAM system based on multimodal environment attention control mechanism according to claim 3 is characterized in that: The step SL1 comprises: The collected lidar point cloud, millimeter-wave radar point cloud and depth color image data are internally aligned and externally aligned; the internal alignment includes correcting the focal length, principal point offset and distortion parameters of the depth camera; the external alignment includes: determining the constraint relationship between the lidar and the millimeter-wave radar, and between the lidar and the depth camera through the relative spatial position and direction between the depth camera, the lidar and the millimeter-wave radar, so that the data of each sensor is mapped to a unified coordinate system.

5. The cloud robot SLAM system based on multimodal environment attention control mechanism according to claim 3 is characterized in that: The step SL2 comprises: Obtain environmental obstacle density based on the LiDAR point cloud; obtain ambient light intensity based on the depth camera’s color image data; Based on the preset scheduling strategy, the data collection tasks of each sensor are dynamically scheduled in real time according to the density of environmental obstacles and the ambient light intensity, including the collection frequency of each sensor; According to the density of environmental obstacles and the intensity of ambient light, adjust the weight adjustment parameters of each sensor and obtain the weight score of each sensor; Based on the weight scores of each sensor, the Softmax function is used to perform normalization processing to obtain the probability weight of each sensor; Based on the probability weight of each sensor, adjust the fusion weight of each sensor during pose estimation.

6. The cloud robot SLAM system based on multimodal environment attention control mechanism according to claim 5, characterized in that: The environmental obstacle density D is: Among them, N obstacles is the number of point clouds identified as obstacles by the LiDAR point cloud, N total is the number of environmental point clouds in the lidar point cloud; The ambient light intensity L is: Among them, N totalpx is the total number of pixels of the color image acquired by the depth camera, ∑gray is the total gray value of the color image acquired by the depth camera; The weight score ω of the i-th sensor i for: oh i =a i ·L+β i ·D Among them, α i is the weight adjustment parameter of L, β i is the weight adjustment parameter of D; The probability weight of the i-th sensor is θ i for:

7. The cloud robot SLAM system based on multimodal environment attention control mechanism according to claim 1, characterized in that: The intelligent acquisition conversion module (3) performs unified computation and integration on heterogeneous data from multiple sensors to achieve unified data acquisition at the physical layer; the embedded service scheduling module (4) schedules data acquisition on demand and simultaneously transmits control instructions issued by the cloud SLAM module (6) to the cloud robot control module (1) to drive the cloud robot.

8. The cloud robot SLAM system based on multimodal environment attention control mechanism according to claim 7, characterized in that: The intelligent acquisition conversion module (3) comprises a data integration interface layer (31) and a data analysis conversion layer (32); the interfaces integrated in the data integration interface layer (31) include USB, CAN, RS485 and RS232; the data integration interface layer (31) is connected to each sensor of the robot sensor perception module (2); The data analysis and conversion layer (32) receives and analyzes the sensor data input by the data integration interface layer (31), and implements analysis and encapsulation of the perception information according to a predefined communication protocol.

9. The cloud robot SLAM system based on multimodal environment attention control mechanism according to claim 7, characterized in that: The embedded service scheduling module (4) schedules data collection on demand, including: Step D1: Divide the cloud robot into tasks, including sensing tasks, communication tasks and control tasks; Step D2: assigning different priorities to various tasks, where the control task is given the highest priority, the communication task is second, and the sensing task is the lowest priority; Step D3: Enable priority-based task scheduling, and the communication task immediately seizes CPU resources and starts execution; Step D4: Waiting for the sensing command from the cloud SLAM module (6), and the sensing task stays in a waiting state before receiving the sensing command; Step D5: Read the instructions from the cloud SLAM module (6), parse them, and obtain the perception instructions and control instructions; Step D6: Determine whether there is a sensing requirement: if yes, proceed to step D7; if there is no sensing requirement, the cloud SLAM module (6) only issues a control instruction, then return to step D4 and continue to wait for the sensing instruction; Step D7: According to the perception instruction of the cloud SLAM module (6), the corresponding sensing task is scheduled to be executed, and only the required specified sensor data is collected; Step D8: parsing and processing the collected sensor raw data, and performing data conversion and integration according to a predefined communication protocol; Step D9: Upload the packaged perception information to the cloud SLAM module (6), then return to step D4 and enter the next task cycle.

10. The cloud robot SLAM system based on multimodal environment attention control mechanism according to claim 1, characterized in that: The cloud robot control module (1) comprises a cloud robot microcontroller (11) and a cloud robot driving submodule (12); the cloud robot microcontroller (11) selects an STM32F405RGT6 chip as a main control chip, and the cloud robot driving submodule (12) selects two DRV8870 chips as motor driving chips, which are respectively connected to the cloud robot microcontroller (11), receive PWM signals from the cloud robot microcontroller (11), and drive two motors to control the movement of the cloud robot.

Images