Autonomous robot decision-making system based on multi-modal perception fusion and method thereof

Abstract

The invention relates to the technical field of autonomous robot decision-making systems, in particular to an autonomous robot decision-making system and method based on multi-modal sensing fusion, and the system comprises a multi-modal data collection module which is used for obtaining multi-modal sensing data of an environment; the data preprocessing module is connected with the multi-modal data acquisition module and is used for preprocessing the multi-modal sensing data; the feature extraction module is connected with the data preprocessing module and used for extracting features from the preprocessed multi-modal sensing data; the dynamic space-time diagram construction module is connected with the feature extraction module and is used for constructing a dynamic space-time diagram based on the features; the multi-modal data fusion module is connected with the dynamic space-time diagram construction module and is used for fusing the multi-modal data in the dynamic space-time diagram; through adaptive time alignment and a cross-modal space attention mechanism, the system can effectively process space-time inconsistency of data of different sensors, and centimeter-level positioning precision is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous robot decision-making systems, and more specifically, to an autonomous robot decision-making system based on multimodal perception fusion and a method thereof. Background Art

[0002] With the rapid development of artificial intelligence and robotics, autonomous robots are increasingly used in industries, homes, and public services. However, achieving efficient and reliable autonomous decision-making in complex and dynamic environments remains a huge challenge. Existing technologies mainly face the following problems:

[0003] First, traditional single-sensor methods, such as navigation systems that rely solely on visual SLAM, often have difficulty coping with various interferences and changes in complex environments. For example, when lighting conditions change or occlusion is severe, positioning loss or navigation failure is prone to occur. Jin et al. (Jin, X., Lan, W., & Chang, X. (2022). Neural Path Planning With Multi-Scale Feature Fusion Networks. IEEE Access, 10, 118176-118186) proposed a neural network-based path planning method. Although it has made innovations in feature fusion, it still does not fully solve the problem of integrating multimodal perception data.

[0004] Secondly, although some studies have tried to use multi-sensor fusion technology to improve the robustness of the system, most of the existing fusion methods use simple Kalman filtering or fixed weight fusion strategies. These methods are difficult to adapt to the dynamic changes of the environment and cannot give full play to the advantages of various sensors. Siddarth et al. (Siddarth, K., Barathraj, M., Dhipika, A., Vignesh, K., & Supriya, P. (2021). Path Planning for Mobile Robots using Deep Learning Architectures. 2021 International Conference on Advances in Electrical, Electronics, Communications, Computing and Automation (ICAECA), 1-6) proposed a path planning method based on deep learning, but failed to effectively deal with the spatiotemporal alignment and adaptive fusion problems of multimodal data.

[0005] In addition, existing autonomous robot systems are still insufficient in environmental understanding and human-computer interaction. In particular, when processing complex voice commands or identifying diverse objects, the accuracy and response speed often cannot meet the needs of practical applications. Sung et al. (Sung, T.-W., Tsai, P.-W., Gaber, T., & Lee, C.-Y. (2021). Artificial intelligence of things (AIoT) technologies and applications. Wireless Communications and Mobile Computing, 2021, 1-2) explored the application of artificial intelligence IoT technology in intelligent systems, but failed to provide a comprehensive multimodal perception fusion solution. Summary of the invention

[0006] The present invention is proposed in this context. To address the above problems, the present invention proposes an autonomous robot decision-making system based on multimodal perception fusion and its method. The system aims to solve the following key technical problems: how to effectively integrate multiple sensor data to achieve high-precision environmental perception and self-positioning; how to achieve adaptive data fusion and decision-making planning in a dynamically changing environment; how to improve the system's ability to understand complex environments and the efficiency of human-computer interaction.

[0007] The present invention provides an autonomous robot decision-making system based on multimodal perception fusion, comprising: a multimodal data acquisition module, used for obtaining multimodal perception data of an environment; a data preprocessing module, connected to the multimodal data acquisition module, used for preprocessing the multimodal perception data; a feature extraction module, connected to the data preprocessing module, used for extracting features from the preprocessed multimodal perception data; a dynamic space-time graph construction module, connected to the feature extraction module, used for constructing a dynamic space-time graph based on the features; a multimodal data fusion module, connected to the dynamic space-time graph construction module, used for fusing the multimodal data in the dynamic space-time graph; an environment understanding and state estimation module, connected to the multimodal data fusion module, used for understanding the environment and estimating the robot state based on the fused data; a decision planning module, connected to the environment understanding and state estimation module, used for making decisions and path planning based on the environment understanding and state estimation results; a control execution module, connected to the decision planning module, used for executing the planning results of the decision planning module; and an online learning and adaptation module, connected to the control execution module, used for online optimization of each module based on system operation data.

[0008] Specifically, the multimodal data acquisition module includes: a visual camera for obtaining RGB image data; a depth camera for obtaining depth map data; a lidar for obtaining 3D point cloud data; an inertial measurement unit IMU for obtaining acceleration and angular velocity data; a global positioning system GPS receiver for obtaining position data; and an acoustic sensor for obtaining sound data.

[0009] Specifically, the data preprocessing module includes: an image preprocessing unit, which is used to denoise, correct distortion and color balance image data; a point cloud preprocessing unit, which is used to downsample point cloud data, segment the ground and remove outliers; an IMU data preprocessing unit, which is used to perform zero bias correction and integral drift compensation on IMU data; a GPS data preprocessing unit, which is used to perform satellite signal filtering and coordinate conversion on GPS data; and an acoustic data preprocessing unit, which is used to perform noise reduction and spectrum analysis on sound data.

[0010] Specifically, the feature extraction module includes: a visual feature extraction unit for extracting visual features from image data; a depth feature extraction unit for extracting depth features from depth map data; a point cloud feature extraction unit for extracting geometric features from point cloud data; an IMU feature extraction unit for extracting motion features from IMU data; a GPS feature extraction unit for extracting position and velocity features from GPS data; and an acoustic feature extraction unit for extracting acoustic features from sound data.

[0011] Specifically, the multimodal data fusion module includes: an adaptive time alignment unit for time alignment of data of different modalities; a cross-modal spatial attention unit for capturing the spatial dependencies between features of different modalities; a temporal graph convolution unit for modeling the temporal evolution of features; and a dynamic feature aggregation unit for adaptively fusing multimodal information.

[0012] Specifically, the adaptive time alignment unit realizes the time alignment of data by minimizing the time alignment loss function, wherein the time alignment loss function is:

[0013] ,

[0014] in, and For Node and timestamp, is the predicted time deviation, is the adaptive weight, ,in is the total number of nodes in the dynamic space-time graph.

[0015] Specifically, the cross-modal spatial attention unit calculates the attention score by the following formula:

[0016] ,

[0017] in, and are query vector and key vector respectively, is the feature dimension, represents the matrix transpose operation, It refers to the dimension of the feature vector after feature extraction. For different modes, their features are uniformly mapped to the same dimension d, and the specific value is 256 dimensions.

[0018] Specifically, the timing graph convolution unit defines the timing graph convolution operation by the following formula: ,

[0019] in, for The adjacency matrix of order, is the weight matrix, is a learnable parameter, is the activation function, ,in is the length of the time series, the default setting ; , represents the order of adjacency relationship considered, Represents directly connected nodes. represents nodes connected through an intermediate node, Represents nodes connected through two intermediate nodes.

[0020] Specifically, the dynamic feature aggregation unit implements feature aggregation through the following formula:

[0021] ,

[0022] in, , is the number of modes, , corresponding to six different sensor modalities, Indicates The weight coefficient of each mode, Indicates The characteristics of a mode, Indicates the corresponding A feature transformation network of two modalities.

[0023] A method for an autonomous robot decision-making system based on multimodal perception fusion, comprising the following steps: obtaining multimodal perception data of an environment; preprocessing the multimodal perception data; extracting features from the preprocessed multimodal perception data; constructing a dynamic spatiotemporal graph based on the features; fusing the multimodal data in the dynamic spatiotemporal graph; understanding the environment and estimating the robot state based on the fused data; making decisions and planning paths based on the results of the environment understanding and state estimation; executing the decisions and path planning results; and online optimizing each step based on system operation data.

[0024] The present invention achieves an effective solution to the above problems by innovatively combining technologies such as multimodal data acquisition, adaptive time alignment, cross-modal spatial attention, temporal graph convolution and dynamic feature aggregation. Specifically, the present invention has the following beneficial effects:

[0025] First, the multimodal perception fusion technology proposed in this paper significantly improves the perception accuracy and robustness of the system. Through adaptive time alignment and cross-modal spatial attention mechanism, the system can effectively handle the spatiotemporal inconsistency of different sensor data and achieve centimeter-level positioning accuracy, which greatly exceeds the existing technical level.

[0026] Secondly, the dynamic feature aggregation and temporal graph convolution technology of the present invention enables the system to better understand and predict the dynamic changes of the environment. This not only improves the success rate of navigation, but also enables the system to respond to complex task scenarios more intelligently. For example, in the application of home service robots, the system can accurately understand and execute complex instructions such as bringing the red apple on the table, demonstrating excellent environmental understanding and task execution capabilities.

[0027] Furthermore, the online learning and adaptation module of the present invention enables the system to have the ability of continuous optimization. By continuously learning and updating the model, the system can gradually adapt to new environments and tasks, which is crucial for autonomous robots deployed in dynamic environments for a long time.

[0028] Finally, although the present invention integrates complex algorithms and multiple data sources, the system achieves near real-time response speed through optimized parallel processing and efficient data flow design. In the test, the average response time of the system is less than 80 milliseconds, which greatly improves the smoothness of human-computer interaction and user experience.

[0029] In summary, the present invention provides a comprehensive, efficient and intelligent solution for autonomous robot decision-making in complex environments. It not only overcomes the shortcomings of existing technologies in multimodal data fusion and environmental adaptability, but also demonstrates significant performance advantages in practical applications. This innovative technical solution is expected to promote the widespread application of autonomous robot technology in multiple fields such as industry, home services, and medical assistance, and open up new directions for the development of intelligent robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is the overall logic block diagram of the system of the present invention.

[0031] Figure 2 It is a logic block diagram of the multimodal data acquisition module of the present invention.

[0032] Figure 3 This is a logic block diagram of the data preprocessing module of the present invention.

[0033] Figure 4 This is a logical block diagram of the feature extraction module of the present invention.

[0034] Figure 5 It is a logical block diagram of the dynamic space-time graph construction module of the present invention.

[0035] Figure 6 It is a logic block diagram of the multimodal data fusion module of the present invention.

[0036] Figure 7 It is a logical block diagram of the environment understanding and state estimation module of the present invention.

[0037] Figure 8 It is a logic block diagram of the decision-making planning module of the present invention.

[0038] Fig. 9 It is a logic block diagram of the control execution module of the present invention.

[0039] Fig.10 It is a logic block diagram of the online learning and adaptation module of the present invention. DETAILED DESCRIPTION

[0040] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects thereof are described in detail below in conjunction with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0041] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0042] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0043] The present invention relates to an autonomous robot decision-making system and method based on multimodal perception fusion, and in particular to an intelligent system capable of achieving efficient and reliable decision-making in a complex dynamic environment.

[0044] like Figure 1-10 As shown, the autonomous robot decision-making system of the present invention includes a multimodal data acquisition module 1, a data preprocessing module 2, a feature extraction module 3, a dynamic spatiotemporal graph construction module 4, a multimodal data fusion module 5, an environment understanding and state estimation module 6, a decision planning module 7, a control execution module 8, and an online learning and adaptation module 9. These modules work closely together to form a closed-loop perception-decision-execution system.

[0045] First, please refer to Figure 2 , the multimodal data acquisition module 1 is responsible for acquiring the multimodal perception data of the environment. The module includes a visual camera 11, a depth camera 12, a laser radar 13, an inertial measurement unit IMU14, a global positioning system GPS receiver 15 and an acoustic sensor 16. Preferably, the visual camera 11 adopts a binocular stereo camera with a resolution of 4K, a frame rate of 60fps, and a field of view of 120°. The depth camera 12 adopts structured light technology, with a resolution of 1080p, a maximum measurement distance of 10m, and an accuracy of ±1cm. The laser radar 13 is a 16-line laser radar with a 360° horizontal field of view and a 30° vertical field of view, and a measurement range of 100m. IMU14 is a 9-axis IMU with a sampling rate of 1kHz, an accelerometer range of ±16g, and a gyroscope range of ±2000° / s. The GPS receiver 15 is a dual-frequency GPS / GLONASS receiver with a positioning accuracy of <1m and an update rate of 10Hz. The acoustic sensor 16 uses an 8-microphone array with a sampling rate of 48kHz and a signal-to-noise ratio of >65dB.

[0046] In order to ensure the synchronization of data, the present invention adopts the Precision Time Protocol (PTP) to achieve hardware-level time synchronization. Specifically, the GPS module is set as the master clock source, and other sensors are synchronized with the master clock. Accurate timestamps are added to the data packets, allowing a maximum time error of ±1ms. This synchronization mechanism can effectively solve the time alignment problem of multimodal data and lay the foundation for subsequent data fusion.

[0047] Next, please refer to Figure 3 The data preprocessing module 2 preprocesses the collected multimodal sensing data. The module includes an image preprocessing unit 21, a point cloud preprocessing unit 22, an IMU data preprocessing unit 23, a GPS data preprocessing unit 24 and an acoustic data preprocessing unit 25.

[0048] The image preprocessing unit 21 first applies bilateral filtering for denoising, and the edge-preserving denoising effect is significant. Then, the camera is calibrated based on the Zhang Zhengyou checkerboard method to generate a distortion matrix for distortion correction. Finally, the gray world algorithm is used for automatic white balance. Preferably, the spatial kernel size of the bilateral filter is set to 5, and the range kernel size is set to 50.

[0049] The point cloud preprocessing unit 22 first uses voxel filtering for downsampling, and the voxel size is set to 0.05m. Then, the RANSAC algorithm is used to fit the ground plane, and the threshold is set to 0.1m. Finally, statistical filtering is used to remove outliers, and the mean distance multiple threshold is set to 2.0. The selection of these parameters is based on a large number of experimental results, which can effectively reduce the amount of data while retaining key information.

[0050] The IMU data preprocessing unit 23 first collects static state data, calculates the mean as zero bias, and performs zero bias correction. Then, a temperature-bias model is established to perform temperature compensation in real time. Finally, the zero-speed update (ZUPT) technology is used to compensate for the integral drift. Preferably, the temperature compensation model adopts cubic polynomial fitting, and the coefficients are determined by the least squares method.

[0051] The GPS data preprocessing unit 24 first removes satellite signals with a position precision factor (PDOP) greater than 5, and then transforms the data in the WGS84 coordinate system to the local ENU coordinate system. Preferably, the coordinate transformation adopts the Helmert transformation method, and the transformation parameters are calculated by least squares adjustment.

[0052] The acoustic data preprocessing unit 25 first uses wavelet transform to reduce noise, and the threshold is 3 times the signal standard deviation. Then a short-time Fourier transform is performed, and the window size is set to 1024 and the overlap rate is 50%. The selection of these parameters can effectively suppress noise while retaining the characteristics of the sound signal.

[0053] Please refer to Figure 4 After preprocessing, the feature extraction module 3 extracts features from the multimodal data. The module includes a visual feature extraction unit 31, a depth feature extraction unit 32, a point cloud feature extraction unit 33, an IMU feature extraction unit 34, a GPS feature extraction unit 35, and an acoustic feature extraction unit 36.

[0054] The visual feature extraction unit 31 uses the pre-trained ResNet50 model, removes the last fully connected layer, and extracts the feature map of the conv5_x layer, which has a size of (2048, H / 32, W / 32). Then, spatial pyramid pooling (SPP) is applied to generate a fixed-size feature vector. Preferably, the output dimension of the SPP layer is set to 2048, and the pyramid level is [1, 2, 4].

[0055] The depth feature extraction unit 32 first converts the depth map into a point cloud representation, and then uses PointNet++ to extract features. The number of sampling points is set to 1024 and the feature dimension is set to 128. Multi-scale feature aggregation is adopted, and the scale ratio is [0.2, 0.4, 0.8]. This multi-scale strategy can effectively capture geometric information at different scales.

[0056] The point cloud feature extraction unit 33 uses VoxelNet, and the voxel size is set to [0.2m, 0.2m, 0.4m]. A maximum of 5 points in each voxel are randomly sampled, and 3 layers of VFE (Voxel Feature Encoding) layers are applied. Preferably, the output channel numbers of the VFE layers are 32, 64, and 128, respectively.

[0057] The IMU feature extraction unit 34 uses 1D-CNN, with an input window size of 256 and a step size of 64. The convolution layer is configured as [(64,3), (128,3), (256,3)], and each layer is followed by batch normalization and ReLU activation function. Finally, fixed-size features are obtained through global maximum pooling. This configuration can effectively capture the temporal characteristics of IMU data.

[0058] The GPS feature extraction unit 35 calculates the position difference of 10 consecutive frames to obtain the speed feature. Then the DOP value is extracted as the accuracy feature. Finally, the position, speed and accuracy features are spliced. Preferably, the speed feature is calculated using the central difference method to reduce the influence of noise.

[0059] The acoustic feature extraction unit 36 ​​first calculates the 13-dimensional MFCC features with a window size of 25ms and a step size of 10ms. Then Delta and Delta-Delta operations are applied to obtain 39-dimensional features. Finally, the acoustic feature extraction unit 36 ​​first calculates the 13-dimensional MFCC features with a window size of 25ms and a step size of 10ms. Then Delta and Delta-Delta operations are applied to obtain 39-dimensional features. Finally, 1D-CNN is used to extract time series features, with the same structure as the IMU feature extraction unit 34. This feature extraction method can effectively capture the spectrum and time series characteristics of the sound signal.

[0060] Please refer to Figure 5 After the feature extraction is completed, the dynamic spatio-temporal graph construction module 4 constructs a dynamic spatio-temporal graph (DSTG) based on the extracted features. The module includes a node generation unit 41, an edge generation unit 42 and a graph structure optimization unit 43.

[0061] The node generation unit 41 is for each moment , each mode Creating a Node The node attributes include timestamp, modality type, feature vector and confidence. Preferably, the confidence is represented by the L2 norm normalized value of the feature vector, reflecting the reliability of the feature.

[0062] The edge generation unit 42 is responsible for creating connections between nodes. First, a time series edge is generated to connect nodes of the same mode and adjacent time. , Represents the weight of the temporal edge, reflecting the degree of association between adjacent nodes in time. is the time attenuation coefficient, the preferred value is 0.1 second, Represents the time interval between two nodes. Then, a spatial edge is generated to connect nodes of different modes at the same time. , Represents the weight of the spatial edge, reflecting the similarity of features between nodes of different modalities, where Indicates The feature vector of a modal node, Indicates The feature vector of a modal node, , is the number of modes, Represents cosine similarity, with a value range of [-1,1]. Finally, local sensitive hashing (LSH) is used to quickly find k nearest neighbors. , generating long-range edges. This multi-type edge connection strategy can fully capture the spatiotemporal relationship between data.

[0063] The graph structure optimization unit 43 is responsible for dynamically adjusting the graph structure. First, the importance score of each edge is calculated: ,in, represents the weight of the edge, which can be or , depends on the type of edge, Indicates The confidence of a node, the value range is [0,1], Indicates The confidence of a node ranges from [0,1]. The confidence is represented by the L2 norm normalized value of the feature vector. , is the total number of nodes. Then, edges with scores lower than the threshold are removed. The threshold is adjusted dynamically, and the initial value is the median of all edge importance scores. Graph sparsification is performed every 100 frames to keep the density of the graph around 2096. This dynamic optimization strategy can reduce computational complexity while retaining key information.

[0064] Please refer to Figure 6Next, the multimodal data fusion module 5 fuses the multimodal data in the dynamic spatiotemporal graph. This module is the core innovative module of the present invention, and includes an adaptive time alignment unit 51, a cross-modal spatial attention unit 52, a temporal graph convolution unit 53, and a dynamic feature aggregation unit 54.

[0065] The adaptive time alignment unit 51 realizes accurate time alignment of data by minimizing the time alignment loss function. Specifically, the time alignment loss function is defined as:

[0066] ,

[0067] in, and For Node and timestamp, is the predicted time deviation, is an adaptive weight. Preferably, , is the Gaussian kernel width, set to the median of the L2 norm of the eigenvector.

[0068] Minimize using the Adam optimizer , the learning rate is set to 0.001, and the iterations are 50 times. Update timestamp: In order to ensure the stability of the alignment, the following constraints are introduced:

[0069] , limit the maximum time offset; , to ensure that the overall time does not shift. At the same time, set the following threshold judgment: If , the alignment is considered successful and the iteration is stopped; if 5 consecutive iterations If the improvement is less than 1%, the algorithm stops early. This adaptive time alignment method can effectively handle the sampling frequency differences and transmission delays of different modal data, laying the foundation for subsequent feature fusion.

[0070] The cross-modal spatial attention unit 52 captures the spatial dependencies between features of different modalities through the attention mechanism. Specifically, the attention score is calculated: ,in, , , , and are query vector and key vector respectively, is a value vector, represents the matrix transpose operation, refers to the dimension of the feature vector after feature extraction. For different modalities, their features are uniformly mapped to the same dimension d, with a specific value of 256. Then, attention weighting is performed. . Finally, residual connections and layer normalization are applied: . is a learnable weight matrix, , is the query transformation matrix; , is the key transformation matrix; is the value transformation matrix, and LayerNorm is the layer normalization.

[0071] In order to improve computational efficiency and model stability, the following constraints are introduced: the attention matrix A is sparse, retaining only the top-k values. ; Introduce a regularization term with a rate of 0.1 to prevent overfitting. At the same time, set the following threshold judgment: If , then the current modality is considered to be abnormal and the abnormality handling mechanism is triggered. This cross-modal spatial attention mechanism can adaptively focus on key information in different modalities and improve the effectiveness of feature extraction.

[0072] The temporal graph convolution unit 53 models the temporal evolution of features through graph convolution operations. Specifically, the temporal graph convolution operation is defined as: ,

[0073] in, for The adjacency matrix of order, is the weight matrix, is a learnable parameter, LeakyRelU is the activation function with a negative slope of 0.2. Three layers of graph convolution are stacked, and batch normalization is performed after each layer. ,in is the length of the time series, the default setting ; , represents the order of adjacency relationship considered, Represents directly connected nodes. represents nodes connected through an intermediate node, Represents nodes connected through two intermediate nodes.

[0074] In order to balance the expressiveness and computational efficiency of the model, the following constraints are introduced:

[0075] The graph is sparse, and each node retains a maximum of 10 edges; L2 regularization, coefficient 0.001, to prevent overfitting. At the same time, the following threshold judgment is set: if the node feature change rate ΔH / H<0.01, the iteration is stopped in advance; the maximum number of iterations is 10 to prevent excessive computational overhead. This time-series graph convolution operation can effectively capture long-term dependencies and improve the system's ability to understand dynamic environments.

[0076] The dynamic feature aggregation unit 54 realizes the fusion of multimodal information through adaptive weights. Specifically, the global feature is calculated: , where MaxPooling is the maximum pooling.

[0077] Then, calculate the dynamic weight: ,

[0078] in, and is a learnable parameter. Next, perform feature transformation: It is a two-layer perceptron with a hidden layer dimension of 256. Finally, feature aggregation is performed:

[0079] ,

[0080] in, , is the number of modes, , corresponding to six different sensor modalities, Indicates The weight coefficient of each mode, Indicates The characteristics of a mode, Indicates the corresponding A feature transformation network of two modalities.

[0081] In order to enhance the expressiveness and robustness of the model, a gating mechanism is introduced: ,in, are learnable parameters.

[0082] In order to ensure the effectiveness and stability of the aggregation, the following constraints are introduced: , weight normalization; introduce L1 regularization to , promote weight sparsity, the regularization coefficient is set to 0.01, and at the same time, the following threshold judgment is set: if , then a certain mode is considered to be over-dominant, triggering the rebalancing mechanism; if , then consider temporarily ignoring the mode to reduce the computational complexity. This dynamic feature aggregation strategy can adaptively fuse information according to the importance of different modes, improving the flexibility and robustness of the system.

[0083] Please refer to Figure 7After completing the multimodal data fusion, the environment understanding and state estimation module 6 performs environment understanding and robot state estimation based on the fused data. This module includes a target detection unit 61, a semantic segmentation unit 62, and a state estimation unit 63. The target detection unit 61 uses the YOLOv5s model with an input size of 640x640. The detection categories include pedestrians, vehicles, obstacles, etc., a total of 20 categories. Non-maximum suppression (NMS) is used for post-processing, and the loU threshold is set to 0.5. Only detection results with confidence > 0.6 are retained.

[0084] Preferably, for dynamic targets, Kalman filtering is used for tracking, and the state vector includes position, velocity and size information. The semantic segmentation unit 62 uses DeepLabv3+, and the backbone network is MobileNetV2. The output resolution is 1 / 4 of the input. The segmentation categories include roads, sidewalks, buildings, etc., a total of 10 categories. Conditional random field (CRF) post-processing is used to optimize the segmentation boundaries. Preferably, a multi-scale input strategy is adopted with a scale ratio of [0.5, 1.0, 1.5] to improve the robustness of the segmentation. The state estimation unit 63 uses an error state extended Kalman filter (ES-EKF). The state vector includes position, velocity, attitude and IMU zero bias. Observations include GPS position, visual odometer and IMU measurement. The prediction frequency is 100Hz and the update frequency is 10Hz. In order to improve the estimation accuracy, an adaptive noise covariance estimation method is adopted, as follows:

[0085] Compute the observation residuals: ,in For actual observation, To predict observations;

[0086] Compute the residual covariance: ,in is the observation matrix, is the prior covariance, is the observed voice covariance;

[0087] Update the observation noise covariance: ,in is the forgetting factor, which takes a value of 0.95. is the observed residual, Represents the outer product of the residual vector, and obtains the covariance matrix. This adaptive method can dynamically adjust the filter parameters according to the actual observation situation and improve the accuracy and robustness of state estimation.

[0088] Please refer to Figure 8 Based on the environment understanding and state estimation results, the decision planning module 7 makes decisions and plans paths. This module includes a task planning unit 71, a path planning unit 72 and a trajectory optimization unit 73.

[0089] The task planning unit 71 uses a hierarchical task network (HTN) to decompose tasks. The atomic actions defined include moving, turning, grasping, placing, etc. The task library contains high-level tasks such as navigation, object manipulation, and human-computer interaction. * The algorithm performs a task graph search. Preferably, a task priority mechanism is introduced to dynamically adjust the search order according to the urgency and importance of the tasks.

[0090] The path planning unit 72 generates a global path using the RRTx algorithm. The sampling strategy is 80% uniform sampling and 20% target-oriented sampling. The network structure radius is adjusted to 5 meters. The maximum number of iterations is 10,000, or the optimization continues for 1 second after the path is found. In order to improve planning efficiency, an adaptive sampling strategy is introduced: the sampling density function is defined as: ,in For point The distance to the nearest obstacle, is the density parameter; according to Perform importance sampling and increase sampling density near obstacles; dynamically adjust The initial value is 5 meters, which gradually decreases according to the planning difficulty, and the minimum value is 1 meter. This adaptive sampling strategy can significantly improve the planning efficiency while ensuring the path quality.

[0091] The trajectory optimization unit 73 uses model predictive control (MPC) with a prediction time domain of 3 seconds. The cost function considers path tracking error, velocity smoothness and control input. The IPOPT solver is used with a maximum number of iterations of 100. Nonholonomic constraints and dynamic constraints are considered. Preferably, an adaptive weight adjustment strategy is adopted.

[0092] Define the weight vector ; Dynamically adjust weights based on current status: , , ,in, is the path deviation, is the current speed, is the expected speed, , , is the adjustment coefficient; normalized weight: .

[0093] In the present invention, the weight The value range and meaning in different modes are as follows: ,in is the number of sensor modes , is the visual data weight, indicating the importance of visual features; is the depth data weight, indicating the importance of the depth feature; is the lidar data weight, indicating the importance of point cloud features; is the IMU data weight, indicating the importance of motion features, is the GPS data weight, indicating the importance of location features; is the acoustic data weight, indicating the importance of the sound feature;

[0094] Weight constraints: , ensure weight normalization, , represents the relative importance of each mode; The value of is adaptively adjusted by the dynamic feature aggregation unit 54 according to the current task requirements and environmental conditions.

[0095] This adaptive weight adjustment strategy can dynamically balance different optimization objectives according to the current state and improve the robustness and adaptability of the trajectory.

[0096] Please refer to Fig. 9 The control execution module 8 is responsible for executing the planning results of the decision planning module. The module includes a trajectory tracking unit 81, a dynamic constraint unit 82 and a safety supervision unit 83.

[0097] The trajectory tracking unit 81 uses a pure tracking algorithm, and the forward distance is dynamically adjusted (1.5~3 meters). PID controller parameters: The velocity planning uses a trapezoidal curve with a maximum acceleration of 1 m / s2. In order to improve the tracking accuracy, an adaptive foresight distance adjustment strategy is introduced: Define the foresight distance: ,in is the current speed, is the proportionality coefficient, is the minimum foresight distance; dynamically adjusted according to the path curvature ,in is the initial proportionality factor, To adjust the parameters, is the path curvature; limit the foresight distance range: ,This adaptive foresight distance adjustment strategy can maintain good tracking ,performance under different speeds and path complexity.

[0098] The dynamic constraint unit 82 considers the maximum speed (2 m / s) and maximum angular velocity (1 radian / s) limits. At the same time, considering the tire friction circle constraint, the maximum lateral acceleration is 0.5g. In order to prevent the risk of rollover, the dynamic stability constraint is introduced: Calculate the dynamic stability margin: ,in, is the wheelbase, is the centroid height, is the lateral acceleration, is the acceleration due to gravity; limit , ensuring sufficient stability margin. This dynamic stability constraint can effectively prevent the risk of tipping while ensuring the high maneuverability of the robot.

[0099] The safety monitoring unit 83 sets a safety buffer zone and keeps a minimum distance of 0.5 meters from obstacles. It detects collision risks in real time and triggers emergency braking if a collision is predicted within 1 second. It monitors the battery power and automatically returns to the charging station when it is less than 20%. It detects the actuator temperature and reduces the output power when it exceeds 60°C. In order to further improve safety, an adaptive safety distance adjustment strategy is introduced: Define the safety distance: ,in, is the current speed, is the current acceleration, and is the adjustment coefficient, The minimum safe distance; dynamically adjusted according to the obstacle type and :

[0100] Static obstacles: ;

[0101] Slow Moving Obstacles: ;

[0102] Fast moving obstacles: ;

[0103] Limit the safe distance range: .

[0104] This adaptive safety distance adjustment strategy can dynamically adjust the safety margin according to the current motion state and environmental characteristics, ensuring safety while not being overly conservative.

[0105] Please refer to Fig.10 Finally, the online learning and adaptation module 9 performs online optimization on each module based on the system operation data. The module includes a data collection unit 91, a performance evaluation unit 92, a model updating unit 93 and a parameter optimization unit 94.

[0106] The data collection unit 91 samples the system status, sensor data and control instructions at a frequency of 10 Hz. A sliding window is used to store the data of the last hour. Key events (such as obstacle avoidance and human-computer interaction) are recorded. An incremental data compression algorithm is used to reduce storage overhead. Specifically, the piecewise linear approximation method is used for data compression:

[0107] 1) Initialize a line segment containing the first two data points;

[0108] 2) Try to add the next data point to the current line segment;

[0109] 3) If the fitting error is less than the threshold ε (such as 0.05), the point is accepted; otherwise, the current segment is ended and a new segment is started;

[0110] 4) Repeat steps 2-3 until all data points have been processed.

[0111] This incremental data compression method significantly reduces storage requirements while preserving critical information.

[0112] The performance evaluation unit 92 calculates key performance indicators (KPIs): positioning accuracy, path smoothness, task completion time, etc. Long-term performance trends are calculated using a sliding average method. Detect performance anomalies: If the KPI is less than 2 standard deviations of the historical average, in-depth analysis is triggered. In order to comprehensively evaluate system performance, comprehensive performance indicators are introduced:

[0113]

[0114] Among them, Accuracy represents positioning and perception accuracy, Efficiency represents task completion efficiency, Safety represents safety index, and Adaptability represents environmental adaptability. Dynamically adjust according to the application scenario.

[0115] The model update unit 93 adopts a federated learning framework, specifically a combination of local updates and central aggregation. Local updates are performed after every 10 tasks or 1 hour. The FedAvg algorithm is used for model aggregation, and the communication frequency is once a day. Differential privacy technology is applied to protect data security and noise level. .

[0116] Specifically, the local update process is as follows: Collect local data ; Calculate the gradient: ; Add noise: ,in For sensitivity, represents a Gaussian distribution; update the model parameters: , This model update method based on federated learning can achieve continuous optimization of the model while protecting privacy.

[0117] The parameter optimization unit 94 uses Bayesian optimization to adjust the system hyperparameters. The optimization goal is to maximize the comprehensive performance index SP1. Gaussian process regression is used to establish the proxy model. The acquisition function selects the expected improvement (El), and the exploration coefficient . Global parameter optimization is performed once a week, with 50 rounds of iterations each time. In order to improve the optimization efficiency, a multi-fidelity Bayesian optimization strategy is introduced: define a low-fidelity evaluation function and high fidelity evaluation function ;use Quickly explore parameter space and select potential good parameters; use Conduct accurate assessments; combine and The Gaussian process model is updated based on the results of the multi-fidelity optimization strategy, which can find excellent parameter configurations more quickly with limited computing resources.

[0118] Through the close collaboration of the above modules, the autonomous robot decision-making system of the present invention can achieve efficient and reliable decision-making and control in complex dynamic environments. The core innovation of the system lies in the multimodal data fusion module, which realizes the efficient fusion of heterogeneous data through technologies such as adaptive time alignment, cross-modal spatial attention, temporal graph convolution and dynamic feature aggregation. At the same time, the online learning and adaptation module of the system ensures the stability and performance improvement of long-term operation.

[0119] The method of the present invention can be summarized as the following steps:

[0120] Step S1: Obtain multimodal perception data of the environment. Specifically, the multimodal data in the environment are collected through various sensors of the multimodal data acquisition module 1, including a visual camera 11, a depth camera 12, a laser radar 13, an IMU 14, a GPS receiver 15, and an acoustic sensor 16. Preferably, the PTP protocol is used to implement hardware-level time synchronization to ensure the time consistency of the data. For example, the GPS module can be set as the main clock source, and other sensors are synchronized with it, allowing a maximum time error of ±1ms.

[0121] Step S2: Preprocess the multimodal sensing data. This step is completed by the data preprocessing module 2, including image preprocessing, point cloud preprocessing, IMU data preprocessing, GPS data preprocessing, and acoustic data preprocessing. For example, for image data, bilateral filtering is first applied for denoising, and then camera calibration and distortion correction are performed based on Zhang Zhengyou's checkerboard method, and finally the gray world algorithm is used for automatic white balance. For point cloud data, downsampling, ground segmentation, and outlier removal are performed. Through these preprocessing steps, the efficiency and accuracy of subsequent processing can be significantly improved.

[0122] Step S3: Extract features from the preprocessed multimodal perception data. Feature extraction module 3 is responsible for this step, and different feature extraction methods are used for data of different modalities. For example, for visual data, a pre-trained ResNet50 model is used to extract features, and then spatial pyramid pooling is applied to generate a fixed-size feature vector. For point cloud data, VoxelNet is used for feature extraction. This modality-specific feature extraction method can make full use of the characteristics of various data and extract the most representative features.

[0123] Step S4: construct a dynamic spatiotemporal graph based on the features. The dynamic spatiotemporal graph construction module 4 completes this step, including node generation, edge generation and graph structure optimization. First, nodes are created for the features of each moment and each mode. Then, edges between nodes are created based on time and space relationships, including temporal edges, spatial edges and long-range edges. Finally, by dynamically adjusting the graph structure, redundant connections are removed and the density of the graph is kept at around 20%. This dynamic spatiotemporal graph structure can effectively capture the complex spatiotemporal relationships between data.

[0124] Step S5: Fusing the multimodal data in the dynamic spatiotemporal graph. This is the core step of the present invention, which is completed by the multimodal data fusion module 5. This step includes the following sub-steps:

[0125] S5.1: Adaptive time alignment. By minimizing the time alignment loss function , to achieve accurate time alignment of different modal data. Preferably, the Adam optimizer is used, the learning rate is set to 0.001, and it is iterated 50 times. At the same time, the constraints are introduced such as , to ensure the rationality of the alignment.

[0126] S5.2: Cross-modal spatial attention calculation. By calculating the attention score , capturing the spatial dependencies between features of different modalities. To improve efficiency, only the top-k attention values ​​can be retained. .

[0127] S5.3: Temporal graph convolution. Apply graph convolution operations , modeling the temporal evolution of features. Preferably, three layers of graph convolution are stacked, each followed by batch normalization, and the LeakyReLU activation function is used.

[0128] S5.4: Dynamic feature aggregation.

[0129] By calculating the dynamic weight , to achieve adaptive fusion of multimodal information. At the same time, the gating mechanism is introduced , enhancing the expressive power of the model. Through these sub-steps, efficient fusion of heterogeneous data is achieved, providing a reliable basis for subsequent decision-making.

[0130] Step S6: Perform environmental understanding and robot state estimation based on the fused data. The environmental understanding and state estimation module 6 is responsible for this step, including target detection, semantic segmentation and state estimation. For example, the YOLOv5s model is used for target detection, DeepLabv3+ for semantic segmentation, and the error state extended Kalman filter (ES-EKF) is used for state estimation. Preferably, an adaptive noise covariance estimation method is used to dynamically adjust the filter parameters to improve the accuracy and robustness of state estimation.

[0131] Step S7: Make decisions and plan paths based on the environment understanding and state estimation results. The decision-making planning module 7 completes this step, including task planning, path planning, and trajectory optimization. For example, the hierarchical task network (HTN) is used for task decomposition, the RRT* algorithm generates the global path, and the model predictive control (MPC) is used for trajectory optimization. To improve planning efficiency, adaptive sampling strategies and adaptive weight adjustment strategies can be introduced.

[0132] Step S8: Execute the decision and path planning results. The control execution module 8 is responsible for this step, including trajectory tracking, dynamic constraint processing and safety supervision. For example, a pure tracking algorithm is used for trajectory tracking, considering the maximum speed and angular velocity limits, and setting a safety buffer to avoid collisions. Preferably, an adaptive foresight distance adjustment strategy and dynamic stability constraints are used to improve the accuracy and safety of control.

[0133] Step S9: Optimize each step online based on the system operation data. The online learning and adaptation module 9 completes this step, including data collection, performance evaluation, model update and parameter optimization. For example, use a sliding window to store the data of the last hour, calculate key performance indicators (KPIs) to evaluate system performance, use a federated learning framework to update the model, and use Bayesian optimization to adjust system hyperparameters. Preferably, a multi-fidelity Bayesian optimization strategy is introduced to improve the efficiency of parameter optimization.

[0134] By repeating the above steps, the method of the present invention can realize the continuous learning and adaptation of autonomous robots in complex dynamic environments. The core advantage of this method lies in the efficient fusion of multimodal data and the online optimization capability of the system, which enables robots to make intelligent and safe decisions in various complex environments.

[0135] It is worth noting that the method of the present invention is highly flexible and scalable. For example, in step S1, the sensor types can be increased or decreased according to the specific application scenario. In the multimodal data fusion of step S5, the fusion strategy can be adjusted according to actual needs. This flexibility enables the method to adapt to various autonomous robot application scenarios.

[0136] In order to further illustrate the technical solution of the present invention, a detailed description is given below in conjunction with specific embodiments.

[0137] Example 1

[0138] In a smart warehousing scenario, the autonomous robot decision-making system of the present invention is applied to an automatic guided vehicle (AGV), which needs to perform autonomous navigation, obstacle avoidance, and cargo handling in a complex warehouse environment.

[0139] In step S1, the AGV is equipped with the following sensors:

[0140] 1) Binocular stereo camera (4K resolution, 60fps);

[0141] 2) 16-line laser radar (360° horizontal field of view, 30° vertical field of view);

[0142] 3) 9-axis IMU (sampling rate 1kHz);

[0143] 4) UWB positioning system (replaces GPS, indoor positioning accuracy ±10cm);

[0144] 5) 8-microphone array (sampling rate 48kHz).

[0145] In step S2, the collected data is preprocessed. For example, the lidar data is downsampled and the voxel size is set to 0.05m; the IMU data is zero-biased and compensated in real time using a temperature-bias model.

[0146] In step S3, features are extracted. For visual data, YOLOv5s is used for target detection to extract the location and category information of targets such as shelves, goods, and people. For lidar data, PointNet++ is used to extract geometric features.

[0147] In step S4, a dynamic spatiotemporal graph is constructed. Nodes represent features of different modes at different times, and edges represent spatiotemporal relationships between nodes. For example, for detected moving objects (such as other AGVs or staff), temporal edges are established to describe their motion trajectories.

[0148] In step S5, multimodal data fusion is performed. This step is particularly important because in a warehousing environment, single-modal data is often insufficient to accurately describe complex scenes. For example:

[0149] S5.1: Adaptive time alignment. Due to the different sampling frequencies of the visual camera and the lidar (60Hz vs 10Hz), accurate time alignment is required. Using the proposed adaptive time alignment algorithm, the time deviation is controlled within 5ms.

[0150] S5.2: Cross-modal spatial attention calculation. Calculate the attention weights between visual features and lidar features. For example, for the shelf area, more attention may be paid to visual features (for identifying product labels), while more attention may be paid to lidar features (for precise positioning and obstacle avoidance) in the open area.

[0151] S5.3: Temporal graph convolution. Through graph convolution operations, the motion patterns of objects in the environment are captured. For example, the common movement paths of workers can be learned, which helps AGV plan avoidance strategies in advance.

[0152] S5.4: Dynamic feature aggregation. Dynamically adjust the weights of different modal features based on the current task (such as navigation or pickup). For example, when performing precise docking and pickup, the weight of visual features may be increased.

[0153] In step S6, the environment understanding and state estimation are performed based on the fused data. For example, a semantic map of the warehouse is generated, including information such as shelf location, aisle width, dynamic obstacles, etc. At the same time, UWB data and IMU data are used to accurately locate the AGV itself.

[0154] In step S7, decision making and path planning are performed. For example, the A* algorithm is used to perform global path planning on the semantic map, and then the dynamic window method is used to perform local obstacle avoidance. During the planning process, special constraints of the warehouse are considered, such as one-way channels, no-U-turn areas, etc.

[0155] In step S8, the planning result is executed. A pure tracking algorithm is used for path tracking, and the forward viewing distance (1.5m to 3m) is dynamically adjusted to adapt to different scenarios. At the same time, the stability of the AGV is monitored in real time to prevent tipping during high-speed turns.

[0156] In step S9, online learning and optimization are performed. For example, the efficiency data of the AGV during the task execution (such as average speed, task completion time, etc.) is recorded, and the path planning strategy is optimized using the reinforcement learning method. At the same time, sensor data and manually annotated data are collected, and the target detection model is updated regularly to adapt to the new types of goods in the warehouse.

[0157] Through the above steps, AGV can achieve efficient and safe autonomous navigation and operation in complex warehouse environments. The system can adapt to changes in warehouse layout, cope with dynamic obstacles, and continuously optimize its performance.

[0158] It is worth noting that the method of the present invention is highly flexible and scalable. For example, in step S1, the sensor types can be increased or decreased according to the specific application scenario. In the multimodal data fusion of step S5, the fusion strategy can be adjusted according to actual needs. This flexibility enables the method to adapt to various autonomous robot application scenarios.

[0159] Example 2

[0160] In a smart home scenario, the autonomous robot decision-making system of the present invention is applied to a home service robot, which needs to perform tasks such as autonomous navigation, object recognition, and human-machine interaction in a home environment.

[0161] In step S1, the home service robot is equipped with the following sensors:

[0162] 1) RGB-D camera (1080p resolution, 30fps, depth measurement range 0.5m-5m);

[0163] 2) 360° 2D LiDAR (measurement range 12m);

[0164] 3) 6-axis IMU (sampling rate 200Hz);

[0165] 4) Omnidirectional microphone array (6 microphones, sampling rate 16kHz);

[0166] 5) Touch sensors (located on the robot head and arms).

[0167] In step S2, the collected data is preprocessed, for example, RGB-D data is registered and denoised to remove invalid points in the depth map, and microphone array data is beamformed to improve the signal-to-noise ratio of the speech signal.

[0168] In step S3, features are extracted. For RGB images, EfficientDet is used for object detection and instance segmentation to identify objects such as furniture, electrical appliances, and daily necessities. For depth images, FPFH features are extracted for scene matching. For speech data, MFCC features are extracted for speech recognition.

[0169] In step S4, a dynamic spatiotemporal graph is constructed. The nodes not only represent sensor data, but also include prior information extracted from the knowledge graph, such as the common locations of items, how to use them, etc. This graph structure that combines sensory data and prior knowledge helps the robot better understand the home environment.

[0170] In step S5, multimodal data fusion is performed. In a home environment, data from different modalities often provide complementary information, and fusing this information is crucial to completing complex tasks. For example:

[0171] S5.1: Adaptive time alignment. Due to the different sampling frequencies of the RGB-D camera and LiDAR (30Hz vs 40Hz), precise time alignment is required. Using the proposed adaptive time alignment algorithm, the time deviation is controlled within 3ms. This is particularly important for accurately locating moving objects (such as pets).

[0172] S5.2: Cross-modal spatial attention calculation. Calculate the attention weights between visual features, deep features, and voice features. For example, when the robot is executing a voice command to bring an apple from the table, it will pay more attention to the apple detection results in the visual features and the desktop location information in the deep features.

[0173] S5.3: Temporal graph convolution. Through graph convolution operations, the activity patterns of objects and people in the home environment are captured. For example, the daily activity patterns of family members can be learned, such as often being in the kitchen in the morning and watching TV in the living room at night. This helps the robot predict the location and needs of people and provide more intelligent services.

[0174] S5.4: Dynamic feature aggregation. Dynamically adjust the weights of different modal features according to the current task. For example, when grasping an object, the weights of visual and depth features will be increased; when conducting human-computer dialogue, the weights of speech features will be increased. Use the attention mechanism to calculate the weights:

[0175] ,in are the characteristics of each mode, It is a global feature.

[0176] In step S6, the environment is understood and the state is estimated based on the fused data. For example, a 3D semantic map of the home is generated, including information such as furniture location, room layout, and item placement. At the same time, the robot is accurately positioned using visual odometer technology combined with IMU data, with a positioning accuracy of ±1cm.

[0177] In step S7, decision making and path planning are performed. For example, a hierarchical task planning method is used to process complex instructions. For a task such as taking an apple from a table, it can be decomposed into the following subtasks: 1) locate the table, 2) navigate to the vicinity of the table, 3) identify and locate the apple, 4) plan the grasping action, 5) perform the grasping, and 6) navigate back to the user's location. When planning the path, the RRT* algorithm is used to perform global planning on the 3D semantic map, taking into account the placement of furniture and the activity areas of people, and avoiding passing through inappropriate areas (such as under the dining table).

[0178] In step S8, the planning result is executed. The model predictive control (MPC) method is used for trajectory tracking, and the prediction time domain is set to 3s. Considering the soft obstacles that may exist in the home environment (such as curtains, carpets, etc.), an adaptive impedance control strategy is introduced. The control law can be expressed as:

[0179]

[0180] in, is the inertia matrix, are the Coriolis force and the centrifugal force, is the gravity term, and are the position gain matrix and the velocity gain matrix respectively, is the position vector of each joint of the robot, , is the number of joints, is the joint acceleration vector, that is The second-order derivative of is the joint velocity vector, that is The first derivative of is the desired position vector, is the desired velocity vector.

[0181] Dynamically adjust according to the detected environmental stiffness and , achieving smooth interactive control.

[0182] In step S9, online learning and optimization are performed. For example, the success rate and efficiency data of the robot in performing various household tasks are recorded, and the A3C (Asynchronous Advantage Actor-Critic) algorithm is used to optimize the task execution strategy. At the same time, through daily interaction with users, the object recognition model and speech recognition model are continuously updated and expanded. Specifically, the incremental learning method is adopted, such as the use of Few-shot learning technology in the object recognition task, which can quickly learn to recognize new objects.

[0183] Preferably, this embodiment also introduces the following innovations:

[0184] 1. Multimodal interactive interface: Combining voice, gestures and touch screens to provide a more natural way of human-computer interaction. For example, users can give preliminary instructions through voice, and then make fine adjustments through gestures or touch screens.

[0185] 2. Context-aware decision-making: Based on the fused multimodal data, the robot can understand the current family context (such as whether there are guests present, whether there are special activities, etc.) and adjust its behavior accordingly. For example, during a family gathering, the robot will automatically reduce its movement speed and volume.

[0186] 3. Personalized service adaptation: Through long-term learning, the robot can adapt to the personal preferences of different family members. For example, for the instruction to pour a glass of water, the robot will choose the appropriate water temperature and container according to the habits of different family members.

[0187] 4. Anomaly detection and emergency response: Taking advantage of multimodal data fusion, the robot can more accurately detect abnormal situations in the home environment (such as falls, abnormal sounds, etc.). Once an abnormality is detected, the emergency response process is immediately initiated, such as notifying family members or calling emergency services.

[0188] Through the above steps and innovations, the home service robot can provide intelligent, safe and personalized services in a complex home environment. The system can adapt to the dynamic changes of the home environment, understand the complex human-machine interaction intentions, and continuously optimize its service quality.

[0189] The autonomous robot decision-making system and method of the present invention realizes efficient decision-making and control in complex dynamic environments through multimodal perception fusion technology. The system has the following advantages:

[0190] 1. High adaptability: Able to cope with various complex environments and mission scenarios, from industrial warehousing to home services.

[0191] 2. Strong robustness: Through multimodal data fusion, the impact of single sensor failure is reduced.

[0192] 3. High degree of intelligence: Combining deep learning and traditional control methods, it achieves advanced environmental understanding and decision-making capabilities.

[0193] 4. Continuous learning capability: Through online learning and adaptation modules, the system is able to continuously optimize its performance.

[0194] 5. Safe and reliable: Introduce multi-layer security mechanism to ensure safe operation in various situations.

[0195] In summary, the present invention provides a comprehensive, efficient and intelligent solution for the application of autonomous robots in complex environments, and is expected to promote the widespread application of intelligent robot technology in various fields.

[0196] In order to verify the superiority of the present invention, the present invention conducted a series of comparative experiments. In these experiments, we selected a representative embodiment and three comparative examples and tested them in complex indoor environments. The experimental scene was set in a simulated smart home environment, including multiple rooms, various furniture and common household items. The test tasks included autonomous navigation, object recognition, voice command execution, etc.

[0197] Example 2 adopts the complete technical solution of the present invention, including core technologies such as multimodal data acquisition, adaptive data fusion, and dynamic decision planning. Comparative Example 1 adopts the traditional single visual SLAM method for navigation and positioning. Comparative Example 2 uses target detection and speech recognition based on deep learning, but there is no multimodal fusion mechanism. Comparative Example 3 adopts a multi-sensor fusion method, but uses a simple Kalman filter instead of the adaptive fusion algorithm proposed in the present invention.

[0198] We selected the following five key indicators for testing: positioning accuracy, navigation success rate, object recognition accuracy, command understanding accuracy, and system response time. These indicators can fully reflect the performance of the system in complex environments.

[0199] The positioning accuracy is tested by setting multiple marking points in the room, letting the robot move between these points, and then comparing the error between the robot's estimated position and the actual position. The navigation success rate is measured by setting multiple navigation tasks and recording the proportion of tasks successfully completed. The object recognition accuracy is measured by letting the robot recognize 50 common objects in the room and calculating the proportion of correct recognition. The command understanding accuracy is measured by sending 100 voice commands of different complexity to the robot and counting the proportion of correct understanding and execution. The system response time measures the average time interval from receiving the command to starting to execute the action.

[0200] After two weeks of testing, the present invention obtained the following results:

[0201] It can be clearly seen from the test results that Example 2 of the present invention has shown significant advantages in all test indicators. In particular, in terms of positioning accuracy, Example 2 has achieved centimeter-level accuracy, far exceeding other comparison methods. This is due to the adaptive multimodal fusion algorithm proposed in the present invention, which can effectively integrate multiple sensor data such as vision, depth and IMU, greatly improving the accuracy of positioning.

[0202] In terms of navigation success rate, Example 2 has a success rate of nearly 99%, demonstrating excellent environmental adaptability. This is mainly due to the dynamic decision-making planning module of the present invention, which can respond to various changes in complex environments in real time, such as moving people, pets, or temporarily placed items. In contrast, Example 1, which relies only on visual SLAM, often loses positioning or fails in path planning in complex environments.

[0203] The significant improvement in object recognition accuracy and command understanding accuracy fully demonstrates the superiority of the present invention in environmental understanding and human-computer interaction. Through the deep fusion of multimodal data, the system can more comprehensively understand environmental information and user intentions. For example, when recognizing complex instructions such as "bring the red apple from the table", the system can simultaneously use visual, depth and voice information, greatly improving the accuracy of understanding and execution.

[0204] The significant reduction in system response time reflects the optimization of the present invention in terms of computing efficiency. Despite the integration of multiple sensor data and complex decision algorithms, Example 2 still achieves a fast response of less than 80ms through optimized data processing flow and efficient parallel computing, which is crucial for real-time human-computer interaction.

[0205] It is worth mentioning that during the test, we also observed that Example 2 showed excellent robustness and adaptability. For example, when we deliberately blocked part of the sensor or changed the room lighting conditions, Example 1 was able to maintain stable performance, while the performance of other comparison methods dropped significantly. This once again proves the important role of multimodal fusion technology in improving system reliability.

[0206] In summary, the autonomous robot decision-making system based on multimodal perception fusion proposed in this invention has shown comprehensive performance advantages in complex environments. It not only performs well in basic functions such as precise positioning and reliable navigation, but also has made significant progress in advanced environmental understanding and human-computer interaction. These advantages enable the system to better adapt to various complex application scenarios and provide strong support for the development of smart homes, service robots and other fields.

[0207] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modification, replacement, and improvement made within the principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An autonomous robot decision-making system based on multimodal perception fusion, characterized in that: include: A multimodal data acquisition module, used to obtain multimodal perception data of the environment; A data preprocessing module, connected to the multimodal data acquisition module, for preprocessing the multimodal sensing data; A feature extraction module, connected to the data preprocessing module, for extracting features from the preprocessed multimodal perception data; A dynamic space-time graph construction module, connected to the feature extraction module, for constructing a dynamic space-time graph based on the features; a multimodal data fusion module, connected to the dynamic space-time graph construction module, for fusing multimodal data in the dynamic space-time graph; An environment understanding and state estimation module, connected to the multimodal data fusion module, for performing environment understanding and robot state estimation based on the fused data; a decision planning module, connected to the environment understanding and state estimation module, for performing decision making and path planning based on the environment understanding and state estimation results; A control execution module, connected to the decision-making and planning module, for executing the planning result of the decision-making and planning module; and an online learning and adaptation module, which is connected to the control execution module and is used to perform online optimization on each module based on system operation data. The multimodal data fusion module includes: an adaptive time alignment unit, which is used to perform time alignment on data of different modalities; a cross-modal spatial attention unit, which is used to capture the spatial dependency between features of different modalities; a temporal graph convolution unit, which is used to model the temporal evolution of features; and a dynamic feature aggregation unit, which is used to adaptively fuse multimodal information.

2. The system according to claim 1, characterized in that The multimodal data acquisition module includes: a visual camera for obtaining RGB image data; a depth camera for obtaining depth map data; LiDAR, used to obtain 3D point cloud data; Inertial Measurement Unit IMU, used to obtain acceleration and angular velocity data; Global Positioning System GPS receiver, used to obtain location data; and Acoustic Sensor, used to obtain sound data.

3. The system according to claim 1, characterized in that The data preprocessing module comprises: An image preprocessing unit is used to perform denoising, distortion correction and color balance on image data; a point cloud preprocessing unit is used to downsample, segment the ground and remove outliers on point cloud data; an IMU data preprocessing unit is used to perform zero bias correction and integral drift compensation on IMU data; a GPS data preprocessing unit is used to perform satellite signal filtering and coordinate conversion on GPS data; and an acoustic data preprocessing unit is used to perform denoising and spectrum analysis on sound data.

4. The system according to claim 1, characterized in that The feature extraction module comprises: A visual feature extraction unit is used to extract visual features from image data; a depth feature extraction unit is used to extract depth features from depth map data; a point cloud feature extraction unit is used to extract geometric features from point cloud data; an IMU feature extraction unit is used to extract motion features from IMU data; a GPS feature extraction unit is used to extract position and velocity features from GPS data; and an acoustic feature extraction unit is used to extract acoustic features from sound data.

5. The system according to claim 1, characterized in that The adaptive time alignment unit realizes the time alignment of data by minimizing the time alignment loss function, wherein the time alignment loss function is: ， in, and For Node and timestamp, is the predicted time deviation, is the adaptive weight, ,in is the total number of nodes in the dynamic space-time graph.

6. The system according to claim 1, characterized in that The cross-modal spatial attention unit calculates the attention score by the following formula: ， in, and are query vector and key vector respectively, represents the matrix transpose operation, It refers to the dimension of the feature vector after feature extraction. For different modes, their features are uniformly mapped to the same dimension d, and the specific value is 256 dimensions.

7. The system according to claim 1, characterized in that The timing graph convolution unit defines the timing graph convolution operation by the following formula: , in, for The adjacency matrix of order, is the weight matrix, is a learnable parameter, is the activation function, ,in is the length of the time series, the default setting ; , represents the order of adjacency relationship considered, Represents directly connected nodes. represents nodes connected through an intermediate node, Represents nodes connected through two intermediate nodes.

8. The system according to claim 1, characterized in that The dynamic feature aggregation unit implements feature aggregation through the following formula: ， in, , is the number of modes, , corresponding to six different sensor modalities, Indicates The weight coefficient of each mode, Indicates The characteristics of a mode, Indicates the corresponding A feature transformation network of two modalities.

9. A method for an autonomous robot decision-making system based on multimodal perception fusion as described in any one of claims 1 to 8, characterized in that: The following steps are involved: Obtain multimodal perception data of the environment; Preprocessing the multimodal sensing data; extracting features from the preprocessed multimodal sensing data; Constructing a dynamic space-time graph based on the features; fusing multimodal data in the dynamic space-time graph; Based on the fused data, the environment is understood and the robot state is estimated; based on the environment understanding and state estimation results, decisions and path planning are made; the decision and path planning results are executed; and each step is optimized online based on the system operation data.

Images