Multi-mode perception fusion industrial robot environment understanding system and method

Abstract

The invention discloses a multi-modal sensing fusion industrial robot environment understanding system, which comprises a multi-modal sensing module used for collecting multi-source data of an environment; the data preprocessing module is used for carrying out denoising, registration and standardization preprocessing on the multi-source data to obtain multi-modal data; the feature extraction module is used for performing feature extraction on the preprocessed different modal data through an end-to-end trained deep learning model and outputting a multi-modal feature set; the multi-modal fusion module is used for receiving the multi-modal feature set and generating a unified environment characterization vector; and the environment understanding module is used for deciding the control module based on the environment characterization vector and is used for receiving a user instruction or a task target. According to the method, deep learning models such as CNN and RNN are utilized, deep and robust features in all modal data can be automatically learned and extracted, and complexity and limitation of manual feature design are avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial robots, more particularly to a multi-modal perception fusion industrial robot environment understanding system and method. BACKGROUND

[0002] At present, with the development of industrial automation, industrial robots are increasingly widely used in intelligent manufacturing. However, traditional industrial robots usually rely on preset programs or single sensors for environment perception, which has the following problems: Single perception information: only relying on single modal data such as vision or force perception, it is difficult to cope with complex environmental changes. Although increasing the types of sensors is known, how to design a fusion architecture that can seamlessly fuse visual, radar, time series and other multi-modal data and maintain the continuity of environment understanding is a problem that has not been solved by the prior art; Limited environmental understanding: lack of deep fusion of multi-source information, resulting in insufficient adaptability of the robot to dynamic environments; Low decision-making accuracy: due to insufficient perception data, the robot is prone to errors when performing tasks, affecting production efficiency and quality.

[0003] Therefore, how to provide a system and method that can fuse multi-modal perception data and achieve efficient environment understanding is a problem that needs to be solved by those skilled in the art. SUMMARY

[0004] In view of the above problems, the present application is proposed to provide a multi-modal perception fusion industrial robot environment understanding system and method which overcomes the above problems or at least partially solves the above problems In order to achieve the above purpose, the present application adopts the following technical solutions: In a first aspect, the present application provides a multi-modal perception fusion industrial robot environment understanding system; comprising a multi-modal perception module for collecting multi-source data of the environment, the multi-source data comprising visual data, laser radar point cloud data and voice data, the multi-modal perception module comprising a visual sensor, a laser radar sensor and a voice input sensor; a data preprocessing module for preprocessing the multi-source data by denoising, registration and standardization to obtain multi-modal data; a feature extraction module for extracting features from different modal data after preprocessing by an end-to-end trained deep learning model, outputting a multi-modal feature set; wherein the spatial hierarchical features are extracted from the visual modal data by a convolutional neural network, the time series features are extracted from the time series modal data by a recurrent neural network or its variants, and the geometric features are extracted from the laser radar point cloud data by a geometric feature extraction model; A multi-modal fusion module is configured to receive a multi-modal feature set, adopt a fusion strategy based on an attention mechanism and a graph neural network, and generate a unified environment representation vector; the fusion strategy includes feature alignment, attention weight calculation, multi-modal semantic graph construction, graph attention propagation, and global representation generation; An environment understanding module is configured to generate a semantic-geometric joint feature tensor through cross-modal hierarchical fusion based on the environment representation vector, perform scene segmentation, target recognition and tracking, target pose estimation, and dynamic obstacle motion intention recognition in parallel, and construct a structured environment semantic graph. A decision control module is configured to receive a user instruction or a task target, analyze a behavior intention in combination with the structured environment semantic graph, generate an obstacle avoidance trajectory through multi-level motion planning, and drive an industrial robot to perform an operation through a motion control unit.

[0005] Further, the visual sensor is a binocular stereo vision system composed of two Intel RealSense D455 depth cameras; The laser radar sensor is a SICK TIM561 2D laser radar; The voice input sensor is a ring-shaped 8-microphone array, and the MEMS microphones are uniformly distributed on the head of the robot.

[0006] Further, the convolutional neural network is a ResNet-50 model, the variant of the recurrent neural network is an LSTM or a BiLSTM, and the geometric feature extraction model is a PointNet++. The feature extraction of the visual modal data maps the original image into a 2048-dimensional global feature vector, the feature extraction of the time series modal data outputs a 512-dimensional time series feature vector, and the feature extraction of the laser radar point cloud data outputs a 512-dimensional geometric feature vector.

[0007] Further, the feature alignment unifies the features of each modality to a 1024-dimensional hidden space through linear mapping or a multi-head attention mechanism; The attention weight calculation adopts scaled dot attention; The nodes of the multi-modal semantic graph are the features of each modality or sub-modality instances thereof, and the edges are fully connected edges or gated edges, the weight of the gated edge is determined by the initial feature similarity, and only the edges with a similarity greater than a learnable threshold τ are retained; The graph attention propagation updates the node features through a graph attention layer; The global representation generation adopts a virtual node method, adds a virtual node with an initial zero vector, and is bidirectionally connected with all nodes, and takes the virtual node feature output by the third layer graph attention layer as the environment representation vector.

[0008] Further, the scene segmentation is implemented through a lightweight U-Net structure, and outputs a semantic label map including a ground, a wall, a device, and a person; Target recognition uses the YOLOv12 model; The target tracking uses the DeepSort algorithm; The target pose estimation is achieved by extracting a 17-point key point heatmap using HRNet-W32 and combining it with the PnP+RANSAC algorithm to achieve 6D pose fitting.

[0009] Furthermore, the multi-level motion planning includes: generating a cost map based on the scene segmentation results, and generating a global coarse path through an improved A* algorithm; A dynamic risk potential field is constructed by combining the target set and the dynamic obstacle set; Under the model predictive control framework, local obstacle avoidance trajectories are generated by optimizing the OSQP solver. The motion control unit uses an adaptive PID controller to compensate for wheel slip and IMU drift errors, and has a hardware-level emergency stop mechanism.

[0010] Secondly, embodiments of the present invention provide a method for understanding the environment of an industrial robot through multimodal perception fusion; comprising the following steps: S1: The multi-source data of the environment is collected synchronously through the multi-modal perception module. The multi-source data includes visual data, LiDAR point cloud data and voice data. S2: Perform preprocessing such as denoising, registration, and standardization on multi-source data to obtain robust multimodal data; S3: Through end-to-end trained deep learning models, feature extraction is performed on different preprocessed modal data to obtain a multimodal feature set; among them, spatial hierarchical features are extracted from visual modal data using convolutional neural networks, temporal features are extracted from temporal modal data using recurrent neural networks or their variants, and geometric features are extracted from lidar point cloud data using a geometric feature extraction model. S4: A fusion strategy based on attention mechanism and graph neural network is adopted to fuse multimodal feature sets and generate a unified environment representation vector; the fusion strategy includes feature alignment, attention weight calculation, multimodal semantic graph construction, graph attention propagation and global representation generation; S5: Based on the environment representation vector, a semantic-geometric joint feature tensor is generated through cross-modal hierarchical fusion, and scene segmentation, target recognition and tracking, target pose estimation and dynamic obstacle motion intent recognition are performed in parallel to construct a structured environment semantic graph; S6: Combines structured environment semantic graph parsing of behavioral intent, generates obstacle avoidance trajectory through multi-level motion planning, and drives industrial robot to perform operations.

[0011] Furthermore, the preprocessing process in step S2 includes: 5×5 Gaussian denoising, lens distortion correction based on intrinsic parameter matrix and distortion coefficient, contrast-limited adaptive histogram equalization, bilinear interpolation scaling to 640×480 and normalization to 0-1, and bilateral filtering to fill invalid depth values ​​in holes. LiDAR data preprocessing includes: range filtering, statistical outlier removal, polar coordinate to Cartesian coordinate conversion, 0.02m voxel downsampling, and projection of extrinsic parameters calibrated by hand and eye onto the camera coordinate system; Speech data preprocessing includes: adaptive echo cancellation, delay summation beamforming, speech activity detection, first-order high-pass filtering pre-emphasis, Hamming window framing with a frame length of 25ms + frame shift of 10ms, and extraction of 39-dimensional MFCC features.

[0012] Furthermore, in step S5, the dynamic obstacle motion intent recognition includes: acquiring the target trajectory in consecutive frames and calculating the velocity vector v in the robot's body coordinate system. k The relative vector, included angle, and radial velocity are used to determine the motion intention as approaching, moving away, or crossing.

[0013] Furthermore, in step S6, the behavioral intent parsing converts the voice command into text through Whisper-small ASR, and then maps it to navigation, operation or obstacle avoidance through a BERT-based fine-tuned intent classifier; when the classification confidence is less than a preset threshold, the interaction clarification mechanism is triggered.

[0014] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a multimodal perception fusion industrial robot environment understanding system, which has the following beneficial effects: This invention utilizes deep learning models such as CNN and RNN to automatically learn and extract deep, robust features from various modal data, avoiding the tediousness and limitations of manually designing features.

[0015] Through attention mechanisms and Generative Neural Networks (GNNs), the model can dynamically and adaptively evaluate the importance of features from different modalities and perform weighted fusion, rather than simply concatenating them. This effectively suppresses interference from noisy modalities, highlights key information, and generates a more discriminative unified representation. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0017] Figure 1This is a structural diagram of the environment understanding system provided in an embodiment of the present invention.

[0018] Figure 2 This is a flowchart of the environment understanding method provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] This invention discloses a multimodal perception fusion industrial robot environment understanding system, such as... Figure 1 As shown, it includes a multimodal perception module, a data preprocessing module, a feature extraction module, a multimodal fusion module, an environmental understanding module, and a decision control module; The multimodal perception module is the data acquisition source of the system, used to synchronously collect multi-source data in the industrial environment. The multi-source data includes visual data, lidar point cloud data and voice data, and the corresponding hardware carriers are visual sensors, lidar sensors and voice input sensors. The vision sensor employs two Intel RealSense D455 depth cameras to form a binocular stereo vision system. The two cameras are mounted parallel to each other on a dedicated bracket at the front of the robot, with a baseline distance of 150mm and an installation height of 1.2m above the ground. The pitch angle is 15° downwards to maximize coverage of the near-field operating area. Acquisition parameters are set as follows: RGB / depth image resolution 1280×720, frame rate 30Hz. It is connected to the computing host via a USB 3.0 interface and supports hardware-triggered synchronization.

[0021] The lidar sensor uses a SICK TIM561 2D lidar, mounted at the center of the top of the robot, with a horizontal deviation ≤0.5° and a scanning range of 270°. 135°~135°, angular resolution 0.33°, scanning frequency 15Hz, ranging range 0.05–10m, ranging accuracy ±30mm, access to the system via Gigabit Ethernet (TCP / IP), and supports PTP time synchronization.

[0022] The voice input sensor is a ring-shaped 8-microphone array, MEMS microphone, model: Knowles SPH0645LM4H-B; it is evenly distributed on the robot's head, with a height of about 1.5m and a radius of 5cm, and has 360° omnidirectional sound pickup capability, a signal-to-noise ratio of ≥65dBA, and is equipped with a dedicated audio processing card to support real-time beamforming, echo cancellation and sound source localization.

[0023] The data preprocessing module receives the raw data output from the multimodal perception module and performs modality-specific cleanup processes to eliminate noise interference, unify the data format, and achieve spatiotemporal alignment, resulting in robust multimodal data that provides high-quality input for subsequent feature extraction. Specific processing steps include denoising, registration, and standardization to ensure the consistency and usability of data from different modalities.

[0024] The feature extraction module uses an end-to-end trained deep learning model to extract deep and discriminative features based on the characteristics of different modal data, and finally outputs a multimodal feature set.

[0025] Specifically, convolutional neural networks are used to extract spatial hierarchical features from visual modal data; recurrent neural networks or their variants are used to extract temporal dependent features from temporal modal data; and geometric feature extraction models are used to extract spatial geometric features from lidar point cloud data.

[0026] In one embodiment, the convolutional neural network is a ResNet-50 model, which maps the preprocessed original image into a 2048-dimensional global feature vector. At the same time, it detects five high-confidence target regions (ROIs) using YOLOv12 and pools each ROI to obtain a 512-dimensional region feature. The recurrent neural network variant is LSTM or BiLSTM. After processing the MFCC sequence corresponding to the speech through three layers of BiLSTM (256 units / layer) and attention pooling, it outputs a 512-dimensional temporal feature vector and extracts two keyword embeddings from the ASR text. The geometric feature extraction model is PointNet++, which extracts global geometric features from the registered LiDAR point cloud, outputs a 512-dimensional geometric feature vector, and clusters it to obtain three key structural points.

[0027] The multimodal fusion module receives the multimodal feature set output by the feature extraction module and adopts a fusion strategy based on attention mechanism and graph neural network. Through steps such as feature alignment, attention weight calculation, multimodal semantic graph construction, graph attention propagation and global representation generation, it generates a unified and highly discriminative environmental representation vector, realizing deep interaction and complementarity of multimodal information.

[0028] In one embodiment, the feature alignment uses linear mapping or multi-head attention to uniformly map each modality feature to a 1024-dimensional latent space, eliminating dimensional differences between modalities. The attention weight calculation uses Scaled Dot-Product Attention to dynamically evaluate the importance of different features. The nodes of the multimodal semantic graph are instances of each modality feature or its submodality, and the edges are fully connected edges or gated edges. That is, the weight of the gated edge is determined by the initial feature similarity, and only edges with similarity greater than the learnable threshold τ are retained to achieve graph sparsity. The graph attention propagation updates node features through the graph attention layer and uses a shared weight matrix and attention vector to achieve feature interaction. The global representation generation uses the virtual node method, adding virtual nodes with an initial zero vector, establishing bidirectional connections with all nodes, and taking the virtual node features output by the third graph attention layer as the final environment representation vector with a dimension of 1024.

[0029] The environment understanding module takes the environment representation vector output by the multimodal fusion module as its core, and combines it with the original features to construct a unified semantic-geometric joint feature tensor. Through parallel task decoding, it simultaneously completes scene segmentation, target recognition and tracking, target pose estimation and dynamic obstacle motion intent recognition, and finally constructs a structured environment semantic graph to provide comprehensive and accurate environmental information support for decision control.

[0030] In one embodiment, scene segmentation is achieved using a lightweight U-Net structure, outputting pixel-level semantic label maps including the ground, walls, devices, and people; target recognition uses a YOLOv12 model with an input size of 640×640, an IOU threshold of 0.45, and a confidence threshold of 0.5; target tracking uses the DeepSort algorithm, with an ID association error of less than 2%, outputting the target's 6D pose ([R j ,t j ]∈SE(3)), category type (type) j ) and unique ID (ID) j The target pose estimation is achieved by extracting a 17-point key point heatmap using HRNet-W32 and combining it with the PnP+RANSAC algorithm to achieve 6D pose fitting, with an average error of less than 15mm and a rotation error of less than 2 degrees. The dynamic obstacle motion intent recognition is achieved by acquiring continuous frames of target trajectory and calculating the velocity vector v in the robot's body coordinate system. k The relative vector, included angle, and radial velocity are used to determine the motion intention as approaching, receding, or crossing; the structured environment semantic graph G={S,O,D}, where S is the static scene mesh, O is the target set {P} j ,type j}, where D is the set of dynamic obstacles {v k ,λ k}

[0031] The decision control module receives user instructions or task objectives, combines them with the structured environmental semantic graph output by the environmental understanding module, and generates drive instructions for the robot through behavioral intent parsing, multi-level motion planning and motion control, thereby driving the industrial robot to perform autonomous operations.

[0032] In one embodiment, the multi-level motion planning includes: generating a cost map based on scene segmentation results; generating a global coarse path using an improved A* algorithm; constructing a dynamic risk potential field by combining the target set and the dynamic obstacle set; under the model predictive control (MPC) framework, with a prediction time domain of N=10 and a sampling time of T=0.1s, solving the problem using an OSQP solver with a solution time of less than 8ms; optimizing and generating a local obstacle avoidance trajectory; the motion control unit adopts an adaptive PID controller with online parameter identification to compensate for roller skating and IMU drift errors, and has a hardware-level emergency stop mechanism with a response delay of ≤10ms. By monitoring the output of the voice front-end thread KWS, if a "stop" command is detected, a motor enable / disable signal is immediately output.

[0033] This invention utilizes deep learning models such as CNN and RNN to automatically learn and extract deep, robust features from various modal data, avoiding the tediousness and limitations of manually designing features.

[0034] Through attention mechanisms and Generative Neural Networks (GNNs), the model can dynamically and adaptively evaluate the importance of features from different modalities and perform weighted fusion, rather than simply concatenating them. This effectively suppresses interference from noisy modalities, highlights key information, and generates a more discriminative unified representation.

[0035] Based on the same inventive concept, embodiments of the present invention also provide a multimodal perception fusion method for understanding the environment of industrial robots, such as... Figure 2 As shown, the specific steps include: S1: Multi-source data of the environment are collected synchronously through the multi-modal perception module. The multi-source data includes visual data, lidar point cloud data and voice data. S2: Perform denoising, registration, and standardization preprocessing on the multi-source data to obtain robust multimodal data; S3: Through end-to-end trained deep learning models, feature extraction is performed on different preprocessed modal data to obtain a multimodal feature set; among them, spatial hierarchical features are extracted from visual modal data using convolutional neural networks, temporal features are extracted from temporal modal data using recurrent neural networks or their variants, and geometric features are extracted from lidar point cloud data using a geometric feature extraction model. S4: A fusion strategy based on attention mechanism and graph neural network is adopted to fuse the multimodal feature set to generate a unified environment representation vector; the fusion strategy includes feature alignment, attention weight calculation, multimodal semantic graph construction, graph attention propagation and global representation generation; S5: Based on the environmental representation vector, a semantic-geometric joint feature tensor is generated through cross-modal hierarchical fusion, and scene segmentation, target recognition and tracking, target pose estimation and dynamic obstacle motion intent recognition are performed in parallel to construct a structured environmental semantic graph; S6: Combining the structured environment semantic graph to parse the behavioral intent, generate obstacle avoidance trajectory through multi-level motion planning, and drive the industrial robot to perform operations.

[0036] In step S1, the multi-source data of the environment are collected synchronously through the multi-modal perception module. The multi-source data includes visual data, lidar point cloud data and voice data. The hardware synchronization mechanism ensures that the timestamp alignment error of each modal data is ≤10ms.

[0037] Step S2 performs preprocessing on the multi-source data, including denoising, registration, and standardization, to obtain robust multimodal data: Visual data preprocessing: 5×5 Gaussian denoising with standard deviation σ=0.8 is performed sequentially; lens distortion correction based on intrinsic parameter matrix K and distortion coefficient k=[k1,k2,p1,p2] is performed; contrast-limited adaptive histogram equalization (CLAHE) is performed with clipping threshold clip_limit=2.0 and grid size 8×8; bilinear interpolation is performed to scale to 640×480 and normalize to [0,1]; bilateral filtering is performed to fill invalid depth values ​​(i.e., 0 or >10m) in holes. LiDAR data preprocessing: Range filtering is performed sequentially to remove points with ρ < 0.1m or ρ > 10.0m; outlier removal is performed; and |ρi| is removed. Points where μ|>2σ, where μ is the mean distance, σ is the standard deviation, and polar coordinates are converted to Cartesian coordinates (x... i ,y i ) =(ρicosθi,ρisinθi) 1. Sampling down to 0.02m voxels, retaining the nearest point within each voxel, and projecting the point cloud onto the camera coordinate system by calibrating the extrinsic parameter Tcam∈SE(3) using hand-eye calibration; Speech data preprocessing: A dual-threaded parallel processing approach is adopted. The front-end real-time thread, with a latency ≤5ms, performs adaptive echo cancellation, delay-summing beamforming, and speech activity detection. The back-end processing thread performs first-order high-pass filtering pre-emphasis (H(z)=1). 0.97z 1) Hamming window framing (frame length 25ms, frame shift 10ms) and extraction of 39-dimensional MFCC features (13-dimensional static + 13-dimensional Δ + 13-dimensional ΔΔ).

[0038] After preprocessing, all modal data is published to the ROS synchronization topic / synced_data.

[0039] Step S3 uses an end-to-end trained deep learning model to extract features from different preprocessed modal data, obtaining a multimodal feature set. Specifically, a convolutional neural network is used to extract spatial hierarchical features from visual modal data, a recurrent neural network or its variant is used to extract temporal features from temporal modal data, and a geometric feature extraction model is used to extract geometric features from LiDAR point cloud data. The specific feature extraction method is consistent with the feature extraction module of the above system.

[0040] Step S4 employs a fusion strategy based on attention mechanism and graph neural network to fuse the multimodal feature set and generate a unified environment representation vector. The fusion strategy includes feature alignment, attention weight calculation, multimodal semantic graph construction, graph attention propagation, and global representation generation. The specific implementation method is consistent with the multimodal fusion module of the above system.

[0041] Step S5, based on the environmental representation vector, generates a semantic-geometric joint feature tensor through cross-modal hierarchical fusion, and performs scene segmentation, target recognition and tracking, target pose estimation, and dynamic obstacle motion intent recognition in parallel to construct a structured environmental semantic graph. The dynamic obstacle motion intent recognition includes: acquiring target trajectories in consecutive frames and calculating the velocity vector v in the robot's body coordinate system. k The relative vector, included angle, and radial velocity are used to determine the motion intention as approaching, moving away, or crossing; the specific form of the structured environment semantic map is consistent with the environment understanding module of the above system.

[0042] Step S6 combines the structured environment semantic graph to parse the behavioral intent, generates an obstacle avoidance trajectory through multi-level motion planning, and drives the industrial robot to perform operations. Specific implementation examples: This embodiment uses a six-degree-of-freedom collaborative industrial robot as a carrier, deployed in a typical assembly workshop environment. The core hardware configuration is as follows: Multimodal perception module: The preferred embodiment above uses the Intel RealSense D455 binocular stereo vision system, SICK TIM561 2D LiDAR, and a ring-shaped 8-microphone array. Computing platform: The core controller is an NVIDIA Jetson AGX Xavier, configured with 32GB LPDDR4x memory and a 1TB NVMe SSD; the operating system is Ubuntu 18.04 + ROS Melodic; real-time performance is guaranteed through the PREEMPT_RT patch and CPU isolation strategy.

[0044] The software implementation process is as follows: 1. Data Acquisition: Start each sensor through the ROS driver, configure the acquisition parameters, and realize the synchronous acquisition and transmission of multimodal data; 2. Preprocessing module: Based on open-source libraries such as OpenCV, PCL (point cloud library), and FFmpeg, preprocessing algorithms for various modal data are implemented, encapsulated as ROS nodes, and can subscribe to raw data topics and publish preprocessed data topics; 3. Feature Extraction Module: Based on the PyTorch framework, this module enables the training and inference of models such as ResNet-50, BiLSTM, and PointNet++, loads pre-trained weights, receives pre-processed data, and outputs feature vectors. 4. Multimodal fusion module: Based on PyTorch, implements a joint fusion algorithm of attention mechanism and graph neural network to complete feature alignment, semantic graph construction, graph attention propagation and global representation generation; 5. Environment Understanding Module: Parallel decoding tasks are implemented based on lightweight U-Net, YOLOv12, DeepSort, HRNet-W32 and other models. 6D pose fitting is completed through PnP+RANSAC algorithm to construct a structured environment semantic graph. 6. Decision Control Module: Implements global path planning based on the improved A* algorithm using the ROS Navigation stack, implements MPC trajectory optimization based on the OSQP solver, generates motor drive commands through an adaptive PID controller, and integrates emergency stop and interactive clarification logic.

[0045] This embodiment conducts a real-world test in a typical assembly workshop scenario. The test environment includes elements such as the floor, walls, assembly equipment, workers, parts to be assembled, and dynamically moving material carts. The test results are as follows: 1. Target recognition accuracy: mAP@0.5=91.2%, which can accurately identify targets such as equipment, parts, and personnel;

[0046] 2. Dynamic obstacle intention recognition accuracy: 88.7%, which can accurately determine the movement intention (approaching / moving away / crossing) of material vehicles and personnel.

[0047] 3. End-to-end response latency: 92±3ms, meeting the real-time operation requirements of industrial robots;

[0048] 4. Attitude estimation error: average position error less than 15mm, rotation error less than 2 degrees;

[0049] 5. Operational safety: The emergency stop mechanism has a response time of ≤10ms, and no collisions or operational errors have occurred.

[0050]

[0051] The above experimental results show that the system and method of the present invention can stably and efficiently realize the understanding and autonomous operation of complex industrial environments, and have practical industrial application value.

[0052] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0053] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal perception fusion industrial robot environmental understanding system, characterized in that, include: A multimodal perception module is used to collect multi-source data of the environment, including visual data, lidar point cloud data and voice data. The multimodal perception module includes a visual sensor, a lidar sensor and a voice input sensor. The data preprocessing module is used to perform noise reduction, registration, and standardization preprocessing on the multi-source data to obtain multimodal data; The feature extraction module is used to extract features from preprocessed data of different modalities using an end-to-end trained deep learning model, and output a multimodal feature set. Specifically, a convolutional neural network is used to extract spatial hierarchical features from visual modal data, a recurrent neural network or its variant is used to extract temporal features from temporal modal data, and a geometric feature extraction model is used to extract geometric features from lidar point cloud data. A multimodal fusion module is used to receive the multimodal feature set and generate a unified environment representation vector by adopting a fusion strategy based on attention mechanism and graph neural network; the fusion strategy includes feature alignment, attention weight calculation, multimodal semantic graph construction, graph attention propagation and global representation generation; The environment understanding module is used to perform environment understanding tasks based on the environment representation vector. It generates a semantic-geometric joint feature tensor through cross-modal hierarchical fusion, and performs scene segmentation, target recognition and tracking, target pose estimation and dynamic obstacle motion intent recognition in parallel to construct a structured environment semantic graph. The decision control module is used to receive user instructions or task objectives, generate control instructions based on the environmental understanding results, analyze behavioral intentions based on the structured environmental semantic graph, generate obstacle avoidance trajectories through multi-level motion planning, and drive the industrial robot to perform operations through the motion control unit.

2. The multimodal perception fusion industrial robot environment understanding system as described in claim 1, characterized in that, The visual sensor is a binocular stereo vision system consisting of two Intel RealSense D455 depth cameras; The lidar sensor is a SICK TIM561 2D lidar; The voice input sensor is a ring-shaped 8-microphone array, with MEMS microphones evenly distributed on the robot's head.

3. The multimodal perception fusion industrial robot environment understanding system as described in claim 1, characterized in that, The convolutional neural network is a ResNet-50 model, the recurrent neural network variant is LSTM or BiLSTM, and the geometric feature extraction model is PointNet++. The feature extraction of the visual modality data maps the original image into a 2048-dimensional global feature vector, the feature extraction of the temporal modality data outputs a 512-dimensional temporal feature vector, and the feature extraction of the lidar point cloud data outputs a 512-dimensional geometric feature vector.

4. The multimodal perception fusion industrial robot environment understanding system as described in claim 1, characterized in that, The feature alignment unifies the features of each modality into a 1024-dimensional latent space through linear mapping or multi-head attention mechanism; The attention weight calculation uses scaling point attention; The nodes of the multimodal semantic graph are each modal feature or its submodal instance, and the edges are fully connected edges or gated edges. The weight of the gated edges is determined by the initial feature similarity, and only edges with similarity greater than the learnable threshold τ are retained. The graph attention propagation updates node features through the graph attention layer; The global representation generation adopts the virtual node method, which adds a virtual node with an initial zero vector, and connects it bidirectionally with all nodes. The virtual node features output by the third graph attention layer are taken as the environment representation vector.

5. The multimodal perception fusion industrial robot environment understanding system as described in claim 1, characterized in that, The scene segmentation is achieved through a lightweight U-Net structure, and the output includes semantic label maps of the ground, walls, devices, and people; The target recognition uses the YOLOv12 model; The target tracking uses the DeepSort algorithm; The target pose estimation is achieved by extracting a 17-point key point heatmap using HRNet-W32 and combining it with the PnP+RANSAC algorithm to achieve 6D pose fitting.

6. The multimodal perception fusion industrial robot environment understanding system as described in claim 1, characterized in that, The multi-level motion planning includes: generating a cost map based on scene segmentation results, and generating a global coarse path through an improved A* algorithm; A dynamic risk potential field is constructed by combining the target set and the dynamic obstacle set; Under the model predictive control framework, local obstacle avoidance trajectories are generated by optimizing the OSQP solver. The motion control unit uses an adaptive PID controller to compensate for wheel skating and IMU drift errors, and has a hardware-level emergency stop mechanism.

7. A multimodal perception fusion method for environmental understanding of industrial robots, characterized in that, Includes the following steps: S1: Multi-source data of the environment are collected synchronously through the multi-modal perception module. The multi-source data includes visual data, lidar point cloud data and voice data. S2: Perform denoising, registration, and standardization preprocessing on the multi-source data to obtain robust multimodal data; S3: Through end-to-end trained deep learning models, feature extraction is performed on different preprocessed modal data to obtain a multimodal feature set; among them, spatial hierarchical features are extracted from visual modal data using convolutional neural networks, temporal features are extracted from temporal modal data using recurrent neural networks or their variants, and geometric features are extracted from lidar point cloud data using a geometric feature extraction model. S4: A fusion strategy based on attention mechanism and graph neural network is adopted to fuse the multimodal feature set to generate a unified environment representation vector; the fusion strategy includes feature alignment, attention weight calculation, multimodal semantic graph construction, graph attention propagation and global representation generation; S5: Based on the environmental representation vector, a semantic-geometric joint feature tensor is generated through cross-modal hierarchical fusion, and scene segmentation, target recognition and tracking, target pose estimation and dynamic obstacle motion intent recognition are performed in parallel to construct a structured environmental semantic graph; S6: Combining the structured environment semantic graph to parse the behavioral intent, generate obstacle avoidance trajectory through multi-level motion planning, and drive the industrial robot to perform operations.

8. The multimodal perception fusion method for understanding the environment of an industrial robot as described in claim 7, characterized in that, The preprocessing process in step S2 includes: 5×5 Gaussian denoising, lens distortion correction based on intrinsic parameter matrix and distortion coefficient, contrast-limited adaptive histogram equalization, bilinear interpolation scaling to 640×480 and normalization to 0-1, and bilateral filtering to fill invalid depth values ​​in holes. LiDAR data preprocessing includes: range filtering, statistical outlier removal, polar coordinate to Cartesian coordinate conversion, 0.02m voxel downsampling, and projection of extrinsic parameters calibrated by hand and eye onto the camera coordinate system; Speech data preprocessing includes: adaptive echo cancellation, delayed summation beamforming, speech activity detection, first-order high-pass filtering pre-emphasis, Hamming window framing with a frame length of 25ms and a frame shift of 10ms, and extraction of 39-dimensional MFCC features.

9. The multimodal perception fusion method for industrial robot environmental understanding as described in claim 7, characterized in that, In step S5, the dynamic obstacle motion intent recognition includes: acquiring continuous frames of target trajectory and calculating the velocity vector v in the robot's body coordinate system. k The relative vector, included angle, and radial velocity are used to determine whether the motion intention is to approach, move away, or cross.

10. The multimodal perception fusion method for understanding the environment of an industrial robot as described in claim 7, characterized in that, In step S6, the behavioral intent parsing converts the voice command into text through Whisper-small ASR, and then maps it to navigation, operation or obstacle avoidance through a BERT-base fine-tuned intent classifier; when the classification confidence is less than a preset threshold, the interaction clarification mechanism is triggered.

Images