Distributed multi-source data fusion sensing system and method based on cloud side end

Abstract

The invention discloses a distributed multi-source data fusion sensing system and method based on a cloud side end, and the system comprises a data obtaining device which is used for obtaining the point cloud data and image data of an environment where a robot is located; the edge device is used for obtaining a target detection result according to the point cloud data, the image data and a target detection model; and the cloud server is used for carrying out training and performance evaluation on the target detection model through the data sent by the edge device, and when the performance of the trained target detection model is superior to that of the current target detection model, the target detection models in the cloud server and the edge device are updated through the trained target detection model. The real-time performance of edge equipment and the powerful computing power advantage of cloud computing are fully utilized, the system performance bottleneck risk and deployment cost are reduced, efficient and accurate environment perception and target detection are achieved, the performance of a target detection model is improved and continuously optimized and updated, and the method can be widely applied to the fields of logistics robots, storage robots, autonomous mobile robots and the like.

Description

Technical Field

[0001] The present invention relates to the field of robot environment perception and target detection technology, and in particular to a distributed multi-source data fusion perception system and method based on cloud-edge. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In the fields of robotic inspection or autonomous driving, precise perception and detection of the environment and targets are required. Currently, there are methods that use multi-sensor fusion technology for environmental perception and target detection. This method combines lidar and cameras to collect environmental information and then locates targets in the environment based on the information collected by both.

[0004] However, the above method requires all data collected by lidar and cameras to be transmitted to a single computing platform for processing. This centralized processing architecture has the following problems:

[0005] (1) The centralized processing of massive data requires extremely high computing and storage resources on a single computing platform, which can easily cause system performance bottlenecks and affect overall operational efficiency;

[0006] (2) When edge devices are used as computing platforms, the deployment cost of edge devices is high; when the cloud is used as a computing platform, the data transmission delay between radar, camera and the cloud is high. Since robots or self-driving vehicles are moving at high speeds, the surrounding environment changes rapidly and they need to perceive and make decisions quickly. However, the large amount of data transmitted to the cloud causes excessive delays and cannot meet real-time requirements.

[0007] Therefore, when the current multi-sensor fusion technology is used for environmental perception and target detection, it cannot take into account both real-time performance and computing performance. Summary of the Invention

[0008] In order to solve the above problems, the present invention proposes a distributed multi-source data fusion perception system and method based on cloud-edge, which improves the computing speed and accuracy of environmental perception and target detection.

[0009] To achieve the above object, the present invention adopts the following technical solutions:

[0010] First, a distributed multi-source data fusion perception system based on cloud, edge, and terminal is proposed, including:

[0011] A data acquisition device, used to acquire point cloud data and image data of the robot's environment and send the acquired data to the edge device;

[0012] Edge devices are used to perform feature fusion on point cloud data and image data to obtain feature fusion results; obtain target detection results based on the target detection model and feature fusion results; and send the point cloud data, image data, and feature fusion results to the cloud server;

[0013] The cloud server is used to annotate and fuse features of point cloud data and image data, obtain training data, use the training data to train the current target detection model, and obtain a trained target detection model; the performance of the trained target detection model and the current target detection model are detected through the feature fusion results; when the performance of the trained target detection model is better than the current target detection model, the current target detection model of the cloud server and edge device is updated through the trained target detection model.

[0014] Furthermore, the edge device is used to project the point cloud data onto the image data to obtain a sparse point cloud image;

[0015] Perform voxel processing on the point cloud data to obtain voxelized point cloud data;

[0016] Perform feature extraction on the voxelized point cloud data and convert the extracted features into BEV features to obtain point cloud features;

[0017] Extract image data features from sparse point cloud images;

[0018] Use point cloud features and image data in sparse point cloud images to perform depth estimation and obtain depth features;

[0019] Combine the depth features with the image data features to obtain the cone features;

[0020] Pooling is performed on the cone features to obtain image features;

[0021] Fuse point cloud features and image features to obtain aggregated features;

[0022] The aggregated features are concatenated with the point cloud features to obtain the feature fusion results.

[0023] Furthermore, the edge device uses a multi-layer perceptron to extract features from the voxelized point cloud data.

[0024] Furthermore, the edge device is also used to preprocess the point cloud data and image data to obtain preprocessed data; perform feature fusion on the preprocessed data to obtain feature fusion results; and send the preprocessed data and feature fusion results to the cloud server.

[0025] Furthermore, the edge device adopts a cross-attention mechanism to fuse point cloud features and image features to obtain aggregated features.

[0026] Furthermore, the edge device is used to perform target detection on the feature fusion result according to the target detection model to obtain a target detection result;

[0027] Determine control instructions based on target detection results;

[0028] Control the robot according to control instructions.

[0029] Furthermore, the target detection results obtained by the edge device include the relative position and distance between the target object and the robot; the determined control instructions include speed, angle and posture.

[0030] Furthermore, an encrypted data transmission protocol is used for data exchange between edge devices and cloud servers.

[0031] Furthermore, the cloud server is used to obtain the target detection result of the trained model based on the trained target detection model and the feature fusion result;

[0032] According to the current target detection model and feature fusion results, the target detection results of the current model are obtained;

[0033] Determine the performance of the trained target detection model through the target detection results of the trained model;

[0034] Determine the performance of the current target detection model based on the target detection results of the current model;

[0035] Determine the performance improvement of the trained target detection model over the current target detection model based on the performance of the trained target detection model and the performance of the current target detection model.

[0036] When the performance improvement exceeds the set threshold, it is determined that the performance of the trained target detection model is better than the current target detection model.

[0037] Secondly, a distributed multi-source data fusion perception method based on cloud-edge-device is proposed, including:

[0038] Acquire point cloud data and image data of the robot's environment through a data acquisition device, and send the acquired data to an edge device;

[0039] Perform feature fusion on point cloud data and image data through edge devices to obtain feature fusion results; obtain target detection results based on the target detection model and feature fusion results; send point cloud data, image data and feature fusion results to the cloud server;

[0040] The point cloud data and image data are annotated and feature-fused through the cloud server to obtain training data, and the training data is used to train the current target detection model to obtain a trained target detection model; the performance of the trained target detection model and the current target detection model are detected through the feature fusion results; when the performance of the trained target detection model is better than the current target detection model, the current target detection model of the cloud server and edge device is updated through the trained target detection model.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention proposes a distributed multi-source data fusion perception system and method based on cloud-edge. The system includes a data acquisition device, an edge device and a cloud server. The data acquisition device is used to acquire point cloud data and image data. The edge device is used to perform target detection based on the point cloud data, image data and target detection model; the cloud server is used to train and update the target detection model, giving full play to the real-time nature of the edge device and the powerful computing power advantages of cloud computing. The edge device is responsible for data processing tasks with high real-time requirements, while the cloud server undertakes data processing tasks with high computing resource requirements, effectively balancing computing resources and data processing efficiency in data fusion and perception problems, avoiding the problem of excessive pressure on a single computing platform in the traditional centralized processing architecture, reducing the risk of system performance bottlenecks, and also reducing deployment costs.

[0043] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings in the specification, which constitute a part of this application, are used to provide further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute improper limitations on this application.

[0045] Figure 1 This is a block diagram of a distributed multi-source data fusion perception system based on cloud, edge, and terminal disclosed in an embodiment;

[0046] Figure 2 This is a block diagram of the principle of a cloud-edge-based distributed multi-source data fusion perception system disclosed in the embodiment;

[0047] Figure 3 This is a schematic diagram of data layer fusion disclosed in the embodiment;

[0048] Figure 4 This is a schematic diagram of feature layer fusion disclosed in the embodiment. DETAILED DESCRIPTION

[0049] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0050] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present application belongs.

[0051] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0052] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0053] Example 1

[0054] In the fields of robotic inspection or autonomous driving, precise perception and detection of the environment and targets are required. In the early days, people relied on a single sensor for environmental perception, such as a laser radar or camera. However, single sensors have many limitations: while laser radar can provide highly accurate depth information and distance measurement data, it lacks visual features such as color and texture, making it difficult to identify the type and details of objects. While cameras can capture rich color and texture information, their depth perception capabilities are poor, making it difficult to directly obtain the precise location and distance of an object. These sensors cannot meet the demand for accurate perception of objects in complex environments.

[0055] To overcome the shortcomings of a single sensor, researchers have begun to explore multi-sensor fusion technology, combining lidar and cameras to achieve complementary advantages. This method uses a combination of lidar and cameras to collect environmental information, and then determines the target in the environment based on the information collected by both.

[0056] However, the above method requires all data collected by lidar and cameras to be transmitted to a single computing platform for processing. This centralized processing architecture has the following problems:

[0057] (1) The centralized processing of massive data requires extremely high computing and storage resources on a single computing platform, which can easily cause system performance bottlenecks and affect overall operational efficiency;

[0058] (2) When using edge devices as a computing platform, the deployment cost of edge devices is high; when using the cloud as a computing platform, the data transmission delay between the radar, camera and the cloud is high, and since the robot or autonomous vehicle is moving at high speed, the surrounding environment changes rapidly, which requires rapid perception and decision-making, and a large amount of data transmission to the cloud results in too large a delay, which cannot meet the real-time requirement.

[0059] Therefore, when the current multi-sensor fusion technology performs environment perception and target detection, the computing resources and data processing efficiency cannot be effectively balanced, and real-time and computing performance cannot be considered.

[0060] In order to overcome the shortcomings of the prior art, the embodiment provides a cloud-edge-end-based distributed multi-source data fusion perception system, which realizes efficient and accurate robot environment perception and 3D target detection, fully utilizes cloud-edge-end computing resources on the basis of ensuring real-time performance, improves target detection model performance, and realizes continuous optimization and update of the model.

[0061] The cloud-edge-end-based distributed multi-source data fusion perception system disclosed by the embodiment comprises: Figures 1-4 as shown in the figure, comprising:

[0062] A data acquisition device is configured to acquire point cloud data and image data of an environment in which a robot is located, and send the acquired data to an edge device.

[0063] The edge device is configured to perform feature fusion on the point cloud data and image data to obtain a feature fusion result, obtain a target detection result according to a target detection model and the feature fusion result, and send the point cloud data, image data and feature fusion result to a cloud server.

[0064] The cloud server is configured to perform labeling and feature fusion on the point cloud data and image data to obtain training data, train a current target detection model using the training data to obtain a trained target detection model, detect the performance of the trained target detection model and the current target detection model through the feature fusion result, and update the current target detection model of the cloud server and the edge device using the trained target detection model when the performance of the trained target detection model is better than that of the current target detection model.

[0065] The edge device and the cloud server use an encrypted data transmission protocol to interact with each other.

[0066] The data acquisition device comprises a laser radar and a camera, which acquires point cloud data through the laser radar and image data through the camera.

[0067] Both the LiDAR and camera are mounted on the robot. Adjust their positions to ensure maximum coverage. Typically, the LiDAR is placed on top of the robot to avoid obstruction due to the robot's structure, while the camera is positioned in front of the robot's forward direction. Secure the LiDAR and camera to prevent their relative positions from shifting due to robot movement. Jointly calibrate the LiDAR and camera to obtain calibration data.

[0068] During the robot's movement, the laser radar performs a 360-degree scan of the robot's surrounding environment at a preset scanning frequency and resolution to obtain point cloud data in the environment. The point cloud data is a tensor with a dimension of (num_points, num_channels), where num_points represents the number of scanning points, num_channels is 3, and P lidar Represents the three-dimensional coordinate information of each scanning point, P lidar =(x, y, z);

[0069] The camera captures the robot's surrounding environment at a preset frame rate and image resolution to obtain color image data, which includes the RGB value of each pixel;

[0070] The collected point cloud data and color image data are transmitted to the edge device in real time through the ultra-low latency wired communication port, ensuring that the transmission delay can meet the minimum requirements of the moving robot and ensure the integrity and accuracy of data transmission.

[0071] In order to improve the accuracy of target detection, this embodiment makes special provisions for the process of obtaining feature fusion results when edge devices perform target detection. Specifically:

[0072] Edge device, used to project point cloud data onto image data to obtain a sparse point cloud map;

[0073] Perform voxel processing on the point cloud data to obtain voxelized point cloud data;

[0074] Perform feature extraction on the voxelized point cloud data and convert the extracted features into BEV features to obtain point cloud features;

[0075] Extract image data features from sparse point cloud images;

[0076] Use point cloud features and image data in sparse point cloud images to perform depth estimation and obtain depth features;

[0077] Combine the depth features with the image data features to obtain the cone features;

[0078] Pooling is performed on the cone features to obtain image features;

[0079] Fuse point cloud features and image features to obtain aggregated features;

[0080] The aggregated features are concatenated with the point cloud features to obtain the feature fusion results.

[0081] Among them, the edge device uses the joint calibration data of the lidar and camera to project the collected point cloud data onto the image data to obtain an image containing depth information, which facilitates the subsequent steps of extracting point cloud features and image features from the sparse point cloud map.

[0082] First, the point cloud data P lidar Convert from radar coordinate system to camera coordinate system. The transformation from radar coordinate system to camera coordinate system can be expressed as:

[0083] P camera =R·P lidar +T

[0084] Among them, R represents the rotation matrix and T represents the translation vector. The two constitute the external parameters of the camera. lidar Convert from the three-dimensional camera coordinate system to the two-dimensional camera coordinate system to obtain the point P of the point cloud data in the camera coordinate system camera , P camera =(X, Y, Z), point P camera Projected to the plane coordinates (u, υ), it can be expressed as:

[0085]

[0086] Among them, K is the camera internal parameter matrix, which contains information such as focal length and principal point coordinates, and Z is the depth value of the point in the camera coordinate system. The depth normalization process is performed, and the resulting image contains depth information. This type of image is also called a sparse point cloud image. The process of obtaining a sparse point cloud image is the data layer fusion process of point cloud data and image data.

[0087] Among them, the edge device uses a multi-layer perceptron to extract features from the voxelized point cloud data.

[0088] For lidar point cloud data, the edge device divides the point cloud data into regular voxels to form a unified network. Here, each point cloud feature represents a voxel, which contains a set of points, and the corresponding camera pixels are located within a polygon. The voxelized point cloud data is input into the feature extraction model, which uses sparse convolution operations to extract features from the voxelized point cloud data through a shared multi-layer perceptron (MLP). The point cloud data dimension is (B×N×3×1), where B represents the number of groups of point cloud data processed at a time, N is the number of points in each point cloud, 3 represents the three-dimensional coordinates of each point, and 1 represents the number of channels. Through the convolution layer and pooling layer, the extracted features are converted into BEV features. The BEV features are the point cloud features extracted from the sparse point cloud image.

[0089] For the image data of the camera, the image data in the sparse point cloud map is processed by the image encoder to extract the image data features. The image data and point cloud features in the sparse point cloud map are used to perform depth estimation to obtain depth features. The depth features are combined with the image data features to generate the cone features. The cone features are pooled to obtain the image BEV features, which are the image features.

[0090] The edge device of this embodiment adopts a cross-attention mechanism to fuse point cloud features and image features to obtain aggregated features.

[0091] The edge device of this embodiment is also used to preprocess point cloud data and image data to obtain preprocessed data; perform feature fusion on the preprocessed data to obtain feature fusion results; and send the preprocessed data and feature fusion results to the cloud server.

[0092] The process of edge devices preprocessing point cloud data and image data includes:

[0093] Perform image enhancement processing on image data stored in edge devices, adjust the image contrast, brightness, and color balance, and randomly flip, rotate, and crop the image to improve visual quality and feature extraction under different environmental conditions, enhance data robustness, and prepare for subsequent model training. Use the InverseAug method to save transformation parameters, facilitating the subsequent use of the InverseAug method to ensure spatial alignment of lidar data and camera data in the enhanced data. Filter the image to reduce the impact of image noise on subsequent steps.

[0094] The lidar point cloud data stored in the edge device is enhanced by random flipping, random rotation, and random scaling to enhance the robustness of the point cloud data. The InverseAug method is used to save the transformation parameters, and the point cloud data is filtered to remove noise points and outliers.

[0095] The preprocessed data and the transformation parameters saved by the InverseAug method are transmitted to the cloud server for storage via the lightweight encrypted data transmission protocol MQTT, ensuring the security and integrity of the data during transmission. The distributed preprocessing of data on multiple edge devices significantly reduces the computing pressure of the cloud service.

[0096] The specific process of the edge device in this embodiment obtaining aggregated features includes:

[0097] The transformation parameters saved by the InverseAug method are used to restore the point cloud data and image data on the edge device to the state before the geometric transformation. This is because the lidar and camera need to be complementary and need to be accurately aligned. However, due to the enhancement operations applied in the data preprocessing stage, the alignment between the two types of data becomes more complicated. The geometric transformation applied to the data before reversing it in the feature fusion stage can ensure that the laser point cloud data and the camera's image data can be accurately aligned.

[0098] Not all pixels are equally important for the point cloud features representing a specific voxel. In order to better align the point cloud features with the most relevant image features, this embodiment uses a cross-attention mechanism to dynamically capture the correlation between the point cloud features and the image features. The cross-attention mechanism uses three fully connected layers to convert the point cloud features into queries q and the image features into keys k and values ​​v. The attention similarity matrix is ​​obtained by calculating the inner product between the query and the key. The attention similarity matrix contains the correlation between the voxel and the corresponding camera feature. The softmax operation is used to normalize the attention similarity matrix to make it a weight. These weights are used to weight and aggregate the values ​​containing the camera information to obtain the aggregated feature FC. The aggregated feature FC is processed by a fully connected layer and then feature concatenated with the point cloud features to obtain the feature fusion result F, which is the result of the fusion of the point cloud features and the image features. The cross-attention mechanism can be expressed as:

[0099]

[0100] Among them, the superscript l represents the parameters corresponding to the laser radar, the superscript c represents the parameters corresponding to the camera, and d k Represents the channel dimension used for scaling in the LiDAR Bayesian space to stabilize the computation of attention weights.

[0101] The feature fusion result F is transmitted to the cloud server for storage via the lightweight encrypted data transmission protocol MQTT, which facilitates the verification of the training model in subsequent steps, reduces the computing pressure on the cloud server, and ensures the security and integrity of the data during transmission.

[0102] The edge device of this embodiment is used to perform target detection on the feature fusion result according to the target detection model to obtain the target detection result;

[0103] Determine control instructions based on target detection results;

[0104] Control the robot according to control instructions.

[0105] Among them, the target detection results obtained by the edge device include the relative position and distance between the target object and the robot; the determined control instructions include speed, angle and posture.

[0106] Specifically, the edge device inputs the feature fusion results obtained in the previous step into the object detection model deployed on the edge device. Based on the position and size information of the target object output by the target detection model, the relative position and distance between the robot and the target object are calculated.

[0107] Based on the tasks that the robot currently needs to perform, combined with the robot's kinematic model and dynamic model, corresponding robot control commands such as speed, direction, posture and other control instructions are generated; the generated control commands are transmitted to the robot terminal in real time through the ultra-low latency wired communication port, ensuring the integrity and accuracy of data transmission while ensuring that the robot can respond quickly to real-time detection results.

[0108] In this embodiment, the target detection model is trained and updated through the cloud server.

[0109] Among them, the cloud server is used to annotate and fuse features of environmental point cloud data and environmental image data, obtain training data, use the training data to train the current target detection model, and obtain a trained target detection model.

[0110] In the cloud server, data from edge devices is organized. There is a many-to-one relationship between edge devices and cloud servers, that is, multiple edge devices are connected to one cloud server. Therefore, the amount of data received by the cloud server is huge. This data includes point cloud data obtained by the lidar, image data obtained by the camera, and transformation parameters saved by the InverseAug method. In the cloud server, annotation tools are used to annotate the point cloud data and image data to build a training dataset.

[0111] The difference between the feature fusion process of the annotated point cloud data and image data in the cloud server and the feature fusion process of the point cloud data and image data on the edge device is that the amount of data to be processed in the cloud server is larger and the real-time performance is not very demanding. However, the feature fusion process is the same. The annotated feature fusion results obtained in the cloud server are used as training data and input into the current object detection model to train the current object detection model.

[0112] Set evaluation criteria for the target detection model, such as detection accuracy and detection speed. During the model training process, regularly evaluate the model performance, calculate the model's accuracy, recall rate, F1 value and other indicators, and evaluate the model's performance and practicality in different scenarios. When the model performance reaches the set threshold or after a certain number of training rounds, stop training and save it to the cloud server as the trained target detection model. Afterwards, the cloud server evaluates the performance of the trained target detection model and the current target detection model. When the performance of the trained target detection model is better than the current target detection model, use the trained target detection model to update the current target detection model of the cloud server and edge device.

[0113] The cloud server is used to obtain the target detection result of the trained model based on the trained target detection model and the feature fusion result;

[0114] According to the current target detection model and feature fusion results, the target detection results of the current model are obtained;

[0115] Determine the performance of the trained target detection model through the target detection results of the trained model;

[0116] Determine the performance of the current target detection model based on the target detection results of the current model;

[0117] Determine the performance improvement of the trained target detection model over the current target detection model based on the performance of the trained target detection model and the performance of the current target detection model.

[0118] When the performance improvement exceeds the set threshold, it is determined that the performance of the trained target detection model is better than the current target detection model.

[0119] This embodiment uses multiple evaluation metrics, such as precision, recall, F1 score, and mAP, to comprehensively evaluate the performance of the object detection model. When the improvement of the trained object detection model over the current object detection model on multiple evaluation metrics exceeds a preset threshold, the trained object detection model is considered to have significantly better performance than the current object detection model, and the cloud server begins preparing for a model update.

[0120] When updating the model, the cloud server pushes the files and update instructions of the trained target detection model to all edge devices that meet the current scenario through the lightweight encrypted data transmission protocol MQTT. After the edge device receives the update instructions and files, it downloads the files of the trained target detection model and replaces the current target detection model files when the terminal robot is in standby state. The edge device uses the updated target detection model to perform target detection tasks and feeds back the detection results and related data to the cloud server. The cloud server continuously monitors and evaluates the performance of the updated model in actual applications to verify the update effect of the model.

[0121] Taking the environmental perception and obstacle target detection of storage robots in the warehouse in the industrial automation production scenario as an example, the distributed multi-source data fusion perception system based on cloud-edge disclosed in this embodiment is described in detail.

[0122] A lidar and camera are mounted on the top and forward of the warehouse robot to ensure maximum coverage and are jointly calibrated. The lidar uses an M10P time-of-flight lidar with a measurement radius of 30 meters and an angular resolution of 0.22°. The radar scans the surrounding environment 360 degrees at a scanning frequency of 12 Hz and a sampling rate of 20,000 Hz, acquiring point cloud data. The camera uses an RGB camera with a field of view of 63.1° × 49.4°. It captures color images at a frame rate of 30 fps and a resolution of 640 × 480. The edge device uses a Raspberry Pi 4 Model B with a computing power of 0.8 TOPS, 8 GB of memory, two USB 3.0 ports, and two USB 2.0 ports. Collected data is transmitted in real time to the edge device mounted on the robot via a USB 3.0 wired communication interface.

[0123] The edge device performs image enhancement processing on the image data, adjusts the image contrast, brightness and color balance, and performs random flipping, rotation and cropping. The InverseAug method is used to save the transformation parameters. At the same time, the image is filtered to reduce noise. The point cloud data obtained by the lidar is enhanced, including random flipping, rotation and scaling. The InverseAug method is also used to save the transformation parameters, and filtering is performed to remove noise points and outliers. Using the joint calibration data of the lidar and camera, refer to Figure 3 The point cloud data is projected onto the image data to obtain a sparse point cloud map containing depth information.

[0124] The voxelized lidar point cloud data is input into a feature extraction model, and features are extracted through a shared multi-layer perception (MLP) to obtain BEV features as point cloud features. The sparse point cloud image is input into an image encoder to extract image data features and perform depth estimation to obtain depth features. The depth features and image data features are combined to generate cone features, and the image BEV features are obtained after pooling processing. The features are image features.

[0125] The point cloud data and image data are restored to the state before geometric transformation using the transformation parameters saved by the InverseAug method. Reference Figure 4 The correlation between point cloud features and image features is dynamically captured through a cross-attention mechanism. The inner product between the query and the key is calculated to obtain an attention similarity matrix, which is normalized to weights through a softmax operation. The values containing camera information are weighted and aggregated, and the feature fusion result F is obtained by splicing the original point cloud features.

[0126] The feature fusion result F is input into a target detection model deployed on an edge device to obtain a detection result. The relative position and distance between the robot and the target object are calculated. The kinematic and dynamic models of the robot are combined to generate corresponding control commands such as speed, direction, and attitude signals. The control signals are transmitted in real time to the robot terminal through a USB3.0 wired communication port to achieve precise control of the warehouse robot.

[0127] Meanwhile, the preprocessed data and the transformation parameters saved by the InverseAug method are transmitted to the cloud server for storage through the MQTT protocol. The cloud server platform is built based on Netty and Vert.x, supports multiple communication protocols, can meet the real-time and low-latency communication requirements, and has the ability to quickly analyze and process large-scale data. In the cloud server, data from multiple edge devices is sorted and labeled to build a training data set. The labeled feature fusion result is obtained according to the feature fusion method of the edge device, which is used to train the target detection model. The model performance is evaluated regularly, and when the model reaches the set threshold, it is saved as a trained target detection model. When the improvement of the trained target detection model on multiple evaluation indicators compared to the current target detection model exceeds the pre-set threshold, the cloud server pushes the trained target detection model to the edge device through the MQTT protocol. The edge device downloads the trained target detection model file and replaces the original model file when the robot is on standby, realizing model updating and optimization, and improving the target detection performance and environmental perception ability of the warehouse robot in complex industrial environments.

[0128] This embodiment discloses a distributed multi-source data fusion and perception system based on cloud-edge-end, which introduces the cloud-edge-end architecture into data fusion and perception, and adopts a distributed processing method for data fusion, giving full play to the real-time nature of edge devices and the powerful computing power advantages of cloud computing. Edge devices are responsible for data processing tasks with high real-time requirements, while cloud servers undertake data processing tasks with high computing resource requirements, effectively balancing computing resources and data processing efficiency in data fusion and perception problems, avoiding the problem of excessive pressure on a single computing platform in the traditional centralized processing architecture, reducing the risk of system performance bottlenecks, and also reducing deployment costs.

[0129] The disclosed system in this embodiment covers both data-level fusion and feature-level fusion. It uses a cloud-edge-end architecture to reduce the computational pressure faced by multi-level data fusion, and accurately projects the point cloud data collected by the lidar into the camera image data through operations such as coordinate transformation, thereby achieving preliminary integration of two different types of data at the data level; the separately extracted point cloud features and image features are deeply fused using a cross-attention mechanism, fully exploring and combining the advantages and correlations of the two features, so that the fused features are richer, more comprehensive, and more accurately describe the characteristics of the environmental targets; compared with the single-level data fusion method, the multi-level data fusion strategy can more efficiently utilize multi-source data resources.

[0130] The system disclosed in this embodiment also ensures the continuous optimization and performance improvement of the target detection model. The cloud server collects a large amount of data from multiple edge devices for training, solving the problem of insufficient relevant data resources. With the continuous accumulation of data and the iterative optimization of the model, the performance of the target detection model can be continuously improved; and based on the cloud-edge architecture, the updated model can be pushed to the edge device in a timely manner, ensuring that the terminal robot can continuously adapt to more complex environments and task requirements, thereby enhancing the practicality and adaptability of the system.

[0131] Example 2

[0132] In this embodiment, a distributed multi-source data fusion perception method based on cloud-edge is disclosed, including:

[0133] Acquire point cloud data and image data of the robot's environment through a data acquisition device, and send the acquired data to an edge device;

[0134] Perform feature fusion on point cloud data and image data through edge devices to obtain feature fusion results; obtain target detection results based on the target detection model and feature fusion results; send point cloud data, image data and feature fusion results to the cloud server;

[0135] The point cloud data and the image data are labeled and feature fusion is performed by the cloud server to obtain training data, the current target detection model is trained by using the training data to obtain a trained target detection model, the performance of the trained target detection model and the current target detection model is detected by using the feature fusion result, and the cloud server and the current target detection model of the edge device are updated by using the trained target detection model when the performance of the trained target detection model is better than that of the current target detection model.

[0136] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments can be realized by electronic hardware or a combination of electronic hardware and computer software. Whether the functions are realized in hardware or software mode depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the application.

[0137] Although the specific embodiments of the application are described above in combination with the drawings, the description is not a limitation on the scope of protection of the application, and those skilled in the art should understand that various modifications or changes made by those skilled in the art on the basis of the technical solutions of the application without creative labor are still within the protection scope of the application.

Claims

1. A distributed multi-source data fusion perception system based on cloud-edge-end, characterized by: include: A data acquisition device, used to acquire point cloud data and image data of the robot's environment and send the acquired data to the edge device; Edge devices are used to perform feature fusion on point cloud data and image data to obtain feature fusion results; Obtain target detection results based on the target detection model and feature fusion results; Send point cloud data, image data and feature fusion results to the cloud server; The cloud server is used to annotate and fuse features of point cloud data and image data to obtain training data, and then use the training data to train the current target detection model to obtain a trained target detection model. The performance of the trained target detection model and the current target detection model is tested through the feature fusion results; when the performance of the trained target detection model is better than the current target detection model, the current target detection model of the cloud server and edge device is updated through the trained target detection model.

2. The distributed multi-source data fusion perception system based on cloud-edge-end according to claim 1 is characterized in that: Edge device, used to project point cloud data onto image data to obtain a sparse point cloud map; Perform voxel processing on the point cloud data to obtain voxelized point cloud data; Perform feature extraction on the voxelized point cloud data and convert the extracted features into BEV features to obtain point cloud features; Extract image data features from sparse point cloud images; Use point cloud features and image data in sparse point cloud images to perform depth estimation and obtain depth features; Combine the depth features with the image data features to obtain the cone features; Pooling is performed on the cone features to obtain image features; Fuse point cloud features and image features to obtain aggregated features; The aggregated features are concatenated with the point cloud features to obtain the feature fusion results.

3. The distributed multi-source data fusion perception system based on cloud-edge-end according to claim 2 is characterized in that: The edge device uses a multi-layer perceptron to extract features from the voxelized point cloud data.

4. The distributed multi-source data fusion perception system based on cloud-edge-end according to claim 2 is characterized in that: The edge device uses a cross-attention mechanism to fuse point cloud features and image features to obtain aggregated features.

5. The distributed multi-source data fusion perception system based on cloud-edge-end according to claim 1 is characterized in that: The edge device is also used to pre-process point cloud data and image data to obtain pre-processed data; Perform feature fusion on the preprocessed data to obtain feature fusion results; The preprocessed data and feature fusion results are sent to the cloud server.

6. The distributed multi-source data fusion perception system based on cloud-edge-end according to claim 1 is characterized in that: The edge device is used to perform target detection on the feature fusion result according to the target detection model to obtain the target detection result; Determine control instructions based on target detection results; Control the robot according to control instructions.

7. The distributed multi-source data fusion perception system based on cloud-edge-end according to claim 6 is characterized in that: The target detection results obtained by the edge device include the relative position and distance between the target object and the robot; the determined control instructions include speed, angle and posture.

8. The distributed multi-source data fusion perception system based on cloud-edge-end according to claim 1 is characterized in that: An encrypted data transmission protocol is used for data exchange between edge devices and cloud servers.

9. The distributed multi-source data fusion perception system based on cloud-edge-end according to claim 1 is characterized in that: The cloud server is used to obtain the target detection results of the trained model based on the trained target detection model and the feature fusion results; According to the current target detection model and feature fusion results, the target detection results of the current model are obtained; Determine the performance of the trained target detection model through the target detection results of the trained model; Determine the performance of the current target detection model based on the target detection results of the current model; Determine the performance improvement of the trained target detection model over the current target detection model based on the performance of the trained target detection model and the performance of the current target detection model. When the performance improvement exceeds the set threshold, it is determined that the performance of the trained target detection model is better than the current target detection model.

10. A distributed multi-source data fusion perception method based on cloud-edge, characterized in that: include: Acquire point cloud data and image data of the robot's environment through a data acquisition device, and send the acquired data to an edge device; Perform feature fusion on point cloud data and image data through edge devices to obtain feature fusion results; Obtain target detection results based on the target detection model and feature fusion results; Send point cloud data, image data and feature fusion results to the cloud server; The cloud server is used to annotate and fuse features of point cloud data and image data to obtain training data, and the training data is used to train the current target detection model to obtain a trained target detection model. The performance of the trained target detection model and the current target detection model is tested through the feature fusion results; when the performance of the trained target detection model is better than the current target detection model, the current target detection model of the cloud server and edge device is updated through the trained target detection model.

Images