Dynamic data fusion method for explosion-proof robot in heterogeneous environment

Abstract

The present application relates to the technical field of robot data processing, in particular to a dynamic data fusion method for explosion-proof robots in a heterogeneous environment, comprising the following steps: according to the type of heterogeneous data, extracting the key features of the heterogeneous data, and constructing a multi-dimensional feature vector; determining the importance of the sensors in combination with the task scene, assigning a basic weight, and adjusting the real-time weight according to the real-time requirement to dynamically output the fusion weight of each sensor; using a deep learning network to learn the complex correlation between heterogeneous data, inputting the dynamic weight as a guide parameter into the network to realize feature weighted fusion and output fusion data; based on the fusion data, completing environment and risk analysis, and generating path planning, obstacle avoidance, and hazard source identification decision suggestions; through the above-mentioned mode, the fusion weight can be adaptively adjusted to the environment change, the reliability of the robot decision is improved, and the environmental perception accuracy and operation safety of the explosion-proof robot are improved.

Description

Technical Field

[0001] This invention relates to the field of robot data processing technology, and in particular to a dynamic data fusion method for explosion-proof robots in heterogeneous environments. Background Technology

[0002] Explosion-proof robots are widely used in flammable and explosive hazardous work environments such as chemical plants, mines, and oil and gas plants. They need to collect environmental information through various devices such as visual sensors, lidar, infrared sensors, gas sensors, and sound sensors to achieve environmental perception, risk identification, path planning, and safe operation. The data collected by the above sensors vary significantly in format, accuracy, temporal sequence, and spatial dimensions, constituting a typical heterogeneous data environment.

[0003] Existing data fusion methods for explosion-proof robots generally adopt fixed fusion strategies and preset weight allocations, without considering dynamic changes in the environment and fluctuations in sensor status. This has the following drawbacks: fixed weights cannot adapt to complex heterogeneous environments; when there is environmental interference, sensor abnormalities, or failures, the accuracy of the fusion results decreases significantly, affecting the reliability of robot decision-making. Summary of the Invention

[0004] The purpose of this invention is to provide a dynamic data fusion method for explosion-proof robots in heterogeneous environments, aiming to solve the technical problem that the fixed weights in the existing technology cannot be adapted to complex heterogeneous environments, and the accuracy of the fusion results will decrease significantly when there is environmental interference, sensor abnormalities or failures, thus affecting the reliability of robot decision-making.

[0005] To achieve the above objectives, the present invention employs a dynamic data fusion method for explosion-proof robots in heterogeneous environments, comprising the following steps: The heterogeneous raw data collected by multiple sensors on the explosion-proof robot are subjected to time synchronization, spatial registration and data cleaning. Among them, time synchronization uses timestamp alignment to achieve time sequence unification of multiple sensors; spatial registration unifies the data of each sensor to the global coordinate system; data cleaning removes noise, outliers and redundant data. Based on the type of heterogeneous data, key features of the heterogeneous data are extracted and multi-dimensional feature vectors are constructed. The types of heterogeneous data include visual data, LiDAR data, gas sensor data, and sound sensor data. Visual data uses a deep learning model to extract image features, LiDAR data extracts point cloud features and spatial distribution features, gas sensor data extracts concentration change rate and feature patterns, and sound sensor data extracts spectral features and voiceprint features. The importance of sensors is determined by the task scenario, basic weights are assigned, and real-time weights are adjusted according to real-time requirements. The fusion weights of each sensor are dynamically output by integrating reliability, importance, and real-time performance through an adaptive algorithm. A preliminary fusion framework is constructed based on evidence theory. Data from each sensor is used as an independent source of evidence to complete uncertainty fusion. A deep learning network is used to learn the complex relationships between heterogeneous data. Dynamic weights are used as guiding parameters to input into the network to achieve feature-weighted fusion and output fused data. Based on fused data, environmental and risk analysis is performed to generate decision suggestions for path planning, obstacle avoidance, and hazard identification.

[0006] The process involves performing time synchronization, spatial registration, and data cleaning on heterogeneous raw data collected from multiple sensors mounted on the explosion-proof robot. Time synchronization utilizes timestamp alignment to achieve time sequence unification across multiple sensors. Spatial registration unifies the data from each sensor to a global coordinate system. Data cleaning removes noise, outliers, and redundant data. Acquire heterogeneous raw data collected by various sensors mounted on the explosion-proof robot; Use timestamp alignment to unify the time dimension of heterogeneous raw data; Transform heterogeneous raw data into a global coordinate system to eliminate spatial differences between data. The heterogeneous raw data is filtered and screened to remove noise, outliers, and redundant information, and the heterogeneous data is output.

[0007] The process involves extracting key features from heterogeneous data based on its type and constructing multidimensional feature vectors. The heterogeneous data types include visual data, LiDAR data, gas sensor data, and sound sensor data. Visual data is extracted using a deep learning model to extract image features; LiDAR data is extracted for point cloud features and spatial distribution features; gas sensor data is extracted for concentration change rate and feature patterns; and sound sensor data is extracted for spectral features and voiceprint features. Determine the type of heterogeneous data; the types of heterogeneous data include visual data, LiDAR data, gas sensor data, and sound sensor data; For visual data, a deep learning model is used to perform convolution, pooling, and feature mapping operations to extract high-dimensional semantic features of edges, textures, contours, and target regions in the image layer by layer, so as to obtain a standardized image feature vector and output the visual data features. For LiDAR data, the point cloud data is downsampled and filtered for optimization. The normal vector, curvature, and neighborhood density are calculated point by point to extract the structural features of the point cloud. The distribution density, distance distribution, and orientation distribution of the point cloud in three-dimensional space are statistically analyzed to obtain spatial distribution-related features and output the LiDAR data features. For gas sensor data, differential calculation is performed on the concentration time series data to obtain the rate of change of gas concentration per unit time. Combined with the concentration fluctuation amplitude, stable duration, and abrupt change threshold, feature patterns are extracted to obtain feature data that can characterize the gas state and output the gas sensor data features. For sound sensor data, the audio data is converted from the time domain to the frequency domain, and the spectral features of amplitude, energy and frequency components are extracted. Based on the unique properties of the sound signal, the voiceprint features are extracted to obtain feature information that can distinguish the type and state of the sound source, and the sound sensor data features are output.

[0008] The process involves several steps, including: performing time-domain and frequency-domain transformations on the audio data from the sound sensor; extracting spectral features of amplitude, energy, and frequency components; extracting voiceprint features based on the unique properties of the sound signal; obtaining feature information that can distinguish the type and state of the sound source; and outputting the sound sensor data features. The visual data features, lidar data features, gas sensor data features, and sound sensor data features are normalized and then spliced ​​together according to dimensions to form a multi-dimensional feature vector.

[0009] The steps include determining sensor importance based on the task scenario, assigning basic weights, adjusting real-time weights according to real-time requirements, and dynamically outputting the fused weights of each sensor through an adaptive algorithm that integrates reliability, importance, and real-time performance: Based on the current task scenario, determine the importance of each sensor's data and assign basic weights; Adjust the real-time weights of the corresponding sensor data according to the task's requirements for data real-time performance; By integrating reliability, importance, and real-time factors through an adaptive algorithm, the final dynamic fusion weights of each sensor are output.

[0010] Before the step of determining the importance of each sensor's data and assigning basic weights based on the current task scenario: The reliability of a sensor is determined by its historical data accuracy, current operating status, and environmental adaptability.

[0011] The steps include: constructing a preliminary fusion framework based on evidence theory; treating each sensor's data as an independent source of evidence to complete uncertainty fusion; employing a deep learning network to learn the complex relationships between heterogeneous data; inputting dynamic weights as guiding parameters into the network to achieve feature-weighted fusion; and outputting fused data. A preliminary fusion framework was built based on evidence theory, and the data from each sensor was used as an independent source of evidence to complete the fusion of uncertainty information. Construct a deep learning network to learn and explore the complex relationships between different types of heterogeneous data, and input dynamic weights as guiding parameters into the deep learning network to perform weighted fusion of features; By iteratively optimizing the deep learning network through a preset loss function, adjusting the network parameters, and outputting globally fused data that can comprehensively represent the current environmental state and is compatible with multi-source heterogeneous information.

[0012] In the step of building a preliminary fusion framework based on evidence theory and using data from each sensor as an independent source of evidence to complete the fusion of uncertain information: A basic probability allocation function is established for the feature data corresponding to each sensor. The feature information output by different sensors is regarded as an independent source of evidence. The multi-source evidence is fused and calculated through evidence combination rules to complete the normalization and credibility synthesis of uncertain and conflicting information, and obtain the preliminary fusion result.

[0013] In the steps of completing environmental and risk analysis based on fused data and generating decision suggestions for path planning, obstacle avoidance, and hazard identification: Based on the fused data, the current environmental status and potential safety risks are analyzed and assessed, and an appropriate robot path planning scheme is generated based on the results of the environmental and risk analysis. Based on the distribution of environmental obstacles, generate obstacle avoidance strategies.

[0014] After the step of generating an obstacle avoidance strategy based on the distribution of environmental obstacles: Detect the characteristics of hazardous scenes, identify the sources of danger, and generate corresponding decision-making recommendations.

[0015] This invention discloses a dynamic data fusion method for explosion-proof robots in heterogeneous environments. The method performs time synchronization, spatial registration, and data cleaning on heterogeneous raw data collected by multiple sensors mounted on the explosion-proof robot. Time synchronization uses timestamp alignment to achieve temporal consistency across multiple sensors. Spatial registration unifies the data from each sensor to a global coordinate system. Data cleaning removes noise, outliers, and redundant data. Based on the type of heterogeneous data, key features are extracted to construct a multi-dimensional feature vector. The heterogeneous data types include visual data, LiDAR data, gas sensor data, and sound sensor data. Visual data uses a deep learning model to extract image features; LiDAR data extracts point cloud features and spatial distribution features; gas sensor data extracts concentration change rate and feature patterns; and sound sensor data extracts... The system utilizes spectral and acoustic signature features; it determines sensor importance based on the task scenario, assigns basic weights, and adjusts real-time weights according to real-time requirements. An adaptive algorithm integrates reliability, importance, and real-time performance to dynamically output the fusion weights for each sensor. A preliminary fusion framework is constructed using evidence theory, treating each sensor's data as an independent source of evidence to achieve uncertain fusion. A deep learning network is employed to learn the complex relationships between heterogeneous data, and the dynamic weights are input as guiding parameters to the network to achieve feature-weighted fusion, outputting fused data. Based on the fused data, environmental and risk analysis is performed, generating decision suggestions for path planning, obstacle avoidance, and hazard identification. Through these methods, the system can adapt to environmental changes and dynamically adjust fusion weights, improving the reliability of robot decision-making and thus enhancing the environmental perception accuracy and operational safety of the explosion-proof robot. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of the steps of the dynamic data fusion method for explosion-proof robots in heterogeneous environments according to the present invention.

[0018] Figure 2 This is a flowchart of steps S100 of the present invention.

[0019] Figure 3 This is a flowchart of steps S200 of the present invention.

[0020] Figure 4 This is a flowchart of steps S300 of the present invention.

[0021] Figure 5This is a flowchart of steps S400 of the present invention.

[0022] Figure 6 This is a flowchart of steps S500 of the present invention.

[0023] Figure 7 This is a structural schematic diagram of the dynamic data fusion system for explosion-proof robots in heterogeneous environments according to the present invention.

[0024] Figure 8 This is a schematic diagram of the electronic device of the present invention.

[0025] 601 - Data Preprocessing Module, 602 - Feature Extraction Module, 603 - Dynamic Weight Allocation Module, 604 - Data Fusion Module, 605 - Decision Suggestion Generation Module. Detailed Implementation

[0026] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0027] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0028] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0029] Please see Figures 1-6 This invention provides a dynamic data fusion method for explosion-proof robots in heterogeneous environments, comprising the following steps: S100: Performs time synchronization, spatial registration, and data cleaning on heterogeneous raw data collected by multiple sensors on the explosion-proof robot; among them, time synchronization uses timestamp alignment to achieve time sequence unification of multiple sensors; spatial registration unifies the data of each sensor to the global coordinate system; data cleaning removes noise, outliers, and redundant data.

[0030] The specific process is as follows: S101: Acquire heterogeneous raw data collected by various sensors mounted on the explosion-proof robot; S102: Use timestamp alignment to unify the time dimension of heterogeneous raw data; S103: Transform heterogeneous raw data to the global coordinate system to eliminate spatial differences between data. S104: Filter and screen the heterogeneous raw data to remove noise, outliers, and redundant information, and output the heterogeneous data.

[0031] During the above process, all heterogeneous raw data collected in real time by the vision sensors, lidar, gas sensors, and sound sensors on the explosion-proof robot within the same work cycle are acquired. The collected information with different interfaces, different transmission rates, and different data formats is uniformly connected to the robot's central processing unit to complete the temporary storage, sorting, and preliminary classification of the data, ensuring that all data to be processed is complete and without omissions.

[0032] Using the system clock of the robot's main control unit as a unified time reference, each set of data collected by each sensor is stamped with high precision. Based on the timestamps, all sensor data are subjected to linear interpolation, resampling, and alignment operations to eliminate timing deviations caused by different acquisition frequencies, transmission delays, and start-up times, ensuring that all heterogeneous data maintain strict consistency in the time dimension.

[0033] A global three-dimensional rectangular coordinate system is established with the geometric center of the explosion-proof robot body as the origin. The internal and external parameters and installation positions of each sensor are calibrated in advance. Based on the calibration results, the image coordinates of the vision sensor, the local coordinates of the lidar point cloud, and the detection coordinates of the sound sensor are uniformly transformed to the global coordinate system to eliminate spatial position deviations caused by different sensor installation angles, installation heights, and detection directions, and to achieve complete spatial position matching of multi-source data. The data that has completed time synchronization and spatial registration undergoes sequential noise filtering, outlier removal, and redundant data deletion. Gaussian filtering, median filtering, and moving average filtering are used to remove random noise caused by environmental electromagnetic interference and mechanical vibration. Outlier values ​​exceeding the normal operating range are identified and removed through threshold judgment, gradient detection, and outlier analysis. Specifically, for visual data: a pixel gradient threshold of ±30 and a brightness outlier threshold of ±2 times the mean are used to remove overexposed, completely black, and salt-and-pepper noise outlier frames; for LiDAR point clouds: an Euclidean distance outlier threshold of 0.1m and a normal vector gradient threshold of ±15° are used to remove spatially isolated and distorted points; for gas sensors: a concentration threshold of ±10%FS and a rate of change gradient threshold of ±5% / s are used to remove drift and abrupt outliers; for sound sensors: an amplitude threshold of ±3dB and a spectral gradient threshold of ±10% are used to remove noise and distorted data. Redundant segments with repeated acquisitions, no valid information, and no change are filtered out, ultimately outputting clean, standardized, and usable heterogeneous data.

[0034] S200: Based on the type of heterogeneous data, extract key features from the heterogeneous data and construct a multi-dimensional feature vector. The heterogeneous data types include visual data, LiDAR data, gas sensor data, and sound sensor data. Visual data uses a deep learning model to extract image features; LiDAR data extracts point cloud features and spatial distribution features; gas sensor data extracts concentration change rate and feature patterns; and sound sensor data extracts spectral features and voiceprint features. The specific process is as follows: S201: Determine the type of heterogeneous data; the types of heterogeneous data include visual data, LiDAR data, gas sensor data, and sound sensor data; S202: For visual data, a deep learning model is used to perform convolution, pooling, and feature mapping operations to extract high-dimensional semantic features of edges, textures, contours, and target regions in the image layer by layer, so as to obtain a standardized image feature vector and output the visual data features. S203: For LiDAR data, downsampling and filtering optimization are performed on the point cloud data. The normal vector, curvature, and neighborhood density are calculated point by point to extract the point cloud structural features. The distribution density, distance distribution, and orientation distribution of the point cloud in three-dimensional space are statistically analyzed to obtain spatial distribution-related features and output LiDAR data features. S204: For gas sensor data, differential calculation is performed on the concentration time series data to obtain the gas concentration change rate per unit time. Combined with the concentration fluctuation amplitude, stable duration, and abrupt change threshold, feature patterns are extracted to obtain feature data that can characterize the gas state and output the gas sensor data features. S205: For sound sensor data, perform time-domain and frequency-domain conversion on the audio data, extract the spectral features of amplitude, energy, and frequency components, and extract voiceprint features based on the unique properties of the sound signal to obtain feature information that can distinguish the type and state of the sound source, and output the sound sensor data features. S206: Normalize the visual data features, lidar data features, gas sensor data features, and sound sensor data features, and then concatenate and integrate them into a multi-dimensional feature vector according to their dimensions.

[0035] In the above process, the standardized heterogeneous data output in step S100 is subjected to modal recognition and type determination. Based on the data format, sampling dimension, and physical meaning, visual image data, LiDAR point cloud data, gas concentration time series data, and sound audio data are automatically distinguished. Different types of data are diverted to the corresponding feature extraction channels to ensure that the feature extraction process is more targeted and accurate. In the diversion of different types of data, the format recognition is as follows: distinguish by file extension / data header - .jpg / .png for visual, .pcd / .bin for LiDAR, .csv for gas, and .wav for sound; sampling dimension: 2D matrix → visual, 3D point set → LiDAR, 1D time series → gas / sound; physical meaning: concentration unit (ppm / LEL) → gas, amplitude / frequency → sound, grayscale / pixel → visual, distance coordinates → LiDAR. Automatic diversion is achieved through a three-layer logic of format verification + dimension determination + unit matching.

[0036] For visual data, image data is first preprocessed by size normalization, grayscale standardization, and contrast enhancement. The processed image is then input into a constructed convolutional neural network model. Multiple convolutional layers are used to extract low-level features such as edges, corners, textures, contours, and target regions. Pooling layers reduce feature dimensionality while preserving key information. Fully connected layers and activation functions are used to map high-dimensional semantic features. Finally, the feature vectors are standardized to output fixed-dimensional, stable, and reliable visual data features. The convolutional neural network model has the following structure: 5 convolutional layers + 3 pooling layers + 2 fully connected layers, with 3×3 kernels, a stride of 1, padding=1, and ReLU activation function. Input / output: Input is a 480×640 normalized image; output is a 1×256 dimensional visual feature vector. Standardization: Features are mapped to the [0,1] interval using max-min normalization to eliminate dimensional differences. Training steps: Training is performed on an explosion-proof scene image dataset with MSE loss function, batch size of 16, 100 iterations, and a learning rate of 1e-4. The core task of this convolutional neural network is to jointly extract multiple classifications of explosion-proof hazardous targets and semantic segmentation of environmental areas. It is trained based on an explosion-proof factory area image dataset to automatically identify four key targets from raw pixel images: leak points, flammable and explosive tanks, obstacles, and open flame hazards.

[0037] For LiDAR data, voxel mesh downsampling and statistical filtering optimization are first performed on the point cloud data to reduce the number of points and remove outliers. Then, the point cloud normal vector, curvature, neighborhood point density, and roughness information are calculated point by point to form point cloud structural features that can characterize the surface structure of the object. At the same time, the distribution density, distance distribution, orientation distribution, and height distribution features of the point cloud in three-dimensional space are statistically analyzed to form spatial distribution features that can characterize the spatial structure of the environment. The structural features and spatial distribution features are combined to output complete LiDAR data features.

[0038] For gas sensor data, regarding numerical feature processing: the rate of change of concentration per unit time (floating-point value) is normalized to maximum-minimum and uniformly mapped to the [0,1] interval, preserving the original numerical change pattern; regarding the digital encoding of text feature patterns: one-hot encoding is used for four types of text features: stable normal [1,0,0,0], slow rise [0,1,0,0], rapid leakage [0,0,1,0], and violent fluctuation [0,0,0,1], transforming the qualitative description of the text into a four-dimensional quantized array; feature integration and splicing rules: the normalized rate of change feature (1-dimensional floating-point) + one-hot encoded feature (4-dimensional) are combined to form a 5-dimensional basic gas feature, which is then zero-padding to a unified format of 256 dimensions, maintaining dimensional standardization with visual, laser, and sound features. In gas sensor data, the continuously acquired concentration time-series data is first smoothed to eliminate instantaneous fluctuation interference. Then, the time-series data is differentially calculated between consecutive time points to obtain the rate of change and the upward / downward trend of gas concentration per unit time. Combining information such as the duration of concentration stabilization, fluctuation amplitude, number of abrupt changes, and exceeding the threshold, typical characteristic patterns such as stable state, slow rise, rapid leakage, and violent fluctuation are extracted. Among them, stable state: concentration fluctuation ≤ ±2%FS, stable duration ≥10s, no abrupt changes; slow rise: change rate 0.1%~5% / s, duration ≥3s; rapid leakage: change rate >5% / s, concentration exceeding the threshold; violent fluctuation: abrupt changes ≥3 times within 1s, fluctuation > ±10%FS. Calculation method: a sliding window difference method is used, with a window size of 1s, to calculate the instantaneous change rate and cumulative fluctuation. Characteristic data that can comprehensively reflect the gas state is formed, and the gas sensor data characteristics are output.

[0039] For sound sensor data, the audio data is first pre-emphasized, framed, and windowed to enhance the high-frequency components of the signal and reduce spectral leakage. Then, the time-domain audio signal is converted into a frequency-domain signal, and spectral features such as signal amplitude, energy, spectral peak value, frequency distribution, and harmonic structure are extracted. At the same time, voiceprint features are extracted based on the unique time-domain envelope and frequency-domain distribution of the sound signal to distinguish the sound source type, device status, and abnormal sounds, and output sound sensor data features that can be used for identification and judgment.

[0040] Visual data features, LiDAR data features, gas sensor data features, and sound sensor data features are each subjected to max-min normalization to eliminate differences in dimensions, numerical ranges, and amplitudes among different features. The four types of features are then spliced ​​and integrated in a fixed dimensional order. First, the four types of features are uniformly mapped to a 1×256 dimension (visual 256, LiDAR 256, gas 128 → zero-padding 256, sound 128 → zero-padding 256). They are then spliced ​​in a fixed order of visual → LiDAR → gas → sound to obtain a 1×1024-dimensional global feature vector, forming a unified, complete, and suitable multi-dimensional feature vector for subsequent fusion.

[0041] S300: Based on the task scenario, the importance of sensors is determined, basic weights are assigned, and real-time weights are adjusted according to real-time requirements. An adaptive algorithm is used to comprehensively consider reliability, importance, and real-time performance, dynamically outputting the fusion weights for each sensor. The specific process is as follows: S301: Determine the reliability of the sensor based on the accuracy of its historical data, current operating status, and environmental adaptability; S302: Determine the importance of each sensor's data and assign basic weights based on the current task scenario; S303: Adjust the real-time weight of the corresponding sensor data according to the task's requirements for data real-time performance; S304: By integrating reliability, importance and real-time factors through an adaptive algorithm, the final dynamic fusion weights of each sensor are output.

[0042] In the above process, a sensor reliability assessment model is established, comprehensively statistically analyzing the sensor's historical data accuracy, effective acquisition rate, and long-term operational stability. Specifically: The model input dimensions are determined by selecting five indicators as inputs: historical acquisition accuracy, data effectiveness rate, continuous fault-free duration, current operating status, and environmental interference level. Indicator quantification rules are constructed: historical acquisition accuracy is calculated by statistically determining the degree of agreement between historical sensor data and standard reference values, quantified as a percentage within a 0-1 range; data effectiveness rate is calculated by dividing the number of effective data frames per unit time by the total number of acquired frames, quantified as a 0-1 range. Working status is quantified as 1, 0.7, 0.4, and 0 for normal, drift, packet loss, and interruption, respectively; environmental adaptability is graded and scored according to temperature, humidity, dust, and electromagnetic interference intensity, and quantified as a value in the range of 0-1; a comprehensive evaluation logic is established: the above indicators are weighted and fused according to preset priorities, and the sensor comprehensive reliability is output through step-by-step judgment and normalization processing, forming a reliability evaluation model that can be calculated in real time; model solidification and online update: the quantification rules and judgment logic are solidified into the robot processing unit, which supports automatic updating of input indicators based on real-time data and dynamic output of the current reliability.

[0043] The sensor's current operating status is monitored in real time to determine whether there are faults such as data loss, signal drift, transmission interruption, or abnormal acquisition. The sensor's environmental adaptability is assessed by combining the current ambient temperature, humidity, dust, vibration, and electromagnetic interference levels. The above information is then comprehensively calculated to obtain a reliability value that objectively reflects the sensor's current usability.

[0044] Based on the current operational scenarios performed by the explosion-proof robot, including routine inspections, leak detection, emergency response, obstacle avoidance, and hazard identification, the contribution and criticality of each type of sensor to task completion are assessed. Sensors crucial to the core of the task are assigned higher base weights, auxiliary sensors are assigned moderate base weights, and sensors not essential for the current task are assigned lower base weights, thus completing the initial allocation of base weights. The base weight allocation (scenario-based) is as follows: Routine inspection: Vision 0.3, Laser 0.3, Gas 0.2, Sound 0.2; Leak detection: gas 0.5, laser 0.2, visual 0.2, sound 0.1; Emergency response: gas 0.4, sound 0.3, laser 0.2, vision 0.1.

[0045] Based on the task's requirement for data real-time performance, sensor data is classified into real-time levels. Sensor data requiring rapid response and real-time judgment receives a higher real-time weight, while sensor data that allows for short delays and emphasizes stability maintains a standard real-time weight. The basic weights are adjusted through the real-time dimension, with high real-time (leakage / emergency): ×1.2; medium real-time (inspection): ×1.0; and low real-time (archiving): ×0.8, making the weight allocation more closely match actual operational needs.

[0046] An adaptive weighted fusion algorithm is employed to comprehensively calculate and normalize three indicators: sensor reliability, task base weight, and real-time adjustment weight, ensuring that the sum of all sensor weights remains within a stable and reasonable range. The algorithm automatically adjusts in real-time based on environmental changes, sensor status changes, and task switching. The dynamic fusion weight formula is as follows: W 最终 =W 基础 ×W 实时 ×W 可靠 Normalization makes ΣW=1; Reliability W 可靠 The final dynamic fusion weight is calculated by weighting historical accuracy, current status, and environmental adaptability, with values ​​ranging from [0,1]. The final output is the dynamic fusion weight corresponding to each sensor.

[0047] S400: A preliminary fusion framework is constructed based on evidence theory. Data from each sensor is treated as an independent source of evidence to achieve uncertainty fusion. A deep learning network is used to learn the complex relationships between heterogeneous data. Dynamic weights are input as guiding parameters into the network to achieve feature-weighted fusion and output fused data. The specific process is as follows: S401: Based on evidence theory, a preliminary fusion framework is built. Basic probability allocation functions are established for the feature data corresponding to each sensor. The feature information output by different sensors is regarded as an independent source of evidence. Multi-source evidence is fused and calculated through evidence combination rules. The normalization and credibility synthesis of uncertain and conflicting information are completed to obtain the preliminary fusion result. S402: Construct a deep learning network to learn and explore the complex relationships between different types of heterogeneous data, and input dynamic weights as guiding parameters into the deep learning network to perform weighted fusion of features; S403: Iteratively optimize the deep learning network by using a preset loss function, adjust the network parameters, and output fused data that can comprehensively represent the current environmental state and is compatible with multi-source heterogeneous information.

[0048] In the above process, a preliminary fusion framework for multi-source information is built based on DS evidence theory. Basic probability allocation functions are established for the feature vectors corresponding to each type of sensor, and the feature information output by different sensors is regarded as an independent source of evidence. According to the evidence combination rules, conflicting, ambiguous and uncertain information from multiple sources is gradually synthesized, conflict resolved and normalized to improve the weight of credible information and reduce the influence of contradictory information, so as to obtain a preliminary fusion feature result with high credibility.

[0049] A deep fusion network model is constructed, comprising an input layer, a feature learning layer, a weighted fusion layer, and an output layer. The preliminary fusion features obtained in step S401 and the multi-dimensional feature vectors obtained in step S200 are input into the network. Through multi-layer nonlinear mapping, feature interaction, and correlation mining, the intrinsic connections, complementary relationships, and constraint rules among four types of heterogeneous data—visual, LiDAR, gas, and sound—are automatically learned. At the same time, the dynamic weights output in step S300 are introduced into the weighted fusion layer to weight and strengthen different feature channels, highlighting the contributions of high-reliability, high-importance, and high-real-time features, and suppressing the interference of low-reliability, abnormal, and low-contribution features, thereby completing high-precision feature weighted fusion.

[0050] Set a loss function suitable for environmental perception, use real environment labeled data as supervision signal, and perform multiple rounds of iterative training and parameter optimization on the deep fusion network; continuously adjust the network weights, biases, and activation thresholds to gradually reduce the error between the fusion result and the real value; when the network loss converges, the output is stable and the accuracy meets the standard, stop the optimization and output global fusion data that can comprehensively, accurately, and stably represent the current environmental state and is compatible with multi-source heterogeneous information.

[0051] S500: Based on fused data, it performs environmental and risk analysis, generating decision suggestions for path planning, obstacle avoidance, and hazard identification. The specific process is as follows: S501: Analyze and assess the current environmental status and potential safety risks based on the fused data, and generate an appropriate robot path planning scheme based on the environmental and risk analysis results. S502: Generate obstacle avoidance strategies based on the distribution of environmental obstacles; S503: Detect the characteristics of hazardous scenes, identify the sources of danger, and generate corresponding disposal decision recommendations.

[0052] In the above process, the global fusion data output from step S400 is input into the environmental analysis and risk assessment model to comprehensively analyze environmental information such as terrain undulation, spatial openness, obstacle distribution, and equipment status. At the same time, it identifies potential safety hazards such as flammable and explosive gases, toxic substances, high-temperature areas, abnormal pressure, and structural damage, and determines the risk level and dangerous range. Based on the target location, risk level, access conditions, and operation requirements, a path search algorithm is used to generate a safe, efficient, and feasible robot driving path and operation route planning scheme.

[0053] The robot extracts obstacle location, size, shape, movement trend, and distance information from the fused data in real time, and continuously calculates the safe distance and collision probability between the robot and the obstacle. When a collision risk is determined, the robot automatically generates graded avoidance strategies such as deceleration, detour, emergency stop, and waiting in place based on obstacle type, movement speed, and on-site space to ensure the robot can pass safely in complex environments.

[0054] Hazardous features are extracted from global fusion data, including gas concentration exceeding limits, abnormal sounds, high temperature points, leak points, signs of open flame, and structural deformation. The extracted features are matched with a preset hazard source database to complete hazard source type identification, hazard level determination, and hazard area delineation. Based on the identification results, decision-making suggestions such as alarm prompts, shutdown protection, on-site evacuation, enhanced monitoring, and emergency response are automatically generated, and the results are output to the operator's terminal in visual or voice form to assist in safety decision-making.

[0055] Corresponding to the aforementioned embodiments of the dynamic data fusion method for explosion-proof robots in heterogeneous environments, this application also provides embodiments of the dynamic data fusion system for explosion-proof robots in heterogeneous environments.

[0056] Figure 7 This is a structural schematic diagram illustrating a dynamic data fusion system for explosion-proof robots in heterogeneous environments, according to an exemplary embodiment. (Refer to...) Figure 7 The system may include: a data preprocessing module 601, a feature extraction module 602, a dynamic weight allocation module 603, a data fusion module 604, and a decision suggestion generation module 605; wherein: The data preprocessing module 601 is used to perform time synchronization, spatial registration, and data cleaning on heterogeneous raw data collected by multiple sensors on the explosion-proof robot. Among them, time synchronization uses timestamp alignment to achieve time sequence unification of multiple sensors; spatial registration unifies the data of each sensor to a global coordinate system; and data cleaning removes noise, outliers, and redundant data. The feature extraction module 602 is used to extract key features of heterogeneous data according to the type of heterogeneous data and construct a multi-dimensional feature vector; wherein the types of heterogeneous data include visual data, lidar data, gas sensor data, and sound sensor data. Visual data uses a deep learning model to extract image features, lidar data extracts point cloud features and spatial distribution features, gas sensor data extracts concentration change rate and feature pattern, and sound sensor data extracts spectral features and voiceprint features. The dynamic weight allocation module 603 is used to determine the importance of sensors in combination with the task scenario, allocate basic weights, adjust real-time weights according to real-time requirements, and dynamically output the fusion weights of each sensor by comprehensively considering reliability, importance, and real-time performance through an adaptive algorithm. The data fusion module 604 is used to construct a preliminary fusion framework based on evidence theory, use each sensor data as an independent source of evidence to complete uncertainty fusion, use a deep learning network to learn the complex correlation between heterogeneous data, input dynamic weights as guiding parameters into the network, realize feature weighted fusion, and output fused data. The decision suggestion generation module 605 is used to perform environmental and risk analysis based on fused data and generate decision suggestions for path planning, obstacle avoidance, and hazard identification.

[0057] In this embodiment, the data preprocessing module 601 performs time synchronization, spatial registration, and data cleaning on the heterogeneous raw data collected by various sensors on the explosion-proof robot. Time synchronization uses timestamp alignment to unify the timing of multiple sensors; spatial registration unifies the data from each sensor to a global coordinate system; data cleaning removes noise, outliers, and redundant data. The feature extraction module 602 extracts key features from the heterogeneous data according to its type, constructing a multi-dimensional feature vector. The heterogeneous data types include visual data, LiDAR data, gas sensor data, and sound sensor data. Visual data uses a deep learning model to extract image features; LiDAR data extracts point cloud features and spatial distribution features; gas sensor data extracts concentration change rate and feature patterns; and sound sensor data extracts spectral features and voiceprint features. The dynamic weighting... The module 603 determines the importance of sensors based on the task scenario, assigns basic weights, and adjusts real-time weights according to real-time requirements. An adaptive algorithm integrates reliability, importance, and real-time performance to dynamically output the fusion weights for each sensor. The data fusion module 604 constructs a preliminary fusion framework based on evidence theory, treating each sensor's data as an independent source of evidence to complete uncertain fusion. A deep learning network is used to learn the complex relationships between heterogeneous data, and dynamic weights are input as guiding parameters to the network to achieve feature-weighted fusion and output fused data. The decision suggestion generation module 605 performs environmental and risk analysis based on the fused data, generating decision suggestions for path planning, obstacle avoidance, and hazard identification. Through these methods, the robot can adapt to environmental changes and dynamically adjust fusion weights, improving the reliability of robot decision-making and thus enhancing the environmental perception accuracy and operational safety of the explosion-proof robot.

[0058] Regarding the system in the above embodiments, the specific ways in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0059] For the system embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0060] Accordingly, this application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the dynamic data fusion method for explosion-proof robots in heterogeneous environments as described above. Figure 8 The diagram shown is a hardware structure diagram of any device with data processing capabilities, used in a dynamic data fusion system for explosion-proof robots in heterogeneous environments, according to an embodiment of the present invention. (Except for...) Figure 8 In addition to the processor, memory, and network interface shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0061] Accordingly, this application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the dynamic data fusion method for explosion-proof robots in heterogeneous environments as described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0062] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0063] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A dynamic data fusion method for explosion-proof robots in heterogeneous environments, characterized in that, Includes the following steps: The heterogeneous raw data collected by multiple sensors on the explosion-proof robot are subjected to time synchronization, spatial registration and data cleaning. Among them, time synchronization uses timestamp alignment to achieve time sequence unification of multiple sensors; spatial registration unifies the data of each sensor to the global coordinate system; data cleaning removes noise, outliers and redundant data. Based on the type of heterogeneous data, key features of the heterogeneous data are extracted and multi-dimensional feature vectors are constructed. The types of heterogeneous data include visual data, LiDAR data, gas sensor data, and sound sensor data. Visual data uses a deep learning model to extract image features, LiDAR data extracts point cloud features and spatial distribution features, gas sensor data extracts concentration change rate and feature patterns, and sound sensor data extracts spectral features and voiceprint features. The importance of sensors is determined by the task scenario, basic weights are assigned, and real-time weights are adjusted according to real-time requirements. The fusion weights of each sensor are dynamically output by integrating reliability, importance, and real-time performance through an adaptive algorithm. A preliminary fusion framework is constructed based on evidence theory. Data from each sensor is used as an independent source of evidence to complete uncertainty fusion. A deep learning network is used to learn the complex relationships between heterogeneous data. Dynamic weights are used as guiding parameters to input into the network to achieve feature-weighted fusion and output fused data. Based on fused data, environmental and risk analysis is performed to generate decision suggestions for path planning, obstacle avoidance, and hazard identification.

2. The dynamic data fusion method for explosion-proof robots in heterogeneous environments as described in claim 1, characterized in that, The process involves performing time synchronization, spatial registration, and data cleaning on heterogeneous raw data collected by multiple sensors mounted on an explosion-proof robot. Time synchronization uses timestamp alignment to unify the timing of multiple sensors; spatial registration unifies the data from each sensor to a global coordinate system; and data cleaning removes noise, outliers, and redundant data. Acquire heterogeneous raw data collected by various sensors mounted on the explosion-proof robot; Use timestamp alignment to unify the time dimension of heterogeneous raw data; Transform heterogeneous raw data into a global coordinate system to eliminate spatial differences between data. The heterogeneous raw data is filtered and screened to remove noise, outliers, and redundant information, and the heterogeneous data is output.

3. The dynamic data fusion method for explosion-proof robots in heterogeneous environments as described in claim 1, characterized in that, Based on the type of heterogeneous data, key features of the heterogeneous data are extracted, and multi-dimensional feature vectors are constructed. The types of heterogeneous data include visual data, LiDAR data, gas sensor data, and sound sensor data. Visual data uses a deep learning model to extract image features; LiDAR data extracts point cloud features and spatial distribution features; gas sensor data extracts concentration change rate and feature patterns; and sound sensor data extracts spectral features and voiceprint features. Determine the type of heterogeneous data; the types of heterogeneous data include visual data, LiDAR data, gas sensor data, and sound sensor data; For visual data, a deep learning model is used to perform convolution, pooling, and feature mapping operations to extract high-dimensional semantic features of edges, textures, contours, and target regions in the image layer by layer, so as to obtain a standardized image feature vector and output the visual data features. For LiDAR data, the point cloud data is downsampled and filtered for optimization. The normal vector, curvature, and neighborhood density are calculated point by point to extract the structural features of the point cloud. The distribution density, distance distribution, and orientation distribution of the point cloud in three-dimensional space are statistically analyzed to obtain spatial distribution-related features and output the LiDAR data features. For gas sensor data, differential calculation is performed on the concentration time series data to obtain the rate of change of gas concentration per unit time. Combined with the concentration fluctuation amplitude, stable duration, and abrupt change threshold, feature patterns are extracted to obtain feature data that can characterize the gas state and output the gas sensor data features. For sound sensor data, the audio data is converted from the time domain to the frequency domain, and the spectral features of amplitude, energy and frequency components are extracted. Based on the unique properties of the sound signal, the voiceprint features are extracted to obtain feature information that can distinguish the type and state of the sound source, and the sound sensor data features are output.

4. The dynamic data fusion method for explosion-proof robots in heterogeneous environments as described in claim 3, characterized in that, After performing time-domain and frequency-domain transformations on the audio data from the sound sensor, extracting spectral features of amplitude, energy, and frequency components, and extracting voiceprint features based on the unique properties of the sound signal to obtain feature information that can distinguish the type and state of the sound source, and outputting the sound sensor data features: The visual data features, lidar data features, gas sensor data features, and sound sensor data features are normalized and then spliced ​​together according to dimensions to form a multi-dimensional feature vector.

5. The dynamic data fusion method for explosion-proof robots in heterogeneous environments as described in claim 1, characterized in that, In the steps of determining sensor importance based on the task scenario, assigning basic weights, adjusting real-time weights according to real-time requirements, and dynamically outputting the fused weights of each sensor through an adaptive algorithm that integrates reliability, importance, and real-time performance: Based on the current task scenario, determine the importance of each sensor's data and assign basic weights; Adjust the real-time weights of the corresponding sensor data according to the task's requirements for data real-time performance; By integrating reliability, importance, and real-time factors through an adaptive algorithm, the final dynamic fusion weights of each sensor are output.

6. The dynamic data fusion method for explosion-proof robots in heterogeneous environments as described in claim 5, characterized in that, Before determining the importance of each sensor's data and assigning basic weights based on the current task scenario: The reliability of a sensor is determined by its historical data accuracy, current operating status, and environmental adaptability.

7. The dynamic data fusion method for explosion-proof robots in heterogeneous environments as described in claim 1, characterized in that, In the steps of constructing a preliminary fusion framework based on evidence theory, treating each sensor's data as an independent source of evidence to complete uncertainty fusion, employing a deep learning network to learn the complex relationships between heterogeneous data, and inputting dynamic weights as guiding parameters into the network to achieve feature-weighted fusion and output fused data: A preliminary fusion framework was built based on evidence theory, and the data from each sensor was used as an independent source of evidence to complete the fusion of uncertainty information. Construct a deep learning network to learn and explore the complex relationships between different types of heterogeneous data, and input dynamic weights as guiding parameters into the deep learning network to perform weighted fusion of features; By iteratively optimizing the deep learning network through a preset loss function, adjusting the network parameters, and outputting globally fused data that can comprehensively represent the current environmental state and is compatible with multi-source heterogeneous information.

8. The dynamic data fusion method for explosion-proof robots in heterogeneous environments as described in claim 7, characterized in that, In the step of building a preliminary fusion framework based on evidence theory and using data from each sensor as an independent source of evidence to complete the fusion of uncertain information: A basic probability allocation function is established for the feature data corresponding to each sensor. The feature information output by different sensors is regarded as an independent source of evidence. The multi-source evidence is fused and calculated through evidence combination rules to complete the normalization and credibility synthesis of uncertain and conflicting information, and obtain the preliminary fusion result.

9. The dynamic data fusion method for explosion-proof robots in heterogeneous environments as described in claim 1, characterized in that, In the steps of completing environmental and risk analysis based on fused data and generating decision suggestions for path planning, obstacle avoidance, and hazard identification: Based on the fused data, the current environmental status and potential safety risks are analyzed and assessed, and an appropriate robot path planning scheme is generated based on the results of the environmental and risk analysis. Based on the distribution of environmental obstacles, generate obstacle avoidance strategies.

10. The dynamic data fusion method for explosion-proof robots in heterogeneous environments as described in claim 9, characterized in that, After the step of generating obstacle avoidance strategies based on the distribution of environmental obstacles: Detect the characteristics of hazardous scenes, identify the sources of danger, and generate corresponding decision-making recommendations.

Images