An environment perception method and system based on multi-source data fusion

Abstract

The application discloses an environment perception method and system based on multi-source data fusion, relates to the technical field of environment perception decision, and comprises the following steps: node mapping and multi-level graph convolution iterative updating are performed on synchronous correction data by using a graph neural network to generate dynamic scene data; the dynamic scene data is combined with multi-modal perception data to generate fusion feature data through feature coding, interactive coupling and dimension optimization; target detection and classification are performed on the fusion feature data to generate target state information; the target state information is combined with the dynamic scene data to generate target prediction trajectory data through target tracking and future motion prediction; and environment perception analysis and decision derivation are performed on the target prediction trajectory data to output environment perception decision results. Through graph neural network modeling on the synchronous correction multi-modal perception data, a multi-level node interaction structure is constructed and iteratively updated, and accurate space-time feature representation of a complex dynamic scene is realized.

Description

Technical Field

[0001] This invention relates to the field of environmental perception and decision-making technology, and in particular to an environmental perception method and system based on multi-source data fusion. Background Technology

[0002] In recent years, with the rapid development of intelligent mobile systems, autonomous driving, and robotic autonomous navigation technologies, environmental perception technology has gradually become a core research direction in related fields. Traditional environmental perception methods mainly rely on data acquired by a single sensor, such as LiDAR, cameras, or inertial measurement units (IMUs), and analyze environmental information through point cloud processing, image analysis, or motion estimation. These methods can achieve relatively accurate environmental modeling and target recognition in static or low-complexity scenarios.

[0003] Existing methods still have room for improvement in dynamic scene modeling, feature fusion, and prediction capabilities. On the one hand, traditional fusion methods are mostly based on manual feature design or shallow feature combination, which makes it difficult to fully characterize the spatial-temporal coupling relationship between multimodal data in complex environments, thus limiting the accurate inference of dynamic target behavior. On the other hand, in terms of multimodal data synchronization, denoising, and handling of missing values, existing methods usually adopt simple interpolation or filtering strategies, which may lead to the loss of key information, thereby affecting the accuracy of subsequent target detection, tracking, and prediction. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides an environment perception method based on multi-source data fusion to solve the problems of insufficient multimodal information fusion and limited target prediction accuracy in dynamic scenes.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides an environmental perception method based on multi-source data fusion, which includes collecting multimodal perception data, performing time synchronization, denoising, correction and missing value filling to obtain synchronized correction data;

[0008] Graph neural networks are used to perform node mapping and multi-level graph convolution iterative updates on synchronous correction data to generate dynamic scene data.

[0009] By combining dynamic scene data with multimodal perception data, feature encoding, interaction coupling, and dimensionality optimization are used to generate fused feature data;

[0010] Target detection and classification are performed on the fused feature data to generate target state information;

[0011] By combining target state information with dynamic scene data, target tracking and future motion prediction are used to generate target predicted trajectory data.

[0012] Perform environmental perception analysis and decision derivation on the target predicted trajectory data, and output the environmental perception decision results.

[0013] As a preferred embodiment of the environmental perception method based on multi-source data fusion described in this invention, the specific steps for acquiring multimodal perception data, performing time synchronization, denoising, correction, and missing value imputation to obtain synchronized and corrected data are as follows.

[0014] Multimodal sensing data is acquired in real time and triggered at a unified time to form initial sensor data. Then, timestamp alignment and time interpolation correction are performed to obtain time-synchronized data.

[0015] The time synchronization data is noise-suppressed to generate denoised data.

[0016] Missing regions in the denoised data are smoothed and filled in to obtain synchronized correction data.

[0017] As a preferred embodiment of the environmental perception method based on multi-source data fusion described in this invention, the specific steps for generating dynamic scene data by using a graph neural network to perform node mapping and multi-level graph convolution iterative updates on the synchronous correction data are as follows.

[0018] Environmental element features are extracted from synchronous correction data, and node mapping is performed through multi-dimensional feature association and spatial semantic projection to generate an initial node set.

[0019] The initial set of nodes is subjected to topological relationship construction, edge weight assignment and space-time dependency analysis to generate a multi-level node interaction structure.

[0020] By using space-time coupled feature reasoning, a dynamic feature representation is obtained from the multi-level node interaction structure.

[0021] Multi-level graph convolution iterative updates are performed on dynamic feature representations to generate dynamic scene data.

[0022] As a preferred embodiment of the environment perception method based on multi-source data fusion described in this invention, the specific steps for performing multi-level graph convolution iterative updates on the dynamic feature representation to generate dynamic scene data are as follows:

[0023] The dynamic feature representation is mapped to graph nodes and the edge relationships between nodes are established to form an initial graph structure. The node feature vectors are generated by iterative updating through multi-level graph convolution.

[0024] The node feature vectors are integrated and normalized according to topological order to generate dynamic scene data.

[0025] As a preferred embodiment of the environmental perception method based on multi-source data fusion described in this invention, the specific steps for generating fused feature data by combining dynamic scene data with multimodal perception data through feature encoding, interactive coupling, and dimensionality optimization are as follows.

[0026] Spatial and temporal attributes are extracted from dynamic scene data, and deep feature encoding and feature fusion are performed in combination with multimodal perception data to generate an initial feature vector set.

[0027] Nonlinear interactive computation and multi-layer feature coupling are performed on the initial feature vector set to obtain a fused feature representation;

[0028] The feature dimension of the fused feature representation is optimized and the global normalization is performed to generate fused feature data.

[0029] As a preferred embodiment of the environment perception method based on multi-source data fusion described in this invention, the specific steps for optimizing the feature dimension and globally normalizing the fused feature representation to generate fused feature data are as follows.

[0030] Feature importance analysis and weight calculation are performed on the fused feature representation to obtain the optimized fused feature vector;

[0031] The fused feature optimization vector is then processed through global normalization and structural integration to output fused feature data.

[0032] As a preferred embodiment of the environmental perception method based on multi-source data fusion described in this invention, the specific steps for performing target detection and classification on the fused feature data to generate target state information are as follows:

[0033] Spatial rasterization decomposition and multidimensional feature mapping are performed on the fused feature data to form a set of candidate elements. Global interactive correlation calculation and nonlinear confidence mapping are then performed to form a scoring vector.

[0034] Spatial overlap analysis and category association suppression are performed on the scoring vectors to obtain the target detection result set;

[0035] The target detection result set is processed for velocity and trajectory calculation, timestamp annotation, and attribute integration to generate target status information.

[0036] As a preferred embodiment of the environmental perception method based on multi-source data fusion described in this invention, the specific steps for generating target prediction trajectory data by combining target state information with dynamic scene data through target tracking and future motion prediction are as follows.

[0037] The position, velocity, category, bounding box size, and timestamp of each target are extracted from the target state information, and then structured and vectorized to form the target initial vector.

[0038] The target initial vector is subjected to multi-target global correlation and state update to obtain the target state vector;

[0039] By combining the target state vector and dynamic scene data, future motion prediction and interaction constraint calculation are performed to generate a predicted position sequence. Then, through formatted encapsulation and continuous time interpolation, the target predicted trajectory data is generated.

[0040] As a preferred embodiment of the environmental perception method based on multi-source data fusion described in this invention, the specific steps for performing environmental perception analysis and decision derivation on the target predicted trajectory data and outputting the environmental perception decision result are as follows.

[0041] Neighborhood feature encoding and spatial attribute coupling are performed on the target predicted trajectory data to generate an interactive feature vector set. Then, environmental perception risk confidence is generated through global nonlinear scoring calculation and normalization mapping.

[0042] Multi-dimensional weight decoupling and feature reconstruction operations are performed on the environmental perception risk confidence to generate a single-objective environmental decision vector.

[0043] The single-objective environmental decision vectors are integrated by global sorting and spatial grouping to generate environmental perception decision results.

[0044] Secondly, the present invention provides an environmental perception system based on multi-source data fusion, including a data preprocessing module for collecting multimodal perception data, performing time synchronization, denoising, correction and missing value filling to obtain synchronized correction data;

[0045] The graph modeling update module is used to perform node mapping and multi-level graph convolution iterative update on the synchronous correction data using graph neural networks to generate dynamic scene data.

[0046] The feature fusion module is used to combine dynamic scene data with multimodal perception data through feature encoding, interactive coupling and dimensional optimization to generate fused feature data;

[0047] The target recognition module is used to detect and classify targets from fused feature data and generate target status information.

[0048] The target prediction module is used to combine target state information with dynamic scene data to generate target prediction trajectory data through target tracking and future motion prediction.

[0049] The decision analysis module is used to perform environmental perception analysis and decision derivation on the target predicted trajectory data, and output the environmental perception decision results.

[0050] The beneficial effects of this invention are as follows: by modeling the synchronously calibrated multimodal perception data using graph neural networks, constructing a multi-level node interaction structure and iteratively updating it, accurate spatiotemporal feature representation of complex dynamic scenes is achieved; then, by performing deep feature encoding, interactive coupling and dimensional optimization on dynamic scene data and multimodal information, highly discriminative fusion features are generated, thereby providing reliable input for target detection, tracking, prediction and environmental decision-making, improving the overall accuracy, continuity and robustness of environmental perception, and enhancing the reliability and adaptability of decision-making in complex dynamic scenes. Attached Figure Description

[0051] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a flowchart of an environment perception method based on multi-source data fusion.

[0053] Figure 2 This is a schematic diagram of an environmental perception system based on multi-source data fusion.

[0054] Figure 3 A flowchart for data preprocessing.

[0055] Figure 4 This is a flowchart for object detection and classification. Detailed Implementation

[0056] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0057] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0058] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0059] Reference Figures 1-4This is one embodiment of the present invention, which provides an environment perception method based on multi-source data fusion, including the following steps:

[0060] S1. Collect multimodal sensing data, perform time synchronization, denoising, correction, and missing value imputation to obtain synchronized and corrected data.

[0061] S1.1. Real-time acquisition and unified time triggering of multimodal sensing data are performed to form initial sensor data, and timestamp alignment and time interpolation correction are performed to obtain time synchronization data.

[0062] Furthermore, sensor synchronization sampling is performed based on a unified time trigger signal to form initial sensor data, which includes the sampling timestamp and corresponding measurement value of each sensor. Timestamp alignment is then performed on the initial sensor data. This alignment uses a unified time reference (e.g., GPS or Network Time Protocol) as the time base, calculates the time offset for each sensor, and applies corresponding offset correction to each sensor timestamp, mapping all initial sensor data to the same time reference coordinate system. After timestamp alignment, time interpolation correction is performed on initial sensor data with insufficient time synchronization accuracy or inconsistent sampling rates. Time interpolation correction resamples the time axis at equal intervals according to a predetermined interpolation method. The interpolation method can use linear interpolation or cubic spline interpolation to reconstruct the measurement value at a unified time point, and the interpolation result is smoothed to suppress high-frequency noise. The interpolated outputs after timestamp alignment and time interpolation correction are integrated in chronological order and subjected to consistency checks and missing point repair to output time-synchronized data.

[0063] S1.2. Suppress noise in the time synchronization data to generate denoised data.

[0064] Furthermore, a sliding window is used to estimate local noise statistics on the time-synchronized data. The mean and variance within the window are calculated, and outliers are identified using the absolute deviation of the median. For time-synchronized data identified as outliers, median replacement or time interpolation is used to remove impulsive noise. A low-pass filter (e.g., finite impulse response filter or Butterworth low-pass filter) is applied to the outlier-processed time-synchronized data to suppress high-frequency noise components and retain low-frequency signal components. For time-synchronized data that needs to retain transient change characteristics, wavelet transform is used to decompose the time-synchronized data, and the time-domain signal is reconstructed after smoothing constraints are applied to the wavelet coefficients. Kalman filtering is applied to the filtered or reconstructed time-synchronized data for smoothing. The process noise variance and measurement noise variance in the Kalman filter are initialized by the sliding window estimate. After smoothing, the integrity of the time axis is checked, and time interpolation is used to repair missing points, outputting denoised data.

[0065] S1.3. Smoothly fill and complete the missing regions in the denoised data to obtain synchronous correction data.

[0066] Furthermore, the start and end times of missing intervals in the denoised data are identified, and the missing length and adjacent valid sample sequences are labeled. For short-term missing intervals, cubic spline interpolation or linear interpolation is used to reconstruct the missing sampling points along the time axis while maintaining consistency with the continuous first derivative of adjacent valid samples. For medium- to long-term missing intervals, a sliding window is used to perform first- or second-order polynomial fitting on adjacent valid samples in the denoised data and extrapolation is performed along the time direction to fill the gaps. If necessary, a Kalman filter prediction step is used to use the time series state estimation results as the filling values ​​to take into account the statistical characteristics of noise. For scenarios with spatial or channel correlation in multimodal sensing data, a collaborative interpolation method based on available channels is used to complete the collaborative completion of missing intervals through least squares fitting using synchronous observations of adjacent channels. After completion, a sliding window moving average or low-pass filter is applied to the completion results to smooth them and remove interpolation / high-frequency artifacts. The endpoint continuity and whole-sequence consistency of the smoothed results are checked. If residual outliers are found, they are replaced with the median or repaired by interpolation again according to the outlier handling rules, and synchronous correction data is output.

[0067] S2. Utilize graph neural networks to perform node mapping and multi-level graph convolution iterative updates on the synchronous correction data to generate dynamic scene data.

[0068] Existing methods typically employ shallow feature propagation or fixed neighborhood aggregation based on graph structures to perform simple node mapping and one-time feature updates on synchronous correction data. The number of graph convolutional layers is limited, making it difficult to fully capture the multi-level spatial-temporal dependencies between environmental elements. The generated dynamic scene data has a weak ability to represent complex environments.

[0069] This invention utilizes graph neural networks to map each environmental element node in the synchronous correction data and iteratively updates it through multi-level graph convolution. In each layer, neighborhood information aggregation, linear transformation, and nonlinear activation processing are performed. At the same time, normalization and residual connections are combined to maintain numerical stability. Layer by layer, it captures multi-dimensional interaction relationships such as spatial proximity, temporal adjacency, and semantic similarity, generating dynamic scene data that can fully reflect the dynamic characteristics of the environment.

[0070] S2.1 Extract environmental element features from the synchronous correction data, and perform node mapping through multi-dimensional feature association and spatial semantic projection to generate an initial node set.

[0071] Furthermore, environmental element features are extracted from the synchronous calibration data according to time series and spatial coordinates, including location coordinates, velocity vectors, bounding box size, category labels, observation confidence, and timestamps. Multidimensional feature association is implemented on the extracted environmental element features. An association mapping is established in the feature space by calculating spatial similarity (e.g., Euclidean distance), temporal correlation (e.g., time-delay cross-correlation), and semantic similarity (e.g., cosine similarity), and the association mapping is represented in the form of a neighborhood matrix or similarity matrix. In the spatial semantic projection stage, the multidimensional feature association mapping is projected onto a unified spatial semantic coordinate system. Principal component analysis or multidimensional scaling is used to map high-dimensional features into low-dimensional semantic vectors to preserve spatial and semantic information. Based on the projected semantic vectors, node mapping operations are performed on each group of environmental element features. Each group of environmental element features is represented as a node and added to the initial node set in time series order. Spatial connectivity and time series continuity checks are performed on the initial node set, and any inconsistencies are supplemented or corrected. The initial node set is then output.

[0072] S2.2. Perform topological relationship construction, edge weight assignment, and space-time dependency analysis on the initial node set to generate a multi-level node interaction structure.

[0073] Furthermore, for each node pair in the initial node set, similarity indices such as Euclidean spatial distance, temporal overlap, and semantic similarity are calculated, and each similarity index is normalized. Based on the principles of spatial proximity, temporal adjacency, and semantic consistency, edge connections are established between node pairs that satisfy the corresponding relationship criteria. The edge connection simultaneously records normalized spatial similarity, normalized temporal correlation, and normalized semantic similarity as edge attributes. Linear fusion is performed on the edge attributes to generate edge weights. The linear fusion uses simple summation and proportional scaling of normalized indices to form comparable edge weights. The temporal coupling strength of node pairs is analyzed by applying time-delay cross-correlation functions and mutual information to the initial node set and edge set. The stability and abrupt change characteristics of node attributes over time are evaluated through sliding window statistical analysis. The spatial proximity layer relationship, semantic similarity layer relationship, and temporal coupling layer relationship are organized hierarchically, and the node attribute and edge attribute information are integrated to output a multi-level node interaction structure.

[0074] S2.3. The multi-level node interaction structure is used to obtain a dynamic feature representation through space-time coupled feature reasoning.

[0075] Furthermore, using a multi-level node interaction structure as input, neighborhood message information is extracted in the topological dimension. Graph convolution operations in graph neural networks are used to aggregate the node attribute vectors of neighboring nodes along the spatial neighborhood of each node to generate spatial feature representations. In the temporal dimension, temporal feature extraction operations are applied to the spatial feature representations. Temporal convolution or recurrent temporal units are used to capture the time-varying patterns of node attributes along the time axis to generate temporal feature representations. In the semantic dimension, spatial and temporal feature representations are interactively fused. An attention mechanism is used to associate and score the spatial and temporal feature representations and to selectively update the node representations based on information, thereby simultaneously reflecting the topological relationships between nodes, the similarity of node attributes, and the temporal coupling characteristics. Normalization and nonlinear transformations are performed on the fused node representations to improve representation stability and suppress outliers. The normalization and nonlinear transformations are performed using batch normalization or layer normalization and differentiable activation functions. The node representations after normalization and nonlinear transformations are integrated in hierarchical order to generate dynamic feature representations.

[0076] S2.4 Map the dynamic feature representation to graph nodes and establish edge relationships between nodes to form an initial graph structure. Then, iterate and update the graph through multi-level graph convolution to generate node feature vectors.

[0077] Furthermore, each node attribute vector, represented by dynamic features, is directly used as the initial node representation of the graph node. All graph nodes are arranged according to spatial location, timestamp, and semantic category index. Edge connections are established between graph nodes based on three criteria: spatial proximity, temporal adjacency, and semantic similarity. Spatial proximity is determined by calculating the Euclidean distance between nodes and determining spatially related node pairs based on the principle of minimum distance; temporal adjacency is determined by comparing the differences in node timestamps and determining temporally contiguous node pairs based on the principle of temporal proximity; semantic similarity is determined by calculating the cosine similarity of node attribute vectors and determining semantically close node pairs based on the principle of maximum similarity. After edge connection determination, node pairs that satisfy any criterion are incorporated into the initial graph structure as edge connections. Using the initial graph structure as input, multi-level graph convolution iterative updates are performed hierarchically. In each layer, neighborhood information aggregation (e.g., taking the mean or sum of neighborhood representations) is performed along the graph neighborhood for each graph node, and linear transformation and differentiable activation function transformation are applied to the current representation of the graph node. Then, normalization processing and residual connections are performed to maintain numerical stability. After iterating through multi-level graph convolution several times, the node feature vectors are output.

[0078] S2.5 Integrate and normalize the node feature vectors according to topological order to generate dynamic scene data.

[0079] Furthermore, the node feature vectors are sorted and their topological order is determined according to the topological index and timestamp in the initial graph structure. The node feature vectors are then concatenated along the feature dimension according to the topological order to form a topological feature matrix. During concatenation, zero-padding or time interpolation is used to align sequences of unequal length and the effective length is recorded. Feature dimension normalization is performed on each column of the topological feature matrix. Normalization can be performed using min-max normalization or Z-score normalization, supplemented by batch normalization or layer normalization to stabilize the numerical distribution. Subsequently, outlier detection is performed on the normalization results, and extreme values ​​are replaced or pruned using the median. The normalized and corrected topological feature matrix is ​​then restored to a temporal node sequence according to the topological order and output as dynamic scene data.

[0080] S3. Dynamic scene data is combined with multimodal perception data to generate fused feature data through feature encoding, interactive coupling and dimensional optimization.

[0081] Existing methods typically perform simple feature concatenation or linear combination on dynamic scene data and multimodal perception data. They may combine basic statistical features or shallow coding methods to generate fused features, but the nonlinear interaction relationships and multidimensional coupling information between features are difficult to fully express, resulting in limited ability of fused features to represent target state or scene information in complex environments.

[0082] This invention performs deep feature encoding on dynamic scene data and multimodal perception data, nonlinearly couples spatial attributes, temporal attributes and multimodal features, and integrates and compresses high-dimensional features by combining dimensionality optimization operations, so that the fused feature data can fully reflect the correlation between different modalities and time series, and improve the discriminative ability of features and the ability to express environmental information.

[0083] S3.1 Extract spatial and temporal attribute features from dynamic scene data, and combine them with multimodal perception data for deep feature encoding and feature fusion to generate an initial feature vector set.

[0084] Furthermore, spatial and temporal attributes are extracted from dynamic scene data according to time series, including spatial and temporal attributes such as position coordinates, bounding box size, orientation angle, motion velocity, acceleration, and trajectory curvature. Modal features are extracted from multimodal perception data: visual modality uses convolutional neural networks to extract visual texture and semantic representation; point cloud modality uses point cloud feature extraction algorithms to obtain 3D shape and distance distribution representation; radar modality uses spectral feature extraction methods to obtain velocity and Doppler representation; and acoustic modality uses short-time Fourier transform to extract time-frequency representation. The spatial and temporal attributes are then matched with the modal features at timestamps. The system unifies the dimensions through feature normalization. The normalized spatial and temporal attributes and modal features are then encoded using deep features. The deep feature encoding uses convolutional neural networks to process visual features, recurrent neural networks or temporal convolutional networks to process time series features, and graph neural networks to process inter-node relationship features. A cross-modal attention mechanism is implemented on the encoded features to perform interactive fusion to enhance complementary information and suppress redundant information. After interactive fusion, the fused features are subjected to dimensionality compression and principal component analysis or autoencoder dimensionality reduction to obtain low-dimensional representations. The low-dimensional representations are then concatenated according to node index and time series order to form an initial feature vector set.

[0085] It should also be noted that the point cloud feature extraction algorithm performs spatial registration and noise filtering on the original point cloud data, and then uses existing methods such as PointNet, PointNet++, Dynamic Graph Convolutional Neural Network, or VoxelNet to extract three-dimensional shape and distance distribution features. Among them, the PointNet algorithm uses a multilayer perceptron to obtain a global shape representation, the PointNet++ algorithm extracts local geometric structure through hierarchical sampling, the Dynamic Graph Convolutional Neural Network algorithm dynamically constructs graph relationships based on the distance between points to capture neighborhood differences, and the VoxelNet algorithm voxels the point cloud and performs three-dimensional convolution to extract voxel distribution features, thus obtaining a point cloud feature representation containing geometric shape and spatial distance information.

[0086] S3.2. Perform nonlinear interactive calculations and multi-layer feature coupling on the initial feature vector set to obtain the fused feature representation.

[0087] Furthermore, the initial feature vector set is batch-divided according to node index and time series order, and the spatial attribute, temporal attribute, and multimodal feature of each node are used as independent input feature vectors. Nonlinear interactive computation is performed on the spatial attribute features, temporal attribute features, and multimodal features within each input feature vector. Existing nonlinear activation functions (such as ReLU, Sigmoid, or Tanh) are used to map the interactions between features to enhance feature expressive power. Multi-layer feature coupling operations are performed on the feature vectors after nonlinear interactive computation along the feature dimension. The coupling operations include fully connected mapping across the feature dimension, multi-level linear transformation, and residual connection to fuse complementary information of spatial attribute, temporal attribute, and multimodal features. After completing multi-layer feature coupling, the fusion result is batch-normalized or layer-normalized to stabilize the numerical distribution, and extreme outliers are pruned or replaced with the median. The normalized and coupled feature vectors are integrated and output as fused feature representations according to node index and time series order.

[0088] S3.3 Perform feature importance analysis and weight calculation on the fused feature representation to obtain the optimized fused feature vector.

[0089] Furthermore, for each feature dimension in the fused feature representation, the distribution characteristics are analyzed by calculating the mean, variance, and maximum / minimum range of that dimension across all samples or time series. The fluctuation amplitude and stability of each feature dimension across different samples or time periods are further statistically analyzed to assess the contribution of each feature to the overall feature representation. Existing mutual information feature importance assessment methods are used, such as feature scores calculated based on information gain, variance contribution, or mutual information, to assign corresponding weight values ​​to each feature dimension. These weight values ​​reflect the relative importance of the feature in the prediction target or task. Each feature dimension in the fused feature representation is multiplied by its corresponding weight value, and the resulting feature vector is normalized to maintain a stable numerical range. Normalization can employ min-max normalization or Z-score normalization. The feature vectors after feature importance analysis, weight adjustment, and normalization are then integrated to form the optimized fused feature vector.

[0090] It should also be noted that the prediction target is the specific output variable corresponding to the fused feature representation, which is used to guide the mutual information feature importance assessment process. It comes from the task definition or the existing annotation results in the dataset. For example, in the environmental perception task, it is the environmental risk level; in the trajectory prediction task, it is the future location sequence; and in the behavior recognition task, it is the action or state category. It is used to measure the correlation between each feature dimension and the task output.

[0091] The mutual information feature importance assessment method is a feature analysis method based on information theory. It measures the amount of useful information provided by a feature by calculating the mutual information value between each feature and the prediction target. The higher the mutual information value, the stronger the correlation between the feature and the prediction target, and the greater its contribution to the overall prediction task. It can be used for feature selection or feature optimization.

[0092] S3.4. The fused feature optimization vector is global normalized and structurally integrated to output fused feature data.

[0093] Furthermore, the fused feature optimization vectors are globally normalized according to feature dimensions. Min-max normalization or Z-score normalization methods are used to map the values ​​of all feature dimensions to a unified range, ensuring that each feature dimension is comparable under the same scale. The normalized fused feature optimization vectors are then structurally integrated based on node indices, time series, and feature categories. This integration includes concatenating feature vectors in node order, connecting consecutive sequences of features from different time steps, and uniformly arranging multimodal feature categories to maintain a consistent sequence structure across the spatiotemporal and modal dimensions. During structural integration, missing feature positions are interpolated or zero-padded to ensure consistent feature vector lengths. The integrated feature sequence is then normalized overall to stabilize the numerical distribution. The normalized and structurally integrated fused feature optimization vector set is output as the fused feature data.

[0094] S4. Perform target detection and classification on the fused feature data to generate target state information.

[0095] S4.1. Spatial rasterization decomposition and multidimensional feature mapping are performed on the fused feature data to form a set of candidate elements. Global interactive correlation calculation and nonlinear confidence mapping are then performed to form a scoring vector.

[0096] Furthermore, the fused feature data is divided into 3D grids based on spatial coordinates. Each grid cell contains node feature vectors and time-series features within the corresponding spatial region. Multidimensional feature mapping is performed on the fused feature data within each grid cell, mapping spatial attributes, temporal attributes, and multimodal features to a unified multidimensional feature representation for subsequent comparison and calculation. Global interaction correlation calculation is performed on the multidimensional feature representations of all grid cells in the candidate feature set. By calculating the spatial proximity, temporal consistency, and feature similarity between grid cells, a global interaction relationship matrix is ​​formed. For each candidate feature in the global interaction relationship matrix, a confidence score is calculated using a nonlinear confidence mapping function (such as Sigmoid or Softmax). The comprehensive confidence value of each candidate feature is used as the corresponding element of the scoring vector. The confidence scores of all candidate features are integrated and output to generate a scoring vector.

[0097] S4.2 Perform spatial overlap analysis and category association suppression on the scoring vectors to obtain the target detection result set.

[0098] Furthermore, spatial overlap analysis is performed on the scoring vectors. By comparing the degree of overlap between the target bounding boxes in space, the spatial relationship between candidate targets is determined. For targets with high spatial overlap, category association suppression is performed. By calculating the similarity and interaction of candidate target category vectors, potentially duplicate or conflicting category predictions are suppressed. The candidate target set after spatial overlap analysis and category association suppression is screened and integrated to form the target detection result set.

[0099] S4.3 Calculate the velocity and trajectory, add timestamps and integrate attributes on the target detection result set to generate target status information.

[0100] Furthermore, based on the position sequence of each target in the target detection result set, a velocity vector is calculated on consecutive time steps. The velocity calculation adopts the method of dividing the position difference between adjacent time steps by the time interval, and instantaneous velocity information of the target is generated. The trajectory of each target is serialized, and the position and velocity information of consecutive time steps are combined in chronological order to form a complete trajectory sequence. A timestamp is marked for each trajectory point according to the detection time to ensure the temporal consistency of the target state. The category information, bounding box size and other attribute information of each target are integrated, and the velocity, trajectory, timestamp, category and bounding box size are uniformly encoded into a structured vector. The integrated target information set is output as target state information.

[0101] S5. Combine target state information with dynamic scene data to generate target prediction trajectory data through target tracking and future motion prediction.

[0102] S5.1 Extract the position, velocity, category, bounding box size and timestamp of each target from the target state information, perform structured combination and vectorization processing to form the target initial vector.

[0103] Furthermore, for each target in the target state information, the position coordinates, velocity vector, category identifier, bounding box size, and timestamp are read sequentially, and these attributes are arranged in a fixed order to form an attribute list. Then, each attribute list is converted into a unified numerical or vector representation. For example, the category identifier is converted into a vector form through one-hot encoding, and the bounding box size and velocity vector are kept as numerical vector representations. The attribute vectors are concatenated in sequence to form the target initial vector of a single target, ensuring that spatial, motion, category, and temporal information are completely expressed in the vector. The same operation is performed on all targets in sequence to generate the target initial vector.

[0104] S5.2 Perform multi-target global association and state update on the initial target vector to obtain the target state vector.

[0105] Furthermore, global similarity and association calculations are performed on each target vector in the initial target vector set with other target vectors, including spatial location similarity, velocity direction consistency, category matching, and timestamp interval evaluation, to identify vector combinations that may belong to the same target or continuous states. Based on the global association results, the initial target vectors are updated in state, including correcting position and velocity estimates, updating category confidence, adjusting bounding box size and timestamp information, so that each target vector reflects the latest globally consistent state. Consistency checks are applied to the updated target vectors to ensure that there are no conflicts between targets and that the temporal information is coherent, while maintaining the independence between the states of multiple targets. Each target vector after global association and state update processing is integrated and output to form a target state vector.

[0106] S5.3. Combining the target state vector with dynamic scene data, perform future motion prediction and interaction constraint calculation to generate a predicted position sequence. Then, through formatted encapsulation and continuous time interpolation, generate target predicted trajectory data.

[0107] Furthermore, the state vector of each target is fused with the dynamic scene data of the corresponding time step to extract spatial constraints, environmental boundaries, information on neighboring targets and obstacles. Based on the target's current position, velocity, category, and dynamic scene features, future motion prediction is performed, the expected position of the target in subsequent time steps is calculated, and the interaction constraints with other targets, such as collision avoidance, velocity coordination, and direction consistency, are considered to ensure that the predicted trajectory is reasonable. The predicted position sequence is formatted and encapsulated in chronological order, and the predicted position, velocity, category, and timestamp of each time step are uniformly organized into a standardized structure. Continuous time interpolation is performed on the predicted position sequence to fill in the prediction points within the time interval, generating continuous, smooth, and structured target prediction trajectory data.

[0108] S6. Perform environmental perception analysis and decision derivation on the target predicted trajectory data, and output the environmental perception decision results.

[0109] S6.1. The target predicted trajectory data is encoded with neighborhood features and coupled with spatial attributes to generate an interactive feature vector set. The environmental perception risk confidence score is generated through global nonlinear scoring calculation and normalization mapping.

[0110] Furthermore, neighborhood features are extracted from the predicted position sequence of each target in the target predicted trajectory data, including the relative position, velocity difference, category relationship, and dynamic scene environment information of adjacent targets. These features are encoded into a unified vector form, and the neighborhood feature vectors are coupled with the spatial attribute features of the target itself to form an interaction feature vector set, which reflects the comprehensive interaction state between the target and the surrounding environment and other targets. A global nonlinear score is calculated on the interaction feature vector set, and each interaction vector is mapped to a risk value through a nonlinear function, considering factors such as the possibility of conflict, the influence of environmental constraints, and the target's movement trend. The global nonlinear score results are normalized and mapped to express the risk confidence within a unified range, and the environmental perception risk confidence is output.

[0111] S6.2 Perform multi-dimensional weight decoupling and feature reconstruction operations on the environmental perception risk confidence level to generate a single-objective environmental decision vector.

[0112] Furthermore, the environmental perception risk confidence is decomposed by independently assessing each risk influencing factor. This includes location risk assessment for risks caused by changes in target location, velocity risk assessment for risks caused by changes in target velocity and acceleration, category-related risk assessment for risks caused by target category attributes and inter-category interactions, and neighborhood interaction risk assessment for risks caused by interactions between the target and other elements in the neighborhood. This extracts the independent contribution values ​​of each dimension. The contribution values ​​of each dimension are then decoupled. By eliminating redundant dependencies and cross-influences, the risk information of each dimension remains independent and comparable. Based on the decoupled multidimensional risk features, feature reconstruction operations are performed. The risks of each dimension are integrated into a single structured vector through nonlinear mapping, normalization, and vector combination operations. This vector reflects the comprehensive environmental risk status of a single target and outputs a structured and analyzable single-target environmental decision vector.

[0113] S6.3 Integrate the single-objective environmental decision vectors by global sorting and spatial grouping to generate environmental perception decision results.

[0114] Furthermore, all single-target environmental decision vectors are globally sorted according to comprehensive risk scores or decision priorities, forming a sequence from high risk to low risk. This clarifies the importance order of each target in environmental perception. Based on the spatial location and proximity of targets in the dynamic scene, the sorted single-target environmental decision vectors are divided into multiple spatial groups. Each spatial group contains a set of targets that significantly influence each other. Within each spatial group, the single-target environmental decision vectors are integrated and calculated, including vector summation, mean calculation, and normalization. The individual decision features within the group are fused into a group-level comprehensive decision representation. Finally, the comprehensive decision representations of all spatial groups are integrated according to the global sorting results to form a structured and analyzable environmental perception decision result.

[0115] This embodiment also provides an environment perception system based on multi-source data fusion, including:

[0116] The data preprocessing module is used to collect multimodal sensing data, perform time synchronization, noise reduction, correction, and missing value imputation to obtain synchronized and corrected data;

[0117] The graph modeling update module is used to perform node mapping and multi-level graph convolution iterative update on the synchronous correction data using graph neural networks to generate dynamic scene data.

[0118] The feature fusion module is used to combine dynamic scene data with multimodal perception data through feature encoding, interactive coupling and dimensional optimization to generate fused feature data;

[0119] The target recognition module is used to detect and classify targets from fused feature data and generate target status information.

[0120] The target prediction module is used to combine target state information with dynamic scene data to generate target prediction trajectory data through target tracking and future motion prediction.

[0121] The decision analysis module is used to perform environmental perception analysis and decision derivation on the target predicted trajectory data, and output the environmental perception decision results.

[0122] In summary, this invention achieves accurate spatiotemporal feature representation of complex dynamic scenes by: modeling synchronously corrected multimodal perception data using graph neural networks, constructing a multi-level node interaction structure and iteratively updating it; and then performing deep feature encoding, interactive coupling, and dimensional optimization on dynamic scene data and multimodal information to generate highly discriminative fusion features, thereby providing reliable input for target detection, tracking, prediction, and environmental decision-making. Overall, it improves the accuracy, continuity, and robustness of environmental perception and enhances the reliability and adaptability of decision-making in complex dynamic scenes.

[0123] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An environmental perception method based on multi-source data fusion, characterized in that: include, Collect multimodal sensing data, perform time synchronization, denoising, correction, and missing value imputation to obtain synchronized and corrected data; Graph neural networks are used to perform node mapping and multi-level graph convolution iterative updates on the synchronization correction data to generate dynamic scene data. The specific steps are as follows. Environmental element features are extracted from synchronous correction data, and node mapping is performed through multi-dimensional feature association and spatial semantic projection to generate an initial node set. The similarity index between nodes in the initial node set is calculated. Based on the criteria of spatial proximity, temporal adjacency and semantic consistency, edge connections between nodes are established and edge weights are generated by fusion. At the same time, the node relationships are hierarchically organized by combining temporal coupling analysis and temporal stability evaluation to construct a multi-level node interaction structure. Based on a multi-level node interaction structure, spatial feature representation is obtained by aggregating the attribute vectors of neighboring nodes in the spatial topology dimension through graph convolution. In the time dimension, the time-varying patterns of node attributes are captured along the time axis to obtain the temporal feature representation. By using an attention mechanism to jointly model the spatial relationships, temporal dependencies, and semantic similarities between nodes in the joint modeling of spatial and temporal feature representations, the spatial relationships, temporal dependencies, and semantic similarities are associated with scoring and selectively updating node representations, thereby generating dynamic feature representations; Multi-level graph convolution iterative updates are performed on dynamic feature representations to generate dynamic scene data; By combining dynamic scene data with multimodal perception data, and through feature encoding, interaction coupling, and dimensionality optimization, fused feature data is generated. The specific steps are as follows: Spatial and temporal attributes are extracted from dynamic scene data, and deep feature encoding and feature fusion are performed in combination with multimodal perception data to generate an initial feature vector set. The initial feature vector set is divided into batches according to node index and time series order. Within each initial feature vector, spatial attribute features, temporal attribute features, and multimodal features are interactively mapped and nonlinear interactively calculated using a nonlinear activation function. Then, deep feature coupling is performed through multi-layer linear transformation across feature dimensions and residual connections. Finally, a fused feature representation is generated through normalization and anomaly suppression. The fused feature representation is optimized in terms of feature dimension and globally normalized to generate fused feature data. Target detection and classification are performed on the fused feature data to generate target state information; By combining target state information with dynamic scene data, target tracking and future motion prediction are used to generate target predicted trajectory data. Perform environmental perception analysis and decision derivation on the target predicted trajectory data, and output the environmental perception decision results.

2. The environmental perception method based on multi-source data fusion as described in claim 1, characterized in that: The process of collecting multimodal sensing data, performing time synchronization, noise reduction, correction, and missing value imputation to obtain synchronized and corrected data is as follows. Multimodal sensing data is acquired in real time and triggered at a unified time to form initial sensor data. Then, timestamp alignment and time interpolation correction are performed to obtain time-synchronized data. The time synchronization data is noise-suppressed to generate denoised data. Missing regions in the denoised data are smoothed and filled in to obtain synchronized correction data.

3. The environmental perception method based on multi-source data fusion as described in claim 1, characterized in that: The process of performing multi-level graph convolution iterative updates on the dynamic feature representation to generate dynamic scene data involves the following specific steps. The dynamic feature representation is mapped to graph nodes and the edge relationships between nodes are established to form an initial graph structure. The node feature vectors are generated by iterative updating through multi-level graph convolution. The node feature vectors are integrated and normalized according to topological order to generate dynamic scene data.

4. The environmental perception method based on multi-source data fusion as described in claim 1, characterized in that: The steps for optimizing the feature dimension and performing global normalization on the fused feature representation to generate fused feature data are as follows: Feature importance analysis and weight calculation are performed on the fused feature representation to obtain the optimized fused feature vector; The fusion feature optimization vector is globally normalized according to the feature dimension. Based on the node index, time series and feature category, the fusion feature optimization vector is structurally integrated by concatenating the node order, connecting the time series and uniform arrangement to form fusion feature data.

5. The environmental perception method based on multi-source data fusion as described in claim 4, characterized in that: The specific steps for performing target detection and classification on the fused feature data to generate target state information are as follows: Spatial rasterization decomposition and multidimensional feature mapping are performed on the fused feature data to form a candidate element set, and global interactive correlation calculation and nonlinear confidence mapping are performed to form a scoring vector; Spatial overlap analysis and category association suppression are performed on the scoring vectors to obtain the target detection result set; The target detection result set is processed for velocity and trajectory calculation, timestamp annotation, and attribute integration to generate target status information.

6. The environmental perception method based on multi-source data fusion as described in claim 5, characterized in that: The process of combining target state information with dynamic scene data to generate target predicted trajectory data through target tracking and future motion prediction involves the following specific steps. The position, velocity, category, bounding box size, and timestamp of each target are extracted from the target state information, and then structured and vectorized to form the target initial vector. The target initial vector is subjected to multi-target global correlation and state update to obtain the target state vector; By combining the target state vector and dynamic scene data, future motion prediction and interaction constraint calculation are performed to generate a predicted position sequence. Then, through formatted encapsulation and continuous time interpolation, the target predicted trajectory data is generated.

7. The environmental perception method based on multi-source data fusion as described in claim 6, characterized in that: The specific steps for performing environmental perception analysis and decision derivation on the target predicted trajectory data, and outputting the environmental perception decision results, are as follows. Neighborhood feature encoding and spatial attribute coupling are performed on the target predicted trajectory data to generate an interactive feature vector set. Then, environmental perception risk confidence is generated through global nonlinear scoring calculation and normalization mapping. Multi-dimensional weight decoupling and feature reconstruction operations are performed on the environmental perception risk confidence to generate a single-objective environmental decision vector. The single-objective environmental decision vectors are integrated by global sorting and spatial grouping to generate environmental perception decision results.

8. An environmental perception system based on multi-source data fusion, based on the environmental perception method based on multi-source data fusion as described in any one of claims 1 to 7, characterized in that: include, The data preprocessing module is used to collect multimodal sensing data, perform time synchronization, noise reduction, correction, and missing value imputation to obtain synchronized and corrected data; The graph modeling and update module utilizes graph neural networks to perform node mapping and multi-level graph convolution iterative updates on the synchronized correction data, generating dynamic scene data. The specific steps are as follows. Environmental element features are extracted from synchronous correction data, and node mapping is performed through multi-dimensional feature association and spatial semantic projection to generate an initial node set. The similarity index between nodes in the initial node set is calculated. Based on the criteria of spatial proximity, temporal adjacency and semantic consistency, edge connections between nodes are established and edge weights are generated by fusion. At the same time, the node relationships are hierarchically organized by combining temporal coupling analysis and temporal stability evaluation to construct a multi-level node interaction structure. Based on a multi-level node interaction structure, spatial feature representation is obtained by aggregating the attribute vectors of neighboring nodes in the spatial topology dimension through graph convolution. In the time dimension, the time-varying patterns of node attributes are captured along the time axis to obtain the temporal feature representation. By using an attention mechanism to jointly model the spatial relationships, temporal dependencies, and semantic similarities between nodes in the joint modeling of spatial and temporal feature representations, the spatial relationships, temporal dependencies, and semantic similarities are associated with scoring and selectively updating node representations, thereby generating dynamic feature representations; Multi-level graph convolution iterative updates are performed on dynamic feature representations to generate dynamic scene data; The feature fusion module is used to combine dynamic scene data with multimodal perception data through feature encoding, interaction coupling, and dimensionality optimization to generate fused feature data. The specific steps are as follows. Spatial and temporal attributes are extracted from dynamic scene data, and deep feature encoding and feature fusion are performed in combination with multimodal perception data to generate an initial feature vector set. The initial feature vector set is divided into batches according to node index and time series order. Within each initial feature vector, spatial attribute features, temporal attribute features, and multimodal features are interactively mapped and nonlinear interactively calculated using a nonlinear activation function. Then, deep feature coupling is performed through multi-layer linear transformation across feature dimensions and residual connections. Finally, a fused feature representation is generated through normalization and anomaly suppression. The fused feature representation is optimized in terms of feature dimension and globally normalized to generate fused feature data. The target recognition module is used to detect and classify targets from fused feature data and generate target status information. The target prediction module is used to combine target state information with dynamic scene data to generate target prediction trajectory data through target tracking and future motion prediction. The decision analysis module is used to perform environmental perception analysis and decision derivation on the target predicted trajectory data, and output the environmental perception decision results.

Images