Driving Scene Recognition and Adaptation System and Method Based on Multi-Sensor Fusion
By using multi-sensor fusion technology, vehicle sensor data is acquired and aligned to generate a credibility and enhanced semantic graph. Sensor parameters are dynamically adjusted, solving the problem of adaptive perception in existing technologies and achieving stable and accurate driving scene recognition and real-time performance improvement of the perception system.
Patent Information
- Application Number
- CN202611216034.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-12
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2046-08-12
AI Technical Summary
Existing multi-sensor fusion driving scene recognition solutions cannot achieve adaptive perception in complex dynamic environments, cannot quantify the degree of sensor perception degradation, and cannot predict the dynamic evolution trend of driving scenarios. This results in the perception system lagging behind scene changes and being unable to adapt to dynamic conditions such as high-speed driving and rapid scene switching.
By acquiring perception data from vehicle-mounted sensors, performing spatiotemporal alignment, determining the sensor's environmental degradation factor, generating credibility, constructing an enhanced semantic graph based on scene features, performing multi-label classification, predicting the scene evolution probability distribution, and dynamically adjusting sensor control parameters, the perception strategy can be adaptively adjusted.
It achieves stable and accurate driving scenario recognition in complex and dynamic environments, improves the adaptability and real-time performance of the perception system, avoids perception lag and abrupt changes in accuracy, and significantly improves the adaptability to dynamic and complex driving scenarios.
Smart Images

Figure CN122732077B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image recognition technology, specifically to a driving scene recognition and adaptation system and method based on multi-sensor fusion. Background Technology
[0002] Driving scene recognition is the core foundation of autonomous driving environmental perception, determining the accuracy of vehicle perception adaptation and driving safety. As autonomous driving gradually iterates towards intelligence and scene adaptation, single-sensor perception solutions are limited by their own hardware and cannot simultaneously ensure perception accuracy, anti-interference ability, and real-time performance under complex road conditions. Multi-sensor fusion perception has become the mainstream development direction in the field of driving scene recognition.
[0003] Current mainstream technologies compensate for the limitations of single-sensor perception by integrating perception data from multiple vehicle sensors, such as LiDAR, millimeter-wave radar, and vision cameras, to achieve full-domain recognition of the road environment, traffic participants, and driving environment. Currently, conventional multi-sensor fusion scene recognition often adopts fixed-weight data fusion and single-frame static scene modeling. After matching the current driving scenario, sensor perception is adjusted based on fixed parameters to meet the requirements of simple, steady-state, and low-interference conventional road scene recognition, and it is widely used in assisted driving perception systems.
[0004] Existing multi-sensor fusion driving scene recognition and adaptation solutions have certain technical shortcomings in real driving conditions with complex dynamics, high interference and rapid scene evolution. They cannot adapt to the adaptive perception requirements of dynamic driving scenarios. Traditional fusion methods generally use fixed weights to complete the fusion of multi-sensor data, which cannot quantify the degree of real-time perception degradation and reliability of each sensor under different conditions, and cannot truly represent the actual state of complex driving environments.
[0005] Current mainstream solutions only build static scene semantic models based on single-frame perception data, which can only achieve static recognition of instantaneous driving scenes and cannot generate enhanced semantic representations with temporal continuity. At the same time, traditional technologies can only complete static classification of scenes and cannot quantify and predict the dynamic evolution trend of driving scenes. They lack the ability to predict scenes in advance, which causes the perception system to always lag behind scene changes and make it difficult to adapt to dynamic conditions such as high-speed driving, close-range interaction and rapid scene switching.
[0006] Traditional solutions match scenarios based on fixed strategy templates, which can only adapt to preset steady-state routine scenarios. When the driving scenario is about to change or the dynamic interaction of traffic intensifies, they cannot identify the scenario evolution trend in advance and actively switch the perception strategy. The sensor control parameters remain in a fixed state for a long time and cannot adapt to the dynamic changes of the scenario. The real-time and dynamic performance of perception adaptation is insufficient.
[0007] In light of the shortcomings of existing technologies, the technical problem to be solved in this application is how to achieve dynamic control of driving scene recognition and sensor perception adaptation in complex dynamic environments. Summary of the Invention
[0008] To address the shortcomings of existing technologies, the purpose of this application is to provide a driving scene recognition adaptation system and method based on multi-sensor fusion, so as to ensure the stability, accuracy and robustness of driving scene recognition under all working conditions and long time series, and realize the adaptive intelligent upgrade of multi-sensor fusion driving scene recognition.
[0009] To achieve the above objectives, this application adopts the following technical solution:
[0010] In the first aspect, this application provides a driving scene recognition and adaptation method based on multi-sensor fusion, which obtains perception data of the corresponding sensors on the vehicle, performs spatiotemporal alignment on the perception data, determines the environmental degradation factor of the sensors respectively, generates the credibility of the corresponding sensors, extracts perception features by weighting the corresponding perception data respectively, and fuses the perception features to generate scene features.
[0011] A scene semantic graph is constructed based on scene features. The scene semantic graphs of the current frame and historical frames are temporally correlated to generate an enhanced semantic graph. The driving scene is classified into multiple labels based on the enhanced semantic graph, and the scene evolution probability distribution is predicted based on the temporal evolution features of the nodes in the enhanced semantic graph.
[0012] Based on the multi-label classification results and scenario evolution probability distribution of driving scenarios, a perception strategy is configured for the current driving scenario. When the scenario evolution probability is greater than a preset threshold, the perception strategy is switched to dynamically adjust the control parameters of the sensors.
[0013] Monitor the adjusted perception performance indicators, and based on the residual between the perception performance indicators and the preset baseline indicators, correct the control parameters of the current perception strategy.
[0014] Furthermore, the fused generated scene features include:
[0015] The global degradation factor and local degradation factor of each sensor are calculated separately, and the global degradation factor and local degradation factor are cross-coupled to obtain the environmental degradation factor of the corresponding sensor.
[0016] Based on environmental degradation factors, and combined with the channel fixation accuracy of the sensor and the interference level of the sensing space partition, a two-dimensional matrix with channels as rows and sensing space partitions as columns is generated as the reliability of the corresponding sensor.
[0017] The perception data of each sensor is weighted based on credibility, and perception features of different granularities are extracted according to the perception domain to which the sensor belongs.
[0018] Perceptual features within the same perceptual domain are aggregated to generate scene features.
[0019] Furthermore, the global degradation factor characterizes the degree of attenuation of the overall sensing capability of the sensor by the global environment, and the local degradation factor characterizes the degree of attenuation of the local sensing capability of the sensor by different sensing spatial partitions.
[0020] The two-dimensional matrix uses the fixed accuracy of the sensor channel as the base weight, and combines the environmental degradation factor and the interference level of the corresponding sensing space partition to perform row-by-row and column-by-column joint operation on the matrix elements to obtain the initial matrix, and sets a threshold lower limit constraint on the initial matrix elements.
[0021] Furthermore, the generation of the enhanced semantic graph includes:
[0022] A scene semantic graph is constructed based on scene features. The scene semantic graph uses road elements, traffic participants and environmental attributes as nodes, and the spatial positional relationship and dynamic interaction relationship between nodes as associated edges.
[0023] Extract the scene semantic graph of historical frames within the sliding window, perform node type matching between the scene semantic graph of the current frame and the historical frames, and configure temporal weights for different nodes to fuse the node features of the current frame and the historical frames.
[0024] Based on the degree of state change of adjacent frame nodes within the sliding window, the interaction correlation coefficient of the corresponding associated edge is dynamically updated to realize the temporal correlation between node features and associated edges. The credibility is then converted into a correction coefficient to perform weighted correction on the node features, generating an enhanced semantic graph.
[0025] Furthermore, the predicted scenario evolution probability distribution includes:
[0026] Temporal sampling of the continuous frame feature change trajectory corresponding to different nodes in the enhanced semantic graph is performed based on the sliding window, and the temporal evolution features of the corresponding nodes are extracted respectively.
[0027] The temporal evolution features are weighted according to the preset scene influence weights to calculate the scene evolution probability of a single node.
[0028] By combining the interaction correlation coefficients of the edges between nodes, the interaction deviation of the scene evolution probability of a single node is corrected, and the scene evolution probability distribution of the driving scenario is obtained by hierarchical fusion and mapping according to node type.
[0029] Furthermore, the degree of state change of adjacent frame nodes is the difference in node features of the same node between two adjacent frames; the interaction correlation coefficient is a quantitative value of the spatial position relationship and dynamic interaction relationship corresponding to the associated edge, used to characterize the spatial binding strength and interaction tightness between nodes, and is synchronously positively adjusted with the degree of state change of adjacent frame nodes.
[0030] The continuous frame feature change trajectory is a temporal sequence formed by sequentially concatenating the node features of the same node frame by frame within a sliding window; the temporal evolution feature is a trend feature extracted after sampling the continuous frame feature change trajectory; the scene influence weight is determined based on the dominant role of the node in the driving scene.
[0031] Furthermore, the control parameters for dynamically adjusting the sensor include:
[0032] When the probability of scene evolution exceeds a preset threshold, the perception strategy is switched, and the parameter adjustment level is divided according to the numerical range of the probability of scene evolution to determine the parameter adjustment benchmark of the sensor.
[0033] The parameter adjustment range is allocated according to the interference level of the sensing space partition, and the control parameters of the sensor are adjusted in conjunction with the parameter adjustment benchmark and the parameter adjustment range.
[0034] Furthermore, the control parameters of the sensor include timing acquisition parameters, sensing gain parameters, and detection field of view parameters; the linkage adjustment specifically involves: adjusting the timing acquisition parameters of the sensor based on the parameter adjustment benchmark, adjusting the sensing gain parameters according to the sensing spatial partitions based on the parameter adjustment range, and dynamically adjusting the detection field of view parameters of the sensor according to the spatial distribution range of nodes in the enhanced semantic map.
[0035] Furthermore, the modification of the control parameters of the current perception strategy includes:
[0036] By comparing the perception performance indicators with the preset baseline indicators and combining the global perception state with the local perception state of different perception spatial partitions, the global residual and local residual are determined in layers.
[0037] Based on the current driving scenario's evolution, correction weights are assigned to the global residual and the local residual respectively, and after weighted processing, the global weighted residual and the local weighted residual are obtained respectively.
[0038] The parameters are obtained by correcting the time series based on global weighted residuals, the sensing gain parameters are corrected by per-sensory spatial partitioning based on local weighted residuals, and the detection field parameters are corrected by combining the distribution state of nodes in the enhanced semantic graph.
[0039] Continuously monitor the changing trend of residuals under multiple frames of data, gradually reduce the correction step size, and lock the control parameters of the current sensing strategy when the residuals stabilize within the allowable range.
[0040] Secondly, this application provides a driving scene recognition and adaptation system based on multi-sensor fusion, comprising a fusion module, an association module, an adjustment module, and a correction module;
[0041] The fusion module is used to acquire perception data from the corresponding sensors on the vehicle, perform spatiotemporal alignment on the perception data, estimate the environmental degradation factor of the sensors respectively, generate the credibility of the corresponding sensors, extract perception features by weighting the corresponding perception data respectively, and fuse the perception features to generate scene features.
[0042] The association module is used to construct a scene semantic graph based on scene features, temporally associate the scene semantic graphs of the current frame with those of historical frames to generate an enhanced semantic graph, perform multi-label classification of driving scenes based on the enhanced semantic graph, and predict the scene evolution probability distribution based on the temporal evolution features of nodes in the enhanced semantic graph.
[0043] The adjustment module is used to configure the perception strategy under the current driving scenario based on the multi-label classification results and scenario evolution probability distribution of the driving scenario. When the scenario evolution probability is greater than the preset threshold, the perception strategy is switched to dynamically adjust the control parameters of the sensor.
[0044] The correction module is used to monitor the adjusted perception performance indicators and correct the control parameters of the current perception strategy based on the residual between the perception performance indicators and the preset baseline indicators.
[0045] Compared with the prior art, the beneficial effects achieved by this application are as follows:
[0046] By quantifying the environmental degradation factors of sensors and generating credibility, weighted fusion of multi-sensor perception data is achieved. After extracting perception features, they are fused to generate scene features, improving the authenticity and accuracy of scene features from the data source. This effectively solves the problems of mixed scene features and decreased recognition accuracy under complex interference environments, and provides high-quality basic feature support for subsequent scene semantic modeling.
[0047] An enhanced semantic graph is generated by temporally associating the scene semantic graphs of the current frame and historical frames. Based on the temporal evolution characteristics of nodes, the probability prediction of scene evolution is realized, combining static scene recognition with dynamic trend prediction. This accurately identifies the real-time driving scene state and predicts the future evolution trend of the scene, greatly improving the comprehensiveness and foresight of driving scene cognition.
[0048] Based on scene classification results and evolution probability distribution, the perception strategy is dynamically configured and threshold-triggered switching is realized. Under critical conditions of scene changes and intensified traffic interaction, the perception parameters are pre-adjusted, effectively avoiding perception lag, target omission and accuracy change during scene switching, significantly improving the perception system's adaptability and real-time performance to dynamic and complex driving scenarios; and the control parameters of the perception strategy are corrected in a closed loop through residual comparison. Attached Figure Description
[0049] Figure 1 This is a flowchart of a driving scene recognition and adaptation method based on multi-sensor fusion.
[0050] Figure 2 A flowchart for generating the enhanced semantic graph;
[0051] Figure 3 This is a schematic diagram of a driving scene recognition and adaptation system based on multi-sensor fusion. Detailed Implementation
[0052] The technical solution of this application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments and specific features in the embodiments are detailed descriptions of the technical solution of this application, rather than limitations thereof.
[0053] Example 1:
[0054] like Figure 1 As shown, this embodiment provides a driving scene recognition and adaptation method based on multi-sensor fusion, including:
[0055] S1. Obtain perception data from the corresponding sensors mounted on the vehicle, perform spatiotemporal alignment on the perception data, determine the environmental degradation factor of each sensor, generate the credibility of the corresponding sensor, extract perception features by weighting the corresponding perception data, and fuse the perception features to generate scene features.
[0056] Specifically, the features of the fused generated scene include:
[0057] The system acquires perception data from each sensor on the vehicle and performs spatiotemporal alignment. It then calculates the global and local degradation factors for each sensor and performs cross-coupling operations on the global and local degradation factors to obtain the environmental degradation factor for the corresponding sensor.
[0058] Based on environmental degradation factors, and combined with the channel fixation accuracy of the sensor and the interference level of the sensing space partition, a two-dimensional matrix with channels as rows and sensing space partitions as columns is generated as the reliability of the corresponding sensor.
[0059] The perception data of each sensor is weighted based on credibility, and perception features of different granularities are extracted according to the perception domain to which the sensor belongs.
[0060] Perceptual features within the same perceptual domain are aggregated to generate scene features.
[0061] Among them, the global degradation factor characterizes the degree of attenuation of the overall sensing ability of the sensor by the global environment, while the local degradation factor characterizes the degree of attenuation of the local sensing ability of the sensor by different sensing spatial partitions.
[0062] The two-dimensional matrix uses the fixed accuracy of the sensor channel as the basic weight, and combines the environmental degradation factor and the interference degree of the corresponding sensing space partition. The matrix elements are subjected to row-by-row and column-by-column joint operation to obtain the initial matrix, and a threshold lower limit constraint is set on the elements of the initial matrix.
[0063] The perception domain is divided according to the sensor function, and different perception domains correspond to fine-grained, medium-grained and coarse-grained perception features respectively. The aggregation is based on the fine-grained perception features, verifies the spatial position residuals of medium-grained and coarse-grained perception features, removes abnormal perception features with spatial position residuals greater than the residual threshold, and then stitches and fuses the remaining perception features one channel at a time to generate scene features.
[0064] Vehicles are equipped with multiple types of sensors to acquire road environment data. Different sensors have inherent biases in sampling timing and spatial coordinate systems, making the raw data unusable for feature extraction. Furthermore, the driving environment is divided into a global, large-scale environment and a localized, segmented environment. These two types of environments cause sensor perception capabilities to degrade from different dimensions, and a single degradation evaluation index cannot objectively reflect the real-time working status of the sensors. This solution first completes the spatiotemporal alignment of multi-sensor perception data, calculates the global degradation factor and the local degradation factor separately, and obtains the environmental degradation factor through cross-coupling calculations, providing basic parameters for subsequent reliable quantification.
[0065] Specifically, the raw perception data output from sensors such as vehicle-mounted LiDAR, millimeter-wave radar, and vision cameras are acquired, and spatiotemporal alignment is performed. Time alignment uses the unified clock of the vehicle's main control system as a reference, reads the timestamps carried in the data frames of each sensor, calculates the timing offset according to the nominal sampling frequency of each device, and calibrates the time dimension of all data frames to eliminate cross-device timing differences. Spatial alignment establishes a unified coordinate system for the entire vehicle with the vehicle's geometric center as the origin. Based on the installation position and angle of each sensor on the vehicle body, the perception coordinates, pixel coordinates, and point cloud coordinates in the sensor's own coordinate system are transformed one by one to the unified coordinate system of the entire vehicle, realizing the normalization of the spatial dimensions of all perception data.
[0066] For example, the vehicle is equipped with a forward-facing vision camera, a forward-facing LiDAR, and millimeter-wave radars on the left and right sides. The vision camera has a sampling frequency of 30Hz, and the LiDAR has a sampling frequency of 10Hz. There is a difference in the original timing of the two types of devices. After correction by a unified clock, it is ensured that each set of fused data corresponds to the driving scene at the same time. The radar detection point cloud and camera image pixels are all converted to the vehicle's central coordinate system to control the accuracy of coordinate conversion.
[0067] The global degradation factor is used to characterize the degree of attenuation of the sensor's overall sensing capability by the global environment. The global environment includes environmental elements covering the entire sensing range, such as meteorological conditions, lighting conditions, and overall occlusion. Specifically, a preset value range is matched according to the real-time identified global environment type. The global environment types are divided into five categories: sunny daytime, strong light and backlight, rainy night, foggy day, and nighttime. The corresponding global degradation factor value ranges are set to 0.05 to 0.15, 0.25 to 0.35, 0.4 to 0.5, 0.55 to 0.65, and 0.3 to 0.4, respectively.
[0068] The local degradation factor is used to characterize the degree of attenuation of the sensor's local sensing capability by the sensing space partition. The sensing space partition is divided into regions according to the sensor's detection distance and field of view. Taking lidar as an example, the sensing space partition is divided into near zone, middle zone and far zone according to the detection distance. The local interference elements in different partitions are different. The near zone is prone to vehicle body obstruction and road debris interference. The middle zone is prone to dynamic interference from the vehicle in front and non-motorized vehicles. The far zone is prone to atmospheric scattering and boundary blurring. The local degradation factor is calculated for each sensing space partition. The local degradation factor is a quantitative evaluation based on the measured interference elements in each partition. That is, for each sensing space partition, the interference elements that affect the sensor's sensing in that partition are obtained, including the proportion of obstructions, clutter reflection intensity and atmospheric attenuation coefficient, etc. After normalizing each interference element index to [0,1], the weighted sum is obtained to obtain the local degradation factor of that partition. The weight of each interference element is determined by the sensor type and partition characteristics. The local degradation factor takes a value from 0 to 1. The larger the value, the higher the degree of attenuation of local sensing.
[0069] The global degradation factor and the local degradation factor are calculated using a weighted cross-coupling method, i.e., D e =a•D g +b•D l D e D represents the environmental degradation factor corresponding to a single perceptual space partition. g D represents the global degradation factor. l The coefficients represent the local degradation factor of the corresponding sensing space partition. a and b are preset coupling weight coefficients, and the coefficient values satisfy a+b=1. Under normal operating conditions, a=0.4 and b=0.6.
[0070] Taking a foggy driving scenario as an example, the overall environment type is determined to be foggy, and the global degradation factor D is... g The value is 0.6, representing the local degradation factor D in the near-field region of the lidar. l Taking a value of 0.3, we can calculate the environmental degradation factor D for this partition. e =0.4×0.6+0.6×0.3=0.42.
[0071] The environmental degradation factor obtained from the calculation is the core input parameter for constructing the sensor credibility matrix. Its value determines the basic calculation result of the matrix and provides a quantitative basis for the next stage of credibility quantification.
[0072] Different data channels of the sensor have inherent precision fixed at the factory. Combined with the environmental degradation factor and the interference level of each sensing space partition obtained above, a single-dimensional parameter cannot distinguish the sensing reliability under the combination of channels and partitions. Therefore, a two-dimensional matrix is constructed to integrate multi-dimensional parameters to realize the structured and refined quantification of sensing reliability, which is used as the judgment standard for weighted processing of sensing data.
[0073] The row dimension of the two-dimensional matrix corresponds to the sensor data channels, and the column dimension corresponds to the pre-divided sensing space partitions. Each element in the matrix corresponds to a combination of channels and sensing space partitions. The channel fixed accuracy is an inherent parameter calibrated by the sensor at the factory and is not affected by the external environment. As the basic weight parameter for matrix operations, the channel fixed accuracy varies for different types of sensors. The single-channel fixed accuracy of 16-line lidar ranges from 0.8 to 0.95, and the single-channel fixed accuracy of millimeter-wave radar ranges from 0.75 to 0.9. The interference level of the sensing space partition is the quantification result of clutter, obstruction, and reflection interference in each partition, and its value range is consistent with the local degradation factor.
[0074] Matrix elements are operated on row-by-row and column-by-column. The formula for single matrix element operation is defined as follows: , where R i,j P represents the matrix element corresponding to the i-th channel and the j-th perceptual space partition. i D represents the channel fixed precision of the i-th channel. e(j) I represents the environmental degradation factor of the j-th perceptual space partition (the degradation factor value is shared by all channels within the same partition). j This represents the interference level of the j-th sensing space partition; and a threshold lower limit constraint is set on the initial matrix obtained by the calculation. The threshold lower limit is uniformly set to 0.2. All elements in the initial matrix with values less than 0.2 are forcibly corrected to 0.2. The matrix after completing the lower limit constraint is the confidence matrix of the sensor.
[0075] Taking a 16-line lidar as an example, this sensor has 16 data channels, and the detection range is divided into 3 sensing space partitions, ultimately generating a 16x3 two-dimensional matrix. The calculation is performed using the 3rd channel and the 2nd partition. This channel has a fixed accuracy P3=0.92, corresponding to the partition's environmental degradation factor D. e(3,2) =0.42, partition interference level I2=0.35, substituting, we get If the value is higher than the lower threshold of 0.2, the original calculation result is retained; if the calculation result of a certain element is 0.16, it is corrected to 0.2 according to the rules.
[0076] This confidence matrix serves as the basis for weighted processing of the perceived data. The values of each element in the two-dimensional matrix will act as weighting coefficients on the perceived data at the corresponding positions, affecting the quality of subsequent extraction of perceived features.
[0077] The reliability of sensing data from different channels and different sensing spatial partitions varies significantly. Extracting features without differentiation will retain a large amount of low-quality data, reducing the overall effectiveness of the features. Sensors are divided into different sensing domains according to their functions, and different sensing domains are adapted to different granularity feature extraction needs. Therefore, it is necessary to perform data weighting based on reliability, and then extract multi-granularity sensing features by sensing domain to achieve hierarchical acquisition of feature information.
[0078] The generated confidence level is retrieved, and the original sensing data is weighted segment by segment according to the correspondence between channels and sensing space partitions. The weighting formula is Data. w =Data o •R i,j Data o For raw sensory data, Data w For the weighted perception data, R i,j These are the elements of the credibility matrix at the corresponding locations; the higher the credibility, the higher the retention of the weighted perception data, thereby strengthening high-reliability data and weakening low-reliability data; the perception domain is divided according to the functional attributes of the sensors, with the visual camera as the image perception domain, the lidar as the point cloud perception domain, the millimeter-wave radar as the ranging and velocity measurement perception domain, and the multi-sensor fusion unit setting the global perception domain.
[0079] Different perception domains correspond to fixed feature granularities. The image perception domain is responsible for extracting fine-grained features, covering details such as lane lines, traffic signs, and pedestrian outlines. The point cloud perception domain is responsible for extracting medium-grained features, covering target information such as vehicles, non-motorized vehicles, and large obstacles. The global perception domain is responsible for extracting coarse-grained features, covering global information such as road direction, traffic flow distribution, and large-scale environmental conditions. Based on the weighted perception data, each perception domain outputs the corresponding granularity features according to a preset feature extraction algorithm. When extracting fine-grained features, the visual camera image perception domain uses the weighted image pixel data as a basis to extract edge, texture, and small target outline features. When extracting medium-grained features, the lidar point cloud perception domain uses the weighted point cloud data to perform clustering and target outline fitting operations.
[0080] The preset feature extraction algorithms are pre-selected and fixed configurations based on the functional attributes and feature granularity of the perception domain. The image perception domain adopts a target detection algorithm based on convolutional neural networks, taking weighted image pixel data as input and outputting fine-grained feature vectors such as edge contours, texture descriptors, and small target bounding boxes. The point cloud perception domain adopts a point cloud processing algorithm based on voxel clustering and contour fitting, taking weighted 3D point cloud data as input and outputting medium-grained feature vectors such as target clusters, 3D bounding boxes, and point cloud density distribution. The global perception domain adopts an environmental analysis algorithm based on regional statistics and situation estimation, taking fused multi-source perception data as input and outputting coarse-grained feature vectors such as road direction vectors, traffic flow density fields, and large-scale environmental situation scores. The ranging and speed measurement perception domain adopts a radar signal processing algorithm based on target tracking and Doppler frequency shift analysis, taking weighted radar echo data as input and outputting ranging and speed measurement feature vectors such as target distance, radial velocity, and target-related trajectory.
[0081] The multi-granularity and sub-perceptual domain perceptual features output in this stage are the processing objects of the subsequent feature aggregation stage. The integrity and hierarchical attributes of the features determine the final effect of feature aggregation.
[0082] Within the same perceptual domain, there are three types of features: fine-grained, medium-grained, and coarse-grained. Due to perceptual bias, some features may exhibit spatial position shifts. Such abnormal features mixed into the aggregation results can interfere with the accuracy of scene features. Therefore, the fine-grained features with the highest positioning accuracy are used as the benchmark to complete the removal of abnormal features, unification of data dimensions, and feature splicing, ultimately generating scene features.
[0083] The feature aggregation process consists of three execution steps: spatial location residual verification, feature dimension unification, and channel-by-channel stitching and fusion. Using fine-grained perceptual features within the same perceptual domain as the location benchmark, it iterates through all medium-grained and coarse-grained features within the same perceptual domain, calculating the spatial location residual between features. The spatial location residual is the Euclidean distance between the corresponding coordinates of two sets of features, i.e. Where Err is the spatial position residual, (x1, y1) is the fine-grained feature coordinates, and (x2, y2) is the feature coordinates to be verified. A preset residual threshold is set to 15 pixels in the pixel coordinate system and 0.5 meters in the point cloud coordinate system. When the spatial position residual is greater than the preset residual threshold, the feature is determined to be an abnormal feature and is directly removed, and only valid features that meet the position matching standard are retained. Taking the image perception domain as an example, based on the fine-grained lane line feature coordinates, if the position residual of the medium-grained vehicle feature reaches 20 pixels, exceeding the residual threshold requirement of 15 pixels, the vehicle feature is removed.
[0084] After removing anomalous features, the data dimensions of the remaining valid features differ. A dimension mapping algorithm is used to unify fine-grained, medium-grained, and coarse-grained features to the same data dimension, ensuring data format compatibility. After dimension unification, all valid features are stitched together channel by channel according to the arrangement order of sensor data channels to complete the feature aggregation of a single sensing domain. After all sensing domain aggregation is completed, the final scene features are obtained by summarizing.
[0085] The scene features output in this step are the input data for constructing the scene semantic graph. The accuracy and completeness of the scene features determine the construction effect of the subsequent scene semantic graph and augmented semantic graph, and also affect the calculation results of multi-label classification of driving scenarios and prediction of scene evolution probability.
[0086] S2. Construct a scene semantic graph based on scene features, temporally associate the scene semantic graphs of the current frame with those of historical frames to generate an enhanced semantic graph, perform multi-label classification of driving scenes based on the enhanced semantic graph, and predict the scene evolution probability distribution based on the temporal evolution features of nodes in the enhanced semantic graph.
[0087] Furthermore, such as Figure 2 As shown, generating an enhanced semantic graph includes:
[0088] A scene semantic graph is constructed based on scene features. The scene semantic graph uses road elements, traffic participants and environmental attributes as nodes, and the spatial positional relationship and dynamic interaction relationship between nodes as associated edges.
[0089] Extract the scene semantic graph of historical frames within the sliding window, perform node type matching between the scene semantic graph of the current frame and the historical frames, and configure temporal weights for different nodes to fuse the node features of the current frame and the historical frames.
[0090] Based on the degree of state change of adjacent frame nodes within the sliding window, the interaction correlation coefficient of the corresponding associated edge is dynamically updated to realize the temporal correlation between node features and associated edges. The credibility is converted into a correction coefficient to perform weighted correction on the node features, generating an enhanced semantic graph. Based on the enhanced semantic graph, multi-label classification of driving scenarios is performed.
[0091] Among them, the degree of state change of adjacent frame nodes is the difference in node features of the same node between two adjacent frames; the interaction correlation coefficient is the quantitative value of the spatial position relationship and dynamic interaction relationship corresponding to the associated edge, which is used to characterize the spatial binding strength and interaction tightness between nodes, and is adjusted positively in sync with the degree of state change of adjacent frame nodes.
[0092] Single-frame scene features can only represent instantaneous static scene information, lacking the inherent temporal continuity and target interaction correlation of driving scenarios. They cannot quantify the dynamic changes of traffic participants and the road environment. Single static features are easily distorted by single-frame perceptual noise and local occlusion interference, and cannot support dynamic scene classification and scene evolution prediction. Therefore, it is necessary to construct a topological semantic structure, integrate multi-frame temporal information, dynamically update the target interaction correlation strength, and combine the reliability of sensors to complete feature correction, generating an enhanced semantic graph with temporal stability and interaction authenticity.
[0093] Specifically, the construction of a basic scene semantic graph is first completed. Road elements, traffic participants, and environmental attributes are used as nodes in the scene semantic graph, and the spatial positional relationships and dynamic interaction relationships between nodes are used as association edges to complete the topological modeling of scene features. Among them, road element nodes include static road structure features such as lane lines, road boundaries, intersection areas, and traffic signs; traffic participant nodes include dynamic target features such as motor vehicles, non-motor vehicles, pedestrians, and static obstacles; and environmental attribute nodes include global environmental features such as light intensity, weather conditions, road surface adhesion status, and global occlusion status. Association edges simultaneously carry two types of quantitative information: spatial positional relationships quantify the relative Euclidean distance, relative deflection angle, and spatial overlap ratio between any two nodes; and dynamic interaction relationships quantify the motion convergence trend, avoidance constraints, and interference association degree between two nodes.
[0094] This topology unifies the data organization of all scene elements, providing a standardized data carrier for subsequent time-series correlation operations.
[0095] After the basic semantic graph is constructed, temporal fusion operation is initiated. A fixed-duration sliding window is used to extract continuous historical semantic data from multiple frames. The sampling interval of the 8-frame sliding window is uniformly used to ensure that the temporal sampling scale is consistent with the aforementioned feature acquisition system. Node type matching is performed on the scene semantic graph of the current frame and the scene semantic graph of the historical frames within the sliding window. Temporal feature fusion is only performed on nodes of the same type to avoid the problem of invalid superposition of cross-type features. Temporal weights are configured for different types of nodes. The temporal weights are used to balance the output ratio of the temporal cumulative information of the historical frames and the real-time information of the current frame. The values are normalized to the range of 0 to 1.
[0096] Among them, traffic participant nodes exhibited the highest frequency of dynamic fluctuations, with temporal weights ranging from 0.7 to 0.8; road element nodes showed stable scene states and minimal inter-frame changes, with temporal weights ranging from 0.3 to 0.4; environmental attribute nodes exhibited smooth global state transitions, with temporal weights ranging from 0.2 to 0.3; the calculation formula for node feature fusion was as follows: In the formula F t The fused node features are represented by n, which is the number of historical frames within the sliding window. In this embodiment, n=8, and w cF is the temporal weight of the current frame. c w represents the original node features of the current frame. h,a Let F be the temporal weight of the historical node in frame a. h,a To correspond to the features of historical frame nodes, all weights satisfy the normalization constraint. For example, for a vehicle traffic participant node, the temporal weight of the current frame is 0.75, and the temporal weights of the four historical frames within the sliding window are averaged at 0.25 for cumulative adaptation, thus completing the weighted fusion of features from multiple frames and preserving the continuous features of the temporal changes of the node.
[0097] After temporal node feature fusion is completed, the degree of state change of adjacent frame nodes is calculated, which is the Euclidean difference between the feature vectors of the same node in two adjacent frames, i.e. In the formula, ΔF represents the degree of state change of a node, m is the dimension of the node's feature vector, and F t,k F is the k-th dimension feature of the current frame node. t-1,k The k-th dimension feature of the node in the previous frame is used; the interaction correlation coefficient of the associated edges is dynamically updated based on the degree of state change. The interaction correlation coefficient takes values from 0 to 1, with an initial baseline value set to 0.5, and is adjusted synchronously with the degree of node state change in a positive direction, i.e., C. new =C base +α×△F, where C new C represents the updated interaction correlation coefficient. base The initial baseline value is α, which is a preset update step size coefficient ranging from [0.1, 1]. This coefficient is determined by calibration using the driving scenario dataset. A larger value indicates that the interaction correlation coefficient is more sensitive to changes in node state. In this embodiment, α = 0.5 is used for calibration. The interaction correlation coefficient C... new The value of is constrained to [0,1], and is truncated to the boundary value when it exceeds the limit.
[0098] For example, as the feature difference between adjacent frames of a vehicle node increases, the node's motion state changes drastically. The interaction correlation coefficient between the vehicle and the lane, and between the vehicle and the obstacle in front, increases positively from the baseline of 0.5 to 0.85, indicating a significant enhancement in the closeness of interaction between nodes. When the node state tends to stabilize, the feature difference converges, and the cross-correlation coefficient falls back to the steady-state range, realizing adaptive dynamic iteration of the interaction correlation strength.
[0099] After completing the dynamic update of the associated edges, the two-dimensional matrix data of the sensor confidence level is retrieved, the confidence level is normalized to the interval of 0 to 1, and converted into node feature correction coefficients. The correction calculation formula is F. correct =F t ×R norm In the formula F correct To correct the node features, R normThis is a normalization correction coefficient for credibility. This operation is used to offset the perceptual feature bias caused by environmental degradation and local partitioning interference, weaken the feature weight of low-credibility perceptual data, strengthen the effective features of high-credibility data, and finally complete the temporal association and error correction of all node features and related edge topology to generate the final enhanced semantic graph. Based on the enhanced semantic graph, multi-label classification of driving scenarios is completed, and standardized scene label results are output.
[0100] This addresses the technical shortcomings of traditional single-frame semantic modeling, which suffers from static fixation, lack of interactive association, and absence of temporal memory. By quantifying inter-frame node changes, it achieves dynamic adaptation of interactive relationships and corrects perceptual biases through credibility mapping. This allows the enhanced semantic graph to align with the dynamic interactive characteristics and perceptual uncertainty features of real driving scenarios, significantly improving the completeness and robustness of scene semantic representation. The output enhanced semantic graph and multi-label classification results serve as effective inputs for subsequent node temporal trajectory sampling and temporal evolution feature extraction. Its topological accuracy, temporal continuity, and interactive realism determine the accuracy of scene evolution probability prediction.
[0101] Specifically, the predicted scenario evolution probability distribution includes:
[0102] Temporal sampling of the continuous frame feature change trajectory corresponding to different nodes in the enhanced semantic graph is performed based on the sliding window, and the temporal evolution features of the corresponding nodes are extracted respectively.
[0103] The temporal evolution features are weighted according to the preset scene influence weights to calculate the scene evolution probability of a single node.
[0104] By combining the interaction correlation coefficients of the edges between nodes, the interaction deviation of the scene evolution probability of a single node is corrected, and the scene evolution probability distribution of the driving scenario is obtained by hierarchical fusion and mapping according to node type.
[0105] Among them, the continuous frame feature change trajectory is a temporal sequence formed by sequentially concatenating the node features of the same node frame by frame within the sliding window; the temporal evolution feature is the trend feature extracted after sampling the continuous frame feature change trajectory; and the scene influence weight is divided according to the dominant role of the node in the driving scene.
[0106] Augmented semantic graphs can only reproduce the scene state and interaction relationships of the current and historical periods, and do not have the ability to predict future scene evolution. Dynamic switching of vehicle perception strategies and advance adaptation of sensor parameters require the future change trend of the scene as a prerequisite. Single node features cannot reflect the evolutionary law of multi-factor linkage. Spatial binding and dynamic interaction between nodes will affect the scene switching trend. Therefore, it is necessary to extract evolution trend features through temporal trajectory sampling, quantify the probability of a single node by combining the dominant weight of the node scene, correct the isolated prediction bias by the interaction correlation coefficient, and finally obtain the scene evolution probability distribution by hierarchical fusion.
[0107] Specifically, temporal sampling is first performed on nodes of various types within the enhanced semantic graph using an 8-frame sliding window. The feature sequences of the same node within the sliding window, corrected frame by frame, are then concatenated to construct a continuous frame feature change trajectory. This trajectory is a one-dimensional temporal sequence recording the short-term state fluctuations and trend shifts of the nodes. The continuous temporal trajectory is then processed using sliding fitting and trend dimensionality reduction sampling to filter out small noise fluctuations in a single frame, retain the long-term stable change patterns of the nodes, and extract low-dimensional, highly representative temporal evolution features, thus achieving a dimensional upgrade from instantaneous node features to trend evolution features. For example, polynomial fitting sampling is performed on the 8-frame continuous feature trajectory of the vehicle traffic participant node to remove small jitter noise between frames, retaining core evolutionary trends such as continuous approach, lateral shift, and uniform steady state of the vehicle, forming temporal evolution features.
[0108] After extracting the temporal evolution features, a weighted operation is performed on the features to quantify the scene evolution probability of a single node. Scene influence weights are stratified based on the dominant role of nodes in the driving scenario. Specifically, based on historical driving scenario datasets, the correlation between the temporal evolution features of each type of node and scene label switching events is statistically analyzed. The correlation coefficient is used as the benchmark for node influence weights. In actual deployment, node weights for the corresponding scene type are directly retrieved from a preset weight table based on the multi-label classification results of the current scene, and normalized to 0 to 1. Traffic participants are the core dominant element in the dynamic evolution of the scene, with a weight range of 0.7 to 0.9; environmental attributes are global constraint elements, with a weight range of 0.4 to 0.6; and road elements are static basic elements, with a weight range of 0.3 to 0.5. The scene evolution probability of a single node is P. s =w s ×S trend In the formula P s w represents the initial scene evolution probability for a single node. s S represents the scene impact weight for the corresponding node. trend This represents the normalized temporal evolution characteristic amplitude.
[0109] S trend The trend amplitude is represented by the absolute value of the slope of the fitted trend line at the end of the sliding window after multinomial fitting of the continuous frame feature change trajectory of the same node within the sliding window. This absolute slope value reflects the drastic change of the node feature at the current moment. This original representation is then mapped to the [0,1] interval through Min-Max normalization. The normalization baseline interval is determined by the statistical distribution of the trend slope of this type of node in the historical scene dataset, where the minimum value is the 5th percentile of the historical slope of this type of node and the maximum value is the 95th percentile. trend The larger the value, the more drastic the current state change trend of the node, and the stronger its driving effect on the scene evolution. When S trendWhen the value exceeds the [0,1] interval after normalization, it is truncated to the boundary value.
[0110] For example, the dynamic vehicle node, as the core evolution node, has a scene influence weight of 0.85. Combined with its temporal evolution characteristic amplitude of 0.78, the weighted calculation yields an initial scene evolution probability of 0.663 for a single node. The road lane node has a scene influence weight of 0.35 and a temporal evolution characteristic amplitude of 0.2, resulting in an initial evolution probability of 0.07. This accurately distinguishes the difference in evolutionary contributions between core dynamic elements and static elements.
[0111] Since the initial scene evolution probability of a single node is calculated solely based on the node's own temporal evolution characteristics without considering spatial binding and dynamic interaction constraints between nodes, prediction bias exists. Therefore, bias correction is performed based on the interaction correlation coefficient of the associated edges. The correction formula is P. correct =P s ×(1+β×C link ), where P correct To correct the single-node scene evolution probability, β is a preset interaction correction coefficient, representing the correction strength of the interaction correlation coefficient on the single-node scene evolution probability, with a value range of [0, 0.5]. β is obtained through grid search calibration on the offline driving scenario dataset, with the scene evolution probability prediction accuracy as the optimization objective. The larger the value of β, the greater the correction magnitude of the interaction correlation coefficient on the single-node scene evolution probability. In this embodiment, β = 0.4 is used for calibration, and β must satisfy 1 + β × C. link ≤1.2, to avoid excessive probability inflation after correction, C link This represents the interaction correlation coefficient of the edges associated with the corresponding nodes. The higher the interaction correlation coefficient, the stronger the correlation between nodes in the evolutionary linkage. This results in a greater correction magnitude for the initial scene evolution probability of a single node, effectively correcting the one-sided error caused by isolated time-series prediction. For example, the interaction correlation coefficient between the edges associated with the vehicle node and the obstacle node ahead is 0.85, and the interaction correction coefficient is 0.4. This positively corrects the initial scene evolution probability of the vehicle node, improving the accuracy of evolution prediction in interactive scenarios and conforming to the real-world road condition linkage change patterns.
[0112] After completing the probability correction of all nodes, hierarchical fusion and global mapping processing are performed according to node type. The correction scenario evolution probability of nodes of the same type is aggregated according to three levels: traffic participant nodes, environmental attribute nodes, and road element nodes. The probability dimension of various types of nodes is unified through hierarchical normalization algorithm to eliminate the numerical scale difference of different types of nodes. Finally, a scenario evolution probability distribution covering all driving scenarios is generated. This scenario evolution probability distribution includes the probability quantification results of switching from the current scenario to various scenarios such as congestion, following other vehicles, lane changing, low visibility, and intersection interaction.
[0113] This constructs a multi-level, quantifiable, and correctable scenario evolution prediction system. It achieves dynamic prediction through node time-series trend extraction, resolves isolated prediction biases through interactive correlation coefficient correction, and achieves unified global scenario quantification through hierarchical fusion, accurately representing the core characteristics of the multi-element linkage evolution of driving scenarios. The output scenario evolution probability distribution is the core judgment basis for subsequent perception strategy switching, parameter adjustment level division, and adjustment magnitude allocation. At the same time, it provides scenario evolution data support for the dynamic weight configuration of subsequent perception strategy correction, realizing the full-link linkage of scenario prediction and adaptive adjustment of perception parameters.
[0114] S3. Based on the multi-label classification results and scene evolution probability distribution of the driving scenario, configure the perception strategy under the current driving scenario. When the scene evolution probability is greater than the preset threshold, switch the perception strategy to dynamically adjust the control parameters of the sensor.
[0115] Specifically, dynamically adjusting the control parameters of the sensor includes:
[0116] During normal operation, based on the multi-label classification results of driving scenarios and the probability distribution of scenario evolution, a preset strategy template is matched and a perception strategy for the current driving scenario is configured.
[0117] When the probability of scene evolution exceeds a preset threshold, the perception strategy is switched, and the parameter adjustment level is divided according to the numerical range of the scene evolution probability to determine the parameter adjustment benchmark of the sensor.
[0118] The parameter adjustment range is allocated according to the interference level of the sensing space partition, and the control parameters of the sensor are adjusted in conjunction with the parameter adjustment benchmark and the parameter adjustment range.
[0119] The sensor's control parameters include timing acquisition parameters, sensing gain parameters, and detection field of view parameters;
[0120] The linkage adjustment specifically involves: adjusting the timing acquisition parameters of the sensor based on the parameter adjustment benchmark; adjusting the sensing gain parameters according to the sensing space partition based on the parameter adjustment range; and dynamically adjusting the sensor's detection field of view parameters based on the spatial distribution range of nodes in the enhanced semantic map.
[0121] Fixed-parameter sensor perception modes cannot adapt to dynamically changing driving scenarios. Conventional perception parameters can only adapt to a single steady-state road condition. Under conditions of rapid scene switching, increased environmental interference, and dynamic evolution of traffic participants, problems such as insufficient perception frame rate, local perception accuracy degradation, and detection field of view matching failure are likely to occur. At the same time, driving scenarios distinguish between steady-state normal conditions and dynamic abrupt conditions. It is necessary to adapt perception strategies in layers, predict changes in conditions in advance based on scene evolution trends, and achieve fine-grained and differentiated parameter adjustment according to scene probability range and regional interference levels. Finally, through the coordinated adaptation of three core parameters—time-series acquisition, perception gain, and detection field of view—the stability and accuracy of sensor perception in all scenarios can be guaranteed.
[0122] Specifically, the perception strategy configuration for the normal operation phase is executed first. The multi-label classification results of the driving scenario and the scenario evolution probability distribution are used as dual matching criteria to retrieve the preset strategy template library to complete the strategy matching configuration. In this solution, the preset strategy template library is a standardized strategy set based on massive driving conditions calibration. Different templates correspond to different steady-state scenario types, basic parameter thresholds and normal adjustment ranges. The template classification corresponds to the multi-label classification scenario, including normal strategy templates such as normal straight roads, urban following, intersection crossing, highway cruising and low visibility environment.
[0123] The judgment rule for normal operation is that if the current scene label matches the steady-state scene type and the evolution probability of all dimensions of the scene is lower than the preset threshold, the driving scene is judged to be in a stable state. There is no need to make significant adjustments to the perception parameters. The preset strategy template of the corresponding scene is directly matched, and the basic perception strategy and sensor baseline parameters under the current working condition are solidified. For example, when the multi-label classification result is a high-speed cruise scene, and the maximum evolution probability of the scene to sudden scenes such as congestion and lane change is 0.22, which is lower than the preset threshold of 0.4, it is judged to be a normal working condition. The high-speed cruise strategy template is matched, and the sensor's normal timing acquisition frequency, standard perception gain and fixed detection field of view are configured.
[0124] Based on the normal strategy configuration, the global scenario evolution probability is monitored in real time, and the logic for dynamic scenario strategy switching and adjustment benchmark determination is executed. The preset threshold is 0.4. When the evolution probability of any scenario detected is greater than the preset threshold, it is determined that the current driving scenario has a dynamic switching trend and the working condition is about to change abruptly. The normal perception strategy is immediately exited, and the dynamic perception strategy switching mechanism is triggered. In order to achieve fine-grained parameter adjustment and avoid over-adjustment or under-adjustment caused by single-amplitude parameter adjustment, multiple parameter adjustment levels are divided according to the numerical range of scenario evolution probability. Different probability ranges correspond to different intensity parameter adjustment benchmarks. The higher the probability value, the more intense the scenario evolution trend, and the greater the correction intensity of the parameter adjustment benchmark.
[0125] This scheme divides the scenario evolution probability range of 0 to 1 into three adjustment levels. The first level is the weak evolution range [0.4, 0.6), corresponding to a mild adjustment benchmark; the second level is the medium evolution range [0.6, 0.8), corresponding to a moderate adjustment benchmark; and the third level is the strong evolution range [0.8, 1], corresponding to a severe adjustment benchmark. For example, in the current scenario of a vehicle cutting into the lane in front at close range, which is a high-risk dynamic interaction situation where a vehicle in an adjacent lane cuts into the lane in front of the current vehicle at a relatively close distance, the evolution probability is 0.72, falling into the medium evolution range. The moderate parameter adjustment benchmark is based on the nominal value of the current sensor control parameters. The initial correction coefficient of each control parameter is determined according to the correction ratio corresponding to the probability interval. The product of the correction coefficient and the nominal value is the parameter adjustment benchmark amount. The correction coefficient corresponding to the medium evolution interval is 10% to 20% of the nominal value (5% to 10% for the weak evolution interval and 20% to 35% for the strong evolution interval). That is, the parameter adjustment benchmark amount = nominal parameter value × correction coefficient, which determines the basic correction level of this sensor parameter linkage adjustment.
[0126] After the parameter adjustment benchmark is determined, differentiated parameter adjustment ranges are allocated according to the interference level of the sensing space partition to achieve precise adaptation between the benchmark level and local interference. In conjunction with the aforementioned sensing space partition, the sensor detection range is divided into three types of sensing space partitions: near, medium and far. The degree of occlusion interference, clutter interference and atmospheric attenuation interference varies significantly among different partitions. The normalized value range of the interference level of the sensing space partition is 0 to 1. The larger the value, the worse the local sensing environment and the higher the risk of sensing distortion.
[0127] A positive mapping relationship is established between the degree of interference in a partition and the adjustment range. The higher the degree of interference in a partition, the larger the corresponding parameter adjustment range is allocated, thereby compensating for the loss of sensing performance in local areas. The formula for allocating the adjustment range is A. j =A base ×I j In the formula A j For the parameter adjustment range of the j-th perceptual space partition, A base I is the base amplitude coefficient corresponding to the gear. j This is the quantification value of the interference level in the j-th sensing space partition. For example, the current interference level in the central sensing space partition is 0.75, and the interference level in the far zone is 0.4. Based on the same moderate adjustment benchmark, the central zone is allocated a larger adjustment range for sensing parameters, while the far zone is allocated a smaller adjustment range, thus achieving differentiated compensation for local interference.
[0128] Based on the aforementioned established parameter adjustment benchmarks and zonal adjustment ranges, differentiated linkage adjustment of the core control parameters of the three types of sensors is completed. The adjustment logic of the three types of parameters is independent and linked and adapted to each other, covering the full-dimensional control of sensor sampling, signal gain and detection range. Among them, the timing acquisition parameter represents the data sampling frequency and sampling interval timing index of the sensor. The overall parameter is uniformly adjusted according to the parameter adjustment benchmark, without zonal differentiation. The more intense the scene evolution trend, the higher the adjustment benchmark level, the higher the timing acquisition frequency and the shorter the sampling interval, so as to improve the real-time data update in dynamic scenes.
[0129] The sensing gain parameter characterizes the gain index of sensor signal amplification and feature extraction weights. It is adjusted partition by partition according to the parameter adjustment range corresponding to each sensing spatial partition. The sensing gain is increased for partitions with high interference, which enhances the extraction of local effective signals and suppresses clutter and noise interference. The detection field of view parameter characterizes the sensor's detection angle, detection distance and effective field of view coverage. It is dynamically adapted and adjusted in combination with the spatial distribution range of nodes in the enhanced semantic map. According to the overall spatial coverage area of the nodes in the enhanced semantic map, the optimal detection field of view of the sensor is matched. When the spatial distribution range of nodes is wide and the target is scattered, the detection field of view is widened. When the nodes are concentrated and the target is locally clustered, the precise field of view is narrowed, taking into account the needs of global perception and local precise perception.
[0130] This establishes a hierarchical perception and control system encompassing steady-state adaptation to normal templates, hierarchical control of mutation probability, differentiated compensation for local interference, and multi-parameter linkage adaptation. This addresses the technical shortcomings of traditional sensors with fixed parameters that cannot adapt to dynamic driving scenarios. Through differentiated linkage adjustment of the three types of parameters, the system simultaneously ensures the real-time performance, local accuracy, and field-of-view adaptability of perception data, significantly improving the adaptive capability of multi-sensor perception in dynamically evolving scenarios. It provides a core control object for parameter correction, determining the final effect of subsequent residual calculation, weighted correction, and parameter locking, and providing a complete precondition basis for iterative parameter correction.
[0131] S4. Monitor the adjusted perception performance indicators, and based on the residual between the perception performance indicators and the preset baseline indicators, correct the control parameters of the current perception strategy.
[0132] Specifically, modifying the control parameters of the current perception strategy includes:
[0133] Monitor the sensor's sensing performance indicators after adjustment and compare them with preset baseline indicators. Combine the global sensing state with the local sensing state of different sensing spatial partitions to determine the global residual and local residual in layers.
[0134] Based on the current driving scenario's evolution, correction weights are assigned to the global residual and the local residual respectively, and after weighted processing, the global weighted residual and the local weighted residual are obtained respectively.
[0135] The parameters are obtained by correcting the time series based on global weighted residuals, the sensing gain parameters are corrected by per-sensory spatial partitioning based on local weighted residuals, and the detection field parameters are corrected by combining the distribution state of nodes in the enhanced semantic graph.
[0136] Continuously monitor the trend of residual changes under multiple frames of data, gradually reduce the correction step size, and lock the control parameters of the current sensing strategy when the residual stabilizes within the allowable range.
[0137] The dynamic linkage adjustment of sensor control parameters is an open-loop pre-adjustment method based on scene prediction. It only pre-adjusts parameters according to the scene evolution trend and the degree of regional interference, without introducing feedback constraints on the actual sensing performance of the sensor. Affected by uncertain factors such as real-time environmental changes, slight drift of sensor hardware, and dynamic deviations in scene interaction, the parameters after pre-adjustment will still have sensing performance deviations, and a single adjustment cannot completely eliminate steady-state errors. At the same time, the impact dimensions of global overall sensing deviation and local regional sensing deviation on sensor parameters are different, requiring hierarchical quantitative correction. However, a fixed correction step size is prone to parameter oscillation and unstable convergence. Therefore, it is necessary to construct a hierarchical residual system based on the deviation between actual sensing performance and baseline indicators, dynamically configure correction weights in combination with scene evolution status, achieve differentiated and accurate correction, and lock the optimal parameters through an adaptive step size convergence mechanism to form a closed-loop iterative system for the sensing strategy.
[0138] Specifically, the calculation and determination of the hierarchical residuals are completed first. The sensor's full-dimensional perception performance indicators after dynamic adjustment are monitored in real time, including the overall recognition accuracy, data efficiency, signal-to-noise ratio, and target matching accuracy. These indicators are used to objectively characterize the actual working performance of the sensor under the current parameter configuration. The real-time monitored perception performance indicators are compared with the preset baseline indicators dimension by dimension. The preset baseline indicators are the optimal calibration performance parameters under standard sensor operating conditions and in a non-interference environment, which serve as the criterion for judging perception performance deviation. This scheme adopts a hierarchical deviation judgment logic, which combines the global perception state and the local perception state of each perception space partition to calculate the global residual and local residual respectively.
[0139] The global perception state refers to the overall working state of the sensors within the entire vehicle's perception range, corresponding to the overall performance deviation of the sensors. The generated global residual is used to characterize the degree of deviation between the overall perception performance of the sensors and the standard baseline. The local perception state refers to the perception working state of the aforementioned near, middle and far perception space partitions, corresponding to the local perception performance deviation within a single partition. The generated local residual is used to characterize the performance deviation of each subdivided perception area.
[0140] The residual calculation adopts the deviation normalization calculation method, which maps the performance deviation of the whole area and each zone to the interval of 0 to 1. The larger the value, the higher the degree of deviation of the current sensing performance from the standard baseline, and the stronger the need for parameter correction. For example, if the overall sensing performance of the sensor deviates from the baseline standard, the global residual calculation after normalization is 0.38. Among them, the near area has serious occlusion interference, with a local residual of 0.65; the middle area has stable operation, with a local residual of 0.21; and the far area has slight attenuation, with a local residual of 0.3, thus completing the accurate quantitative differentiation of the layered residuals.
[0141] After the hierarchical residual calculation is completed, dynamic correction weight configuration and weighted residual operation are performed. The current driving scenario's scenario evolution state is represented by the aforementioned output scenario evolution probability distribution, node dynamic change degree, and scenario switching intensity, which is used to determine the dynamic fluctuation level of the current scenario in real time and provide a scenario basis for residual correction weight configuration. The correction weight is dynamically configured according to the intensity of scenario evolution. The more intense the scenario evolution and the more obvious the scenario switching trend, the higher the residual correction weight value, strengthening the deviation correction force. When the scenario state tends to be steady, the correction weight is reduced to avoid excessive correction interfering with perception stability. The weight values are normalized to 0 to 1, and correction weights are configured for the global residual and the local residuals of each partition, respectively.
[0142] Based on the configured dynamic correction weights, weighted operations are performed on the two types of residuals. The core logic of the weighted operation is the coupling operation between the residual value and the corresponding correction weight. Finally, normalized global weighted residuals and local weighted residuals of each partition are obtained. The weighted residuals are integrated with the performance deviation magnitude and the dynamic level of the current scenario, providing a scenario-adaptive correction basis for subsequent parameter corrections. For example, if the current driving scenario is a strongly evolving state with a close-range intrusion from the vehicle in front, and the scenario fluctuation level is high, a correction weight of 0.75 is configured for the global residuals, and a correction weight of 0.8 is configured for the high-deviation local residuals in the near zone. After weighted coupling operation, a global weighted residual with enhanced correction strength and a local weighted residual in the near zone are obtained, which are adapted to the correction requirements of dynamic change scenarios. For the steady-state partitions in the mid- and far-field regions, lower correction weights are configured to weaken ineffective correction actions.
[0143] After solving the weighted residuals, precise corrections are performed on three types of sensor control parameters. The timing acquisition parameters are uniform sampling frequency and sampling interval timing parameters across the entire sensor domain, without distinguishing between sensing zones. Therefore, a uniform correction is performed across the entire domain based on the global weighted residuals. The overall sampling timing specifications of the sensor are adjusted according to the overall deviation of global perception to compensate for overall performance defects such as global perception lag and insufficient frame rate. The sensing gain parameters are the signal gain parameters of each sensing spatial zone, which have the attribute of zone difference adjustment. Therefore, the gain is corrected independently for each zone based on the local weighted residuals corresponding to each zone. The gain correction magnitude is increased for zones with large residuals and serious performance deviations to accurately compensate for signal attenuation and feature extraction failure caused by local interference. The steady-state zones are slightly corrected or the original gain parameters are maintained.
[0144] The correction magnitudes for the parameters are ΔParam global =W global ×R global ×S init ΔParam local,j =W local,j ×R local,j ×S init In the formula, ΔParam global W is the correction amount for timing acquisition parameters. global For globally adjusted weights, R global For the globally weighted residual, S init As the initial correction step size, this embodiment is calibrated to 5% of the current parameter value, ΔParam local,j W is the sensing gain correction for the j-th sensing space partition. local,j R is the adjusted weight for this partition. local,j This is the local weighted residual for this partition; the corrected parameter value = current parameter value ± parameter correction amount (the sign is determined by the direction of the residual; increase the parameter when the performance is lower than the baseline, and decrease the parameter when the performance is higher than the baseline).
[0145] The detection field of view parameters are dynamically corrected based on the distribution status of nodes within the enhanced semantic graph. The node distribution status is a comprehensive representation of the spatial distribution range of nodes, including the spatial coverage of semantic graph nodes, the degree of node clustering, the density of target distribution, and the dynamic offset trend of nodes. Unlike the coarse adjustment of the basic spatial range, this step completes the secondary correction of the field of view parameters based on the comprehensive distribution status of nodes, so as to achieve the adaptation of the detection field of view with the target distribution of the real-time scene and avoid the subtle field of view matching deviation after the coarse adjustment of parameters.
[0146] After parameter correction is completed, residual trend monitoring and parameter convergence locking logic are executed. The residual change trend corresponding to multiple consecutive frames of sensing data is continuously monitored. Based on the convergence state of the residuals, the correction step size is dynamically and gradually reduced. The larger the residual deviation, the larger the initial correction step size is to ensure rapid correction of deviations. As the residuals gradually converge and the sensing performance approaches the baseline standard, the correction step size is continuously reduced to avoid parameter oscillation and performance fluctuations caused by excessively large step sizes. A preset allowable stability range for residuals is defined. When the global residuals and local residuals of each partition monitored for multiple consecutive frames are all stably within the preset allowable stability range, it is determined that the current sensor control parameters and sensing strategy parameters have reached the optimal adaptation state. The control parameters corresponding to the current sensing strategy are locked, and the iterative correction process is terminated.
[0147] Initial correction step size S init The calibration is set to 5% of the current control parameter value, i.e., S init =0.05×Param current Param current Given the current control parameter value, the step size decay rule is to halve the step size after each round of correction, i.e., S. p =S init / 2 p p represents the number of correction cycles completed, and S represents the number of correction cycles completed. p To determine the correction step size after the p-th round of correction, the lower bound S of the correction step size is... min It is 0.5% of the current parameter value, i.e., S min =0.005×Param current Once the step size decays to the lower limit, it will no longer decrease. The convergence criterion is that the global residual and the local residual of all partitions of 10 consecutive frames of monitoring data are less than 0.05 (the allowable stable range of normalized residuals). If the condition is met, the current parameters are locked and the iterative correction is terminated.
[0148] For example, if the initial residual is too large, a large correction step size is used to quickly correct the parameters, and the residual decreases rapidly. When the residual approaches the allowable stable range, the correction step size is gradually halved to fine-tune the parameters. When the residual is stable in the allowable range for 10 consecutive frames, convergence is determined to be complete, and the current time-series acquisition parameters, sensing gain parameters, and detection field of view parameters are locked.
[0149] This leads to the construction of a feedback correction mechanism that incorporates hierarchical residual quantization, dynamic scene weighting, differentiated parameter correction, and adaptive step-size convergence. This mechanism overcomes the technical limitations of traditional open-loop pre-adjustment, which cannot eliminate steady-state errors or adapt to dynamic scene deviations. Through global and local hierarchical residual design, it balances overall perception accuracy with local partition perception performance. It ensures that the correction intensity matches the real-time operating conditions of the scene, avoiding the limitations of static fixed weights. Through targeted weighted correction of three types of parameters, it achieves refined, differentiated, and closed-loop parameter adjustment. Ultimately, it achieves optimal adaptation of multi-sensor perception parameters to dynamic driving scenarios, significantly improving the stability, accuracy, and robustness of the perception system in complex dynamic scenarios.
[0150] This step is the final stage of the entire driving scene recognition and adaptation method. After the parameters are locked, a normalized steady-state perception strategy loop is formed, continuously outputting high-precision and high-stability scene perception data and scene recognition results. This enables the adaptive adaptation function of multi-sensor perception parameters from dynamic pre-adjustment to closed-loop precise correction and finally steady-state locking, completing the logical closed loop of the entire technical solution.
[0151] Example 2:
[0152] like Figure 3 As shown, this embodiment provides a driving scene recognition and adaptation system based on multi-sensor fusion, including a fusion module, an association module, an adjustment module, and a correction module.
[0153] The fusion module is used to acquire perception data from the corresponding sensors on the vehicle, perform spatiotemporal alignment on the perception data, estimate the environmental degradation factor of the sensors respectively, generate the credibility of the corresponding sensors, extract perception features by weighting the corresponding perception data respectively, and fuse the perception features to generate scene features.
[0154] The association module is used to construct a scene semantic graph based on scene features, temporally associate the scene semantic graphs of the current frame with those of historical frames to generate an enhanced semantic graph, perform multi-label classification of driving scenes based on the enhanced semantic graph, and predict the scene evolution probability distribution based on the temporal evolution features of nodes in the enhanced semantic graph.
[0155] The adjustment module is used to configure the perception strategy under the current driving scenario based on the multi-label classification results and scenario evolution probability distribution of the driving scenario. When the scenario evolution probability is greater than the preset threshold, the perception strategy is switched to dynamically adjust the control parameters of the sensor.
[0156] The correction module is used to monitor the adjusted perception performance indicators and correct the control parameters of the current perception strategy based on the residual between the perception performance indicators and the preset baseline indicators.
[0157] The above description is merely a preferred embodiment of this application. The scope of protection of this application is not limited to the above embodiments. All technical solutions falling within the scope of this application's concept are within the scope of protection of this application. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of this application should also be considered within the scope of protection of this application.
Claims
1. A driving scene recognition and adaptation method based on multi-sensor fusion, characterized in that, include: Acquire perception data from the corresponding sensors on the vehicle, perform spatiotemporal alignment on the perception data, determine the environmental degradation factor of each sensor, generate the credibility of the corresponding sensor, extract perception features by weighting the corresponding perception data, and fuse the perception features to generate scene features. A scene semantic graph is constructed based on scene features. The scene semantic graphs of the current frame and historical frames are temporally correlated to generate an enhanced semantic graph. The driving scene is classified into multiple labels based on the enhanced semantic graph, and the scene evolution probability distribution is predicted based on the temporal evolution features of the nodes in the enhanced semantic graph. The generation of the enhanced semantic graph includes: A scene semantic graph is constructed based on scene features. The scene semantic graph uses road elements, traffic participants and environmental attributes as nodes, and the spatial positional relationship and dynamic interaction relationship between nodes as associated edges. Extract the scene semantic graph of historical frames within the sliding window, perform node type matching between the scene semantic graph of the current frame and the historical frames, and configure temporal weights for different nodes to fuse the node features of the current frame and the historical frames. Based on the degree of state change of adjacent frame nodes within the sliding window, the interaction correlation coefficient of the corresponding associated edge is dynamically updated to realize the temporal correlation between node features and associated edges, and the credibility is converted into a correction coefficient to perform weighted correction on node features and generate an enhanced semantic graph. Based on the multi-label classification results and scenario evolution probability distribution of driving scenarios, a perception strategy is configured for the current driving scenario. When the scenario evolution probability is greater than a preset threshold, the perception strategy is switched to dynamically adjust the control parameters of the sensors. Monitor the adjusted perception performance indicators, and based on the residual between the perception performance indicators and the preset baseline indicators, correct the control parameters of the current perception strategy.
2. The driving scene recognition and adaptation method based on multi-sensor fusion according to claim 1, characterized in that, The features of the fused generated scene include: The global degradation factor and local degradation factor of each sensor are calculated separately, and the global degradation factor and local degradation factor are cross-coupled to obtain the environmental degradation factor of the corresponding sensor. Based on environmental degradation factors, and combined with the channel fixation accuracy of the sensor and the interference level of the sensing space partition, a two-dimensional matrix with channels as rows and sensing space partitions as columns is generated as the reliability of the corresponding sensor. The perception data of each sensor is weighted based on credibility, and perception features of different granularities are extracted according to the perception domain to which the sensor belongs. Perceptual features within the same perceptual domain are aggregated to generate scene features.
3. The driving scene recognition and adaptation method based on multi-sensor fusion according to claim 2, characterized in that, The global degradation factor characterizes the degree of attenuation of the overall sensing capability of the sensor by the global environment, while the local degradation factor characterizes the degree of attenuation of the sensor's local sensing capability by different sensing spatial partitions. The two-dimensional matrix uses the fixed accuracy of the sensor channel as the base weight, and combines the environmental degradation factor and the interference level of the corresponding sensing space partition to perform row-by-row and column-by-column joint operation on the matrix elements to obtain the initial matrix, and sets a threshold lower limit constraint on the initial matrix elements.
4. The driving scene recognition and adaptation method based on multi-sensor fusion according to claim 3, characterized in that, The predicted scenario evolution probability distribution includes: Temporal sampling of the continuous frame feature change trajectory corresponding to different nodes in the enhanced semantic graph is performed based on the sliding window, and the temporal evolution features of the corresponding nodes are extracted respectively. The temporal evolution features are weighted according to the preset scene influence weights to calculate the scene evolution probability of a single node. By combining the interaction correlation coefficients of the edges between nodes, the interaction deviation of the scene evolution probability of a single node is corrected, and the scene evolution probability distribution of the driving scenario is obtained by hierarchical fusion and mapping according to node type.
5. The driving scene recognition and adaptation method based on multi-sensor fusion according to claim 4, characterized in that, The degree of state change of adjacent frame nodes is the difference in node features of the same node between two adjacent frames. The interaction correlation coefficient is a quantitative value of the spatial position relationship and dynamic interaction relationship corresponding to the associated edge. It is used to characterize the spatial binding strength and interaction tightness between nodes, and is positively adjusted synchronously with the degree of state change of adjacent frame nodes. The continuous frame feature change trajectory is a temporal sequence formed by sequentially concatenating the node features of the same node frame by frame within a sliding window; the temporal evolution feature is a trend feature extracted after sampling the continuous frame feature change trajectory; The scenario influence weights are determined based on the dominant role of nodes in the driving scenario.
6. The driving scene recognition and adaptation method based on multi-sensor fusion according to claim 5, characterized in that, The control parameters for dynamically adjusting the sensor include: When the probability of scene evolution exceeds a preset threshold, the perception strategy is switched, and the parameter adjustment level is divided according to the numerical range of the scene evolution probability to determine the parameter adjustment benchmark of the sensor. The parameter adjustment range is allocated according to the interference level of the sensing space partition, and the control parameters of the sensor are adjusted in conjunction with the parameter adjustment benchmark and the parameter adjustment range.
7. The driving scene recognition and adaptation method based on multi-sensor fusion according to claim 6, characterized in that, The control parameters of the sensor include timing acquisition parameters, sensing gain parameters, and detection field of view parameters; The linkage adjustment specifically involves: adjusting the timing acquisition parameters of the sensor based on the parameter adjustment benchmark; adjusting the sensing gain parameters according to the sensing space partition based on the parameter adjustment range; and dynamically adjusting the sensor's detection field of view parameters based on the spatial distribution range of nodes in the enhanced semantic map.
8. The driving scene recognition and adaptation method based on multi-sensor fusion according to claim 7, characterized in that, The modification of the control parameters of the current perception strategy includes: By comparing the perception performance indicators with the preset baseline indicators and combining the global perception state with the local perception state of different perception spatial partitions, the global residual and local residual are determined in layers. Based on the current driving scenario's evolution, correction weights are assigned to the global residual and the local residual respectively, and after weighted processing, the global weighted residual and the local weighted residual are obtained respectively. The parameters are obtained by correcting the time series based on global weighted residuals, the sensing gain parameters are corrected by per-sensory spatial partitioning based on local weighted residuals, and the detection field parameters are corrected by combining the distribution state of nodes in the enhanced semantic graph. Continuously monitor the trend of residual changes under multiple frames of data, gradually reduce the correction step size, and lock the control parameters of the current sensing strategy when the residual stabilizes within the allowable range.
9. A driving scene recognition and adaptation system based on multi-sensor fusion, used to implement the driving scene recognition and adaptation method based on multi-sensor fusion as described in any one of claims 1-8, characterized in that, include: Integration module, association module, adjustment module, and correction module; The fusion module is used to acquire perception data from the corresponding sensors on the vehicle, perform spatiotemporal alignment on the perception data, estimate the environmental degradation factor of the sensors respectively, generate the credibility of the corresponding sensors, extract perception features by weighting the corresponding perception data respectively, and fuse the perception features to generate scene features. The association module is used to construct a scene semantic graph based on scene features, temporally associate the scene semantic graphs of the current frame with those of historical frames to generate an enhanced semantic graph, perform multi-label classification of driving scenes based on the enhanced semantic graph, and predict the scene evolution probability distribution based on the temporal evolution features of nodes in the enhanced semantic graph. The adjustment module is used to configure the perception strategy under the current driving scenario based on the multi-label classification results and scenario evolution probability distribution of the driving scenario. When the scenario evolution probability is greater than the preset threshold, the perception strategy is switched to dynamically adjust the control parameters of the sensor. The correction module is used to monitor the adjusted perception performance indicators and correct the control parameters of the current perception strategy based on the residual between the perception performance indicators and the preset baseline indicators.
Citation Information
Patent Citations
Automatic driving scene evolution knowledge graph construction method and risk prediction evaluation method
CN119721221A
Multi-modal fusion vehicle dynamic sensing method, device and equipment and storage medium
CN121211340A
