A target perception method based on feature difference and error propagation

Abstract

The embodiment of the application provides a target perception method based on feature difference and error propagation, which comprises the following steps: pixel-level comparison and residual extraction are performed on t-frame image features and t-1 frame image features by using a feature difference space, and feature difference information is obtained; vehicle attitude information is used as a geometric constraint, a distance of a target in an image is determined based on the feature difference information by using a depth estimation network, a two-dimensional environment depth map is obtained, target detection is performed based on the two-dimensional environment depth map, and a first detection result is obtained; the two-dimensional environment depth map is mapped to a three-dimensional space, a pseudo point cloud map is generated, target detection is performed on the pseudo point cloud map, and a second detection result is obtained; the first detection result and the second detection result are fused by using a weighted confidence, and a target detection result is determined; in the process of performing target detection, the weight distribution of the feature difference space and the depth estimation network is adjusted based on a target detection error and a feature detection error.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, specifically to a target perception method based on feature difference and error propagation. Background Technology

[0002] As a core component of autonomous driving and intelligent assisted driving, the environmental perception system directly determines the accuracy of a vehicle's localization, ranging, and behavior prediction of surrounding obstacles. In-depth research and feedback from practical applications have revealed significant technical limitations in handling complex dynamic traffic scenarios. First, regarding temporal feature fusion, existing technologies largely focus on stacking static features, making it difficult to accurately capture subtle movement trends of obstacles. This leads to severe temporal jitter and edge blurring in perception results when the target is moving at high speed or the vehicle experiences severe vibrations. Second, current depth estimation cannot compensate for the impact of attitude changes on depth features in real time when the vehicle's attitude changes frequently (such as acceleration pitch and cornering roll), resulting in a significant decrease in ranging accuracy. The most critical flaw is that most existing technologies follow an open-loop "perception-output" logic, lacking an effective closed-loop error correction and feedback mechanism. This results in poor robustness to occlusion, extreme lighting conditions, or sensor noise, making it difficult to maintain long-term temporal consistency of perception results. Summary of the Invention

[0003] The purpose of this application is to provide a target perception method, apparatus, and device based on feature difference and error propagation, and the specific technical solution adopted is as follows: Firstly, a target perception method based on feature difference and error propagation is provided, the method comprising: The feature difference space is used to perform pixel-level comparison and residual extraction on the features of frame t and frame t-1 to obtain feature difference information, where t is an integer greater than or equal to 1; Using vehicle attitude information as geometric constraints, a depth estimation network is used to determine the distance of targets in the image based on the feature difference information, resulting in a two-dimensional environmental depth map. Target detection is then performed based on the two-dimensional environmental depth map to obtain a first detection result. The two-dimensional environmental depth map is mapped to three-dimensional space to generate a pseudo point cloud map, and the target detection is performed on the pseudo point cloud map to obtain a second detection result; The first detection result and the second detection result are fused using a weighted confidence score to determine the target detection result, wherein the target detection result includes at least the target's position, velocity, attitude, and category information; During the target detection process, the weight distribution of the feature difference space and the depth estimation network is adjusted based on the target detection error and the feature detection error. The target detection error is obtained by comparing the first detection result and the second detection result, and the feature detection error is obtained by comparing the predicted feature distribution and the actual feature distribution. The predicted feature distribution is the predicted feature distribution of the t-th frame image based on the two-dimensional environment depth map, vehicle attitude information and the features of the t-1 frame image. The features of the t-th frame image are then determined as the actual feature distribution.

[0004] Secondly, a target sensing device based on feature difference and error propagation is provided, the device comprising: The feature difference module is used to perform pixel-level comparison and residual extraction of the features of frame t and frame t-1 using the feature difference space to obtain feature difference information, where t is an integer greater than or equal to 1; The first detection module is used to use vehicle attitude information as geometric constraints, and to use a depth estimation network to determine the distance of the target in the image based on the feature difference information to obtain a two-dimensional environment depth map, and to perform target detection based on the two-dimensional environment depth map to obtain a first detection result. The second detection module is used to map the two-dimensional environment depth map to three-dimensional space to generate a pseudo point cloud map, and to perform target detection on the pseudo point cloud map to obtain a second detection result. The result fusion module is used to fuse the first detection result and the second detection result using weighted confidence to determine the target detection result, wherein the target detection result includes at least the target's position, velocity, attitude and category information; The weight adjustment module is used to adjust the weight distribution of the feature difference space and the depth estimation network based on the target detection error and the feature detection error during the target detection process. The target detection error is obtained by comparing the first detection result and the second detection result, and the feature detection error is obtained by comparing the predicted feature distribution and the actual feature distribution. The predicted feature distribution is the predicted feature distribution of the t-th frame image based on the two-dimensional environment depth map, vehicle pose information and the features of the t-1 frame image, and the features of the t-th frame image are determined as the actual feature distribution.

[0005] Thirdly, an electronic device is provided, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor invokes the instructions in the memory to cause the electronic device to execute the target perception method based on feature difference and error propagation described above.

[0006] Fourthly, a computer program product is provided, comprising: computer program code, which, when executed on an electronic device, causes the electronic device to perform the method described in the first aspect or any possible implementation thereof.

[0007] Fifthly, a computer-readable storage medium is provided that stores computer program code, which, when executed on a computer, causes the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0008] This application offers the following advantages: By employing feature differentiation, depth estimation, pseudo-point cloud fusion, and adaptive error adjustment, a closed-loop perception system is constructed, spanning from 2D to 3D and from static to dynamic modes. This system boasts high robustness, adaptability, and multimodal collaboration, making it suitable for scenarios requiring precise target perception, such as autonomous driving and intelligent monitoring. It integrates 2D image features, 3D point cloud information, and vehicle posture data, overcoming the limitations of single-modal perception (such as the scale problem of monocular vision and the sparsity problem of point clouds). Through dual feedback of detection and feature errors, online adjustment of network weights is achieved, enhancing the system's adaptability to environmental changes (such as changes in illumination and target occlusion). Compared to LiDAR, pseudo-point clouds are generated based on visual data, resulting in lower costs and compatibility with existing visual algorithms, making them suitable for cost-sensitive scenarios such as autonomous driving and robotics. Robust target detection (position, velocity, and attitude) is achieved in dynamic traffic scenarios, supporting path planning and obstacle avoidance. Multi-target tracking and behavior analysis are realized in complex scenarios (such as densely populated areas). 3D spatial perception and dynamic target avoidance are achieved in unknown environments. Attached Figure Description

[0009] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A flowchart illustrating a target perception method based on feature difference and error propagation provided in an embodiment of this application; Figure 2 This application provides an autonomous driving perception fusion system based on spatiotemporal feature difference and multipath error propagation. Figure 3 A schematic diagram illustrating the implementation of depth estimation in an embodiment of this application; Figure 4A schematic diagram of the structure of a target sensing device based on feature difference and error propagation provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0011] To further illustrate the technical means and effects adopted by this application to achieve the intended inventive objective, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a target perception method based on feature difference and error propagation proposed in this application. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined from any suitable form.

[0012] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0013] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0014] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0015] This application provides a target perception method based on feature difference and error propagation, such as... Figure 1 As shown, this can be achieved through the following steps: Step S110: Use the feature difference space to perform pixel-level comparison and residual extraction on the features of frame t and frame t-1 to obtain feature difference information, where t is an integer greater than or equal to 1. During implementation, motion residuals can be extracted by comparing the image features of two consecutive frames (t and t-1) at the pixel level. This is known as dynamic change detection. By utilizing feature differences rather than direct pixel differences, interference from illumination changes and noise can be suppressed, highlighting the feature differences of moving targets.

[0016] Here, the feature difference space may employ convolutional neural networks (such as ResNet) to extract multi-scale features, and combine difference operators (such as Sobel) or attention mechanisms to enhance the response of the moving region.

[0017] Step S120: Using the vehicle attitude information as a geometric constraint, a depth estimation network is used to determine the distance of the target in the image based on the feature difference information to obtain a two-dimensional environmental depth map. Target detection is then performed based on the two-dimensional environmental depth map to obtain a first detection result. During implementation, the original image undergoes distortion correction (such as fisheye correction), white balance adjustment, and dynamic range compression to generate a standard RGB image. Simultaneously, vehicle attitude information (such as speed, acceleration, and yaw angle) is acquired to provide input for subsequent geometric constraints. Vehicle attitude information (such as IMU, GPS, and wheel speed sensors) is used as geometric constraints and combined with depth estimation networks (such as Monodepth2 and PackNet) to generate a two-dimensional depth map.

[0018] Geometric constraint corrections are applied to features by incorporating vehicle attitude information (such as speed and steering angle), for example, by aligning feature maps of consecutive frames through affine transformations to reduce feature offsets caused by camera motion.

[0019] Here, pose information provides camera motion parameters, and the scale consistency of depth estimation is constrained by structure of motion reconstruction (SfM) or visual odometry (VO), avoiding the scale ambiguity problem of traditional single depth estimation. The depth map is then used for object detection (such as YOLO, Faster R-CNN) to obtain the first detection result (location, category, etc.).

[0020] Step S130: Map the two-dimensional environment depth map to three-dimensional space to generate a pseudo point cloud map, and perform target detection on the pseudo point cloud map to obtain a second detection result; During implementation, a 2D depth map can be mapped to 3D space to generate a pseudo-point cloud. This pseudo-point cloud can then be used for 3D target detection via a point cloud processing network (such as PointNet or VoxelNet) to obtain a second detection result. The advantage of pseudo-point clouds is that they preserve spatial geometric information, making them suitable for detecting targets in 3D space (such as vehicles and pedestrians). Furthermore, they are compatible with LiDAR point cloud processing frameworks, reducing the engineering complexity of multi-sensor fusion.

[0021] Step S140: Use weighted confidence to fuse the first detection result and the second detection result to determine the target detection result, wherein the target detection result includes at least the target's position, velocity, attitude and category information; During implementation, the first (two-dimensional) detection results and the second (three-dimensional) detection results are weighted and fused. The weights can be dynamically adjusted based on detection confidence, target category (e.g., dynamic / static), and scene complexity (e.g., congested / open). For example, in high-speed scenes, three-dimensional detection results may be given higher weight to utilize spatial information; in low-light scenes, two-dimensional detection results may be more reliable.

[0022] Step S150: During the target detection process, the weight distribution of the feature difference space and the depth estimation network is adjusted based on the target detection error and the feature detection error. The target detection error is obtained by comparing the first detection result and the second detection result, and the feature detection error is obtained by comparing the predicted feature distribution and the actual feature distribution. The predicted feature distribution is the predicted feature distribution of the t-th frame image based on the two-dimensional environment depth map, vehicle pose information and the features of the t-1 frame image. The features of the t-th frame image are then determined as the actual feature distribution.

[0023] During implementation, the target detection error is calculated by comparing the first and second detection results, and the feature detection error is calculated by comparing the predicted feature distribution with the actual feature distribution. Both factors jointly drive the weight adjustment of the feature difference space and the depth estimation network, forming a closed-loop optimization of error, weight, and performance. The calculated detection error signal is injected into the "differential propagation" module, and the weight distribution of the "feature difference space" is readjusted through the feedback path. This closed-loop mechanism ensures that the system can correct perception biases caused by occlusion, sudden changes in illumination, etc., in real time, adapting to dynamic environmental changes.

[0024] In this embodiment, a closed-loop perception system is constructed from 2D to 3D and from static to dynamic modes through feature differentiation, depth estimation, pseudo-point cloud fusion, and adaptive error adjustment. This system boasts high robustness, adaptability, and multimodal collaboration advantages, making it suitable for scenarios requiring precise target perception, such as autonomous driving and intelligent monitoring. It integrates 2D image features, 3D point cloud information, and vehicle posture data, overcoming the limitations of single-modal perception (such as the scale problem of monocular vision and the sparsity problem of point clouds). Through dual feedback of detection error and feature error, online adjustment of network weights is achieved, improving the system's adaptability to environmental changes (such as changes in illumination and target occlusion). Compared to LiDAR, pseudo-point clouds are generated based on visual data, are low-cost, and compatible with existing visual algorithms, making them suitable for cost-sensitive scenarios such as autonomous driving and robotics. Robust target detection (position, velocity, and attitude) is achieved in dynamic traffic scenarios, supporting path planning and obstacle avoidance. Multi-target tracking and behavior analysis are implemented in complex scenarios (such as densely populated areas). 3D spatial perception and dynamic target avoidance are achieved in unknown environments.

[0025] This application embodiment also provides a method for obtaining image features of frame t and image features of frame t-1, which can be achieved through the following steps: Step 1: Acquire raw image data of the current moment using the vehicle-mounted camera device; Here, a high dynamic range (HDR) vehicle camera can be used, which supports a wide dynamic range of over 120dB to adapt to extreme lighting scenarios such as tunnel entrances and exits and day-night cycles.

[0026] During the process, the original image undergoes distortion correction (such as fisheye correction), white balance adjustment, dynamic range compression, and other operations to generate a standard RGB image.

[0027] Step 2: Perform convolution operations on the original image using the backbone network to extract high-dimensional semantic features; Here, a lightweight backbone network (such as MobileNetV3 or EfficientNet-Lite) can be used to balance accuracy and real-time performance, or a pre-trained large model (such as ResNet-101 or ViT) can be used to transfer knowledge distillation to the automotive scenario. The network contains multiple convolutional layers, activation functions (such as ReLU and Swish), and pooling layers to progressively extract features from low-level textures to high-level semantics.

[0028] In some embodiments, shallow (edge, texture) and deep (semantic, context) features can be fused through a Feature Pyramid Network (FPN) to enhance the detection capability of small targets (such as pedestrians in the distance).

[0029] In some embodiments, spatial attention (such as CBAM) or channel attention (such as SENet) can be introduced to suppress background noise and enhance the feature response of the moving target region.

[0030] In some embodiments, the weights of the convolution kernel can be dynamically adjusted according to the complexity of the scene (such as congestion / high speed) to improve the adaptability of feature extraction.

[0031] Step 3: Perform feature fusion on the high-dimensional semantic features and store them as the image features of frame t and the image features of frame t-1, respectively.

[0032] During implementation, the high-dimensional features of the current frame and the features of the previous frame can be spatiotemporally fused using a recurrent neural network (such as ConvLSTM) or optical flow estimation to capture the target's motion trajectory and velocity information.

[0033] Here, a ring buffer can be used to store the feature data of the most recent N frames (e.g., 10 frames), supporting efficient read / write and memory management; The features of frame t and frame t-1 are stored independently, while retaining the timestamp and attitude information to ensure the traceability and consistency of subsequent feature difference calculations.

[0034] In this embodiment, a foundational data layer for dynamic scene perception is constructed through vehicle-mounted camera acquisition, backbone network feature extraction, and multi-scale feature fusion. This provides high-quality input features for the feature difference space, directly impacting the effectiveness of feature difference and the accuracy of depth estimation. Accurate feature extraction and fusion can improve the detection rate of moving targets and reduce the false detection rate.

[0035] In some embodiments, the vehicle attitude information includes vehicle distance data and vehicle attitude data. This application also provides a method for obtaining vehicle attitude information, which can be achieved through the following steps: Step A: Use the vehicle-mounted activated radar to acquire the vehicle distance data; Here, millimeter-wave radar (such as 77GHz / 24GHz) or lidar (such as 1550nm) can be used as ranging sensors. Millimeter-wave radar calculates the target distance, velocity, and azimuth by transmitting frequency-modulated continuous wave (FMCW) and receiving the echo signal, using the Doppler effect. It has advantages such as resistance to rain and fog, strong penetration, and low cost. Lidar measures distance using time-of-flight (ToF) or phase difference, with accuracy down to the centimeter level, but it is more expensive and susceptible to interference from dust, rain, and snow.

[0036] In some embodiments, multiple radars (such as forward long-range radar + lateral short-range radar) are maliciously deployed around the vehicle, and 360° coverage without blind spots is achieved through data fusion (such as Kalman filtering), thereby improving the integrity and reliability of vehicle distance measurement.

[0037] In some embodiments, the radar can be periodically calibrated online (e.g., by passing known targets such as road signs or cones) to compensate for ranging deviations caused by temperature changes or mechanical vibrations, ensuring long-term accuracy stability.

[0038] Step B: Obtain the vehicle's attitude data using the vehicle-mounted posture sensor.

[0039] Here, the vehicle's attitude information (such as vehicle speed and yaw angle) is combined with feature difference information to provide kinematic constraints for the depth estimation network, reduce the scale ambiguity problem in depth estimation, and improve the accuracy of the two-dimensional depth map.

[0040] During implementation, the aforementioned vehicle attitude information can be obtained using sensors installed on the vehicle that are capable of acquiring position and posture.

[0041] In some embodiments, the vehicle-mounted posture sensor includes an inertial measurement unit and a wheel speedometer; step B above can be achieved through the following process: The inertial measurement unit and the wheel speed meter are used to acquire the vehicle's acceleration, speed and steering angle data.

[0042] Here, the core parameters of the inertial measurement unit include a three-axis accelerometer (measuring linear acceleration) and a three-axis gyroscope (measuring angular velocity). Through integration, the vehicle's attitude angles (pitch, roll, yaw), velocity, and position can be derived.

[0043] During implementation, zero-bias stability optimization (such as temperature compensation), random walk suppression (such as Kalman filtering), and sensor fusion (such as fusion with wheel speedometer and GPS data) can be adopted to reduce the impact of integral drift on long-term positioning.

[0044] In autonomous driving scenarios, tactical-grade IMUs (such as fiber optic gyroscopes) can achieve zero-bias stability of 0.01° / h, meeting the requirements for high-precision positioning (such as centimeter-level positioning).

[0045] Wheel speed gauges measure wheel speed (e.g., Hall effect sensors, encoders) and combine this with vehicle dynamics models (e.g., Ackermann steering models) to calculate the vehicle's longitudinal speed, lateral slip angle, and steering angle.

[0046] In wet or slippery road conditions or sharp turns, the robustness of attitude estimation can be improved by jointly calibrating the wheel speed sensor and the IMU (e.g., by estimating slip ratio). This can compensate for speed measurement errors caused by tire slippage.

[0047] In some embodiments, extended Kalman filtering (EKF) or unscented Kalman filtering (UKF) can be used to fuse the high-frequency updates (200-500Hz) of the IMU with the low-frequency updates (10-50Hz) of the wheel speedometer to achieve joint estimation of attitude, velocity, and position.

[0048] In complex scenarios (such as urban roads), factor graph optimization (such as the GTSAM library) can be used to fuse multi-frame IMU data, wheel speed measurement data, and GPS positioning results to improve global consistency and reduce the accumulation of local errors.

[0049] During implementation, hardware triggering (such as PPS signal) or software timestamp alignment (such as NTP protocol) can be used to ensure that the time error of radar data, IMU data, wheel speedometer data and image data is ≤1ms, so as to avoid perception deviation caused by time delay.

[0050] By calibrating sensor extrinsic parameters (such as the Zhang Zhengyou calibration method), the relative positions and attitudes of radar, IMU, and camera in the vehicle coordinate system are determined, enabling the fusion of multi-source data in a unified coordinate system (such as the world coordinate system or the vehicle coordinate system).

[0051] In this embodiment, a high-precision and robust vehicle attitude perception system is constructed by fusing radar ranging and pose sensors, combined with spatiotemporal synchronization and multi-source data alignment technologies. This system focuses on sensor selection, error suppression, multi-source fusion algorithms, and hardware acceleration, ultimately supporting the efficient operation of depth estimation, pseudo-point cloud generation, and error propagation optimization in target perception methods. It is suitable for scenarios requiring precise vehicle state perception, such as autonomous driving and intelligent transportation.

[0052] This application also provides a method for determining target detection error, which can be achieved through the following steps: Step 1: Compare the consistency between the first detection result and the second detection result; During implementation, the Intersection over Union (IoU) ratio can be used to evaluate the spatial overlap between the first (two-dimensional) detection result and the second (three-dimensional) detection result. For example, when the IoU of the two detection boxes is ≥0.5, they are judged as the same target. The class probability similarity (such as KL divergence) is combined to verify the class consistency and avoid misclassifying "vehicles" as "pedestrians".

[0053] In some embodiments, the IoU threshold can be dynamically adjusted according to the complexity of the scenario (e.g., congestion / high speed) (e.g., reduced to 0.3 for congestion scenarios and increased to 0.7 for high speed scenarios) to balance the false negative rate and false positive rate.

[0054] In some embodiments, the Hungarian algorithm or the KM algorithm can be used to achieve optimal matching among multiple targets and solve the matching problem when the number of detection results is inconsistent (e.g., three targets are detected in two dimensions and two targets are detected in three dimensions).

[0055] Step 2: Calculate the confidence residual based on the consistency of the results; Here, the confidence residual is the difference between the two-dimensional detection confidence and the three-dimensional detection confidence for the same target. The confidence residual ΔC can be calculated using the following formula (1): ΔC=C 2D C 3D (1); Among them, C 2D For two-dimensional detection confidence, C 3D This represents the confidence level for two-dimensional detection.

[0056] If the target is detected by only a single mode (e.g., detected in two dimensions but not in three dimensions), the residual is calculated based on the following formula (2): ΔC=C 2D θ(2); Where θ is the confidence threshold.

[0057] Step 3: Use the confidence residual as the target detection error.

[0058] During implementation, the confidence residual is input as the target detection error to the error propagation module in step S150, and together with the feature detection error (such as the difference between the predicted feature distribution and the actual feature distribution), it drives the weight adjustment of the feature difference space and the depth estimation network. For example, if the two-dimensional and three-dimensional detection residuals of a certain type of target (such as a pedestrian in the distance) remain large, the convolutional kernel weights of the feature difference space are adjusted to enhance the feature extraction capability for this type of target.

[0059] In this embodiment, a dynamic fusion and self-optimization mechanism for two-dimensional and three-dimensional detection results is constructed through result consistency comparison and confidence residual calculation. This mechanism focuses on quantification index design, residual calculation strategies, error propagation mechanisms, and adaptive weight adjustment, ultimately supporting the high-precision and robust operation of the target perception system in scenarios such as autonomous driving and intelligent monitoring, forming a closed-loop perception system from detection to optimization.

[0060] This application also provides a method for determining feature detection error, which can be achieved through the following steps: Step A: Using the predicted depth of the two-dimensional environment depth map and the vehicle attitude information, transform the features of the t-1 frame image to the predicted feature distribution of the t frame image to obtain the predicted feature distribution; Here, the predicted feature distribution is based on a two-dimensional environmental depth map. Generative models (such as GANs and VAEs) or regression models (such as support vector regression and random forests) are used to predict features from the depth map. For example, the intermediate feature layer of a depth estimation network (such as Monodepth2) is used to extract high-dimensional semantic features through a convolutional neural network. The spatial attention mechanism is combined to enhance the feature response of the moving target region, and the predicted feature distribution of the t-frame image is generated based on the features of the t-1 frame image.

[0061] In some embodiments, optical flow tracing or motion trajectory prediction (such as Kalman filtering) can be introduced to verify the temporal consistency of the predicted features and avoid prediction bias caused by dynamic target occlusion or motion blur.

[0062] In some embodiments, shallow (edge, texture) and deep (semantic, context) features are fused through a Feature Pyramid Network (FPN) to improve the ability of the predicted feature distribution to detect small targets (such as pedestrians in the distance).

[0063] Step B: Determine the actual feature distribution as the image features of frame t; Step C: Compare the predicted feature distribution with the actual feature distribution to identify perceived noise; During implementation, the difference between the predicted feature distribution and the actual feature distribution can be identified through structural similarity index (SSIM), mean squared error (MSE), or KL divergence quantification, thus identifying perceived noise. For example, when the SSIM value is below a threshold (e.g., 0.7), it is determined that there is significant perceived noise, and further analysis of the noise source (e.g., sensor error, algorithm bias, environmental interference) is required.

[0064] In some embodiments, vehicle attitude information and feature difference information can be combined to locate the source of noise. For example, if the noise is concentrated in a specific area (such as around the vehicle), it may be due to radar ranging error or IMU integral drift; if the noise fluctuates over time, it may be due to environmental interference (such as rain, fog, or changes in lighting).

[0065] In some embodiments, Z-score or isolated forest algorithms can be used to detect outliers in the feature distribution to avoid misjudgments or missed detections due to noise.

[0066] In some embodiments, the implicit logic is quantified by the following formula (1): Errorfeat = ||Feature [t]–Warp(Feature[t-1], Pose, Depth)||(1); Here, Errorfeat represents the feature detection error, which is a measure of difference in the feature space and indicates the degree of mismatch between the model's predicted features and the true features.

[0067] Feature[t] represents the feature of frame t.

[0068] Warp(Feature[t-1], Pose, Depth) represents transforming the features (Feature[t-1]) of the previous frame to their spatial position in the current frame using pose (Pose) and depth (Depth), achieving spatiotemporal alignment. Pose can be obtained by combining vehicle pose information with feature difference information.

[0069] ||·|| is a mathematical tool for measuring feature differences, usually using the L2 norm (Euclidean distance), which calculates the square root of the sum of the squares of the feature vectors.

[0070] In the implementation process, in order to calculate the feature detection error, the feature detection error module needs to obtain the original [t] frame feature map without differential processing as the "true value" and perform precise pixel-level or feature point-level residual comparison.

[0071] Quantization logic: The features from the previous frame are projected onto the current frame using "motion pose" and "depth information". If the projected features highly overlap with the features actually extracted in the current frame, it indicates that the contributions of the depth and pose branches are accurate.

[0072] Step D: Use the perceived noise as the feature detection error.

[0073] Here, perceived noise is input as feature detection error to the error propagation module in step S150, and together with target detection error (such as confidence residual), it drives the weight adjustment of the feature difference space and the depth estimation network. For example, if the predicted feature distribution of a certain type of target (such as a pedestrian in the distance) differs significantly from the actual feature distribution, the convolution kernel weights of the feature difference space are adjusted to enhance the feature extraction capability for that type of target; if the noise originates from sensor error, the loss function weights of the depth estimation network are adjusted to suppress the influence of noise.

[0074] In some embodiments, gradient descent, reinforcement learning, or genetic algorithms can be used to optimize network weights. The gradient of the error with respect to the network parameters is calculated through backpropagation, and the parameters of the feature extraction network (such as ResNet) or depth estimation network (such as Monodepth2) are adjusted. Simultaneously, an online learning mechanism is incorporated to dynamically update model parameters based on real-time detection errors, improving the system's adaptability to environmental changes.

[0075] In this embodiment, by comparing the predicted and actual feature distributions, perceived noise is quantified and used as feature detection error. This drives the weight adjustment of the feature difference space and the depth estimation network, forming a closed-loop optimization mechanism. This achieves a focus on prediction modeling, actual feature calculation, noise quantification, and error propagation, ultimately supporting the high-precision and robust operation of the target perception system in scenarios such as autonomous driving and intelligent monitoring, forming a complete technical chain from feature extraction to error optimization.

[0076] In some embodiments, the step S150 above, "adjusting the weight distribution of the feature difference space and the depth estimation network based on the target detection error and the feature detection error," can be achieved through the following steps: Step 151: Input the target detection error and the feature detection error into the differential propagation network for encoding to obtain a dynamic differential propagation signal; Here, a differential propagation network with an encoder-decoder architecture can be used. The encoder consists of multiple fully connected layers or convolutional layers (such as 1×1 convolutions), with inputs being object detection errors (such as confidence residuals) and feature detection errors (such as perceptual noise), and output being a dynamic differential propagation signal. The decoder recovers the spatial / temporal dimensions of the signal through skip connections or upsampling, ensuring that the signal can be accurately propagated to the feature difference space and the parameters of each layer of the depth estimation network.

[0077] In some embodiments, the two errors can be integrated through weighted fusion or attention mechanisms (such as multi-head attention in Transformer) to highlight key error information (such as detection bias of high-confidence targets).

[0078] In some embodiments, activation functions (such as ReLU, Swish) or gating mechanisms (such as the forget gate of LSTM) can be used to perform nonlinear transformation on the error signal to generate a time-dependent dynamic propagation signal that adapts to environmental changes (such as sudden changes in illumination or target occlusion).

[0079] Step 152: Use the dynamic differential propagation signal to reverse correct the parameters of the feature difference space and the depth estimation network.

[0080] Here, the gradient of the dynamic differential propagation signal can be calculated using backpropagation algorithms (such as the chain rule) to determine the gradient between the feature difference space (e.g., convolutional kernel weights) and the depth estimation network (e.g., encoder-decoder parameters of Monodepth2). For example, the direction and magnitude of weight adjustment can be determined by calculating the partial derivative of the signal with respect to the convolutional kernel weights.

[0081] In some embodiments, gradient descent (such as Adam, SGD) or second-order optimization algorithms (such as L-BFGS) can be used, combined with learning rate scheduling (such as cosine annealing) to dynamically adjust the optimization step size, balancing convergence speed and accuracy.

[0082] In some embodiments, L2 regularization or weight pruning can be introduced to prevent overfitting; geometric constraints (such as vehicle posture information) can be combined to limit the range of weight adjustment to ensure that parameter updates conform to physical laws (such as kinematic models).

[0083] During implementation, the target detection error and feature detection error form a closed-loop feedback through the differential propagation network, driving the dynamic adjustment of the weights of the feature difference space and the depth estimation network. For example, if the detection error of a certain type of target (such as a pedestrian in the distance) remains large, the weights of the convolutional kernels corresponding to that type of target are increased through signal backpropagation to improve feature extraction capability.

[0084] In some embodiments, the optimization strategy can be dynamically adjusted based on the complexity of the scenario (such as congestion / highway, weather conditions). For example, in congested scenarios, the weight adjustment range for nearby targets can be increased, while in high-speed scenarios, the weight adjustment range for distant targets can be increased.

[0085] In this embodiment, an error-driven dynamic weight optimization mechanism is constructed through the encoding of a differential propagation network and the backpropagation of dynamic signals. This mechanism encompasses network architecture design, error fusion strategies, backpropagation algorithms, and adaptive optimization, enabling high-precision and robust operation of the target perception system in scenarios such as autonomous driving and intelligent monitoring, forming a closed-loop perception system from error detection to parameter optimization.

[0086] As a core component of autonomous driving and intelligent assisted driving, the environmental perception system directly determines the accuracy of a vehicle's localization, ranging, and behavior prediction of surrounding obstacles. In existing technological systems, mainstream perception solutions are mostly based on deep neural networks (DNNs) for multi-task learning. Their typical software and algorithm architecture usually consists of an image feature extraction module, a spatial transformation module, an object detection branch, and a depth estimation branch. In these solutions, the image feature extraction module often uses ResNet or a hierarchical feature pyramid network (FPN) to obtain high-dimensional semantic features by performing convolution operations on single or multiple consecutive images captured by a camera. The spatial transformation module utilizes geometric projection principles or attention mechanisms to map the image spatial features to three-dimensional space or a bird's-eye view (BEV) space. Finally, different detection heads output the location, category, and environmental depth information of obstacles. In actual operation, existing algorithms first acquire raw images through sensors, encode features using a pre-trained model, then achieve temporal fusion through simple feature concatenation or weighted averaging, and finally output the perception results.

[0087] However, in-depth research and feedback from practical applications have revealed significant technical limitations in handling complex and dynamic traffic scenarios. Firstly, regarding temporal feature fusion, existing technologies primarily focus on stacking static features, lacking explicit modeling of the "feature difference" between consecutive frames. Due to the lack of dedicated feature difference space processing, the system struggles to accurately capture subtle motion trends of obstacles, leading to severe temporal jitter and edge blurring in perception results when the target moves at high speed or the vehicle experiences severe vibrations. Secondly, current depth estimation and motion attitude analysis are mostly independent parallel branches, lacking deep geometric coupling and logical constraints. This prevents the system from compensating for the impact of attitude changes on depth features in real time when the vehicle's attitude frequently changes (such as acceleration pitch and cornering roll), resulting in a significant decrease in ranging accuracy.

[0088] The most critical drawback lies in the fact that most existing technologies follow an open-loop "perception-output" logic, lacking an effective closed-loop error correction and feedback mechanism. In traditional algorithm flows, if the target detection module produces a missed detection or a false detection at time t, this error information accumulates over time and cannot be reversed to the feature extraction layer or used for cross-frame logical offsetting. Furthermore, the conversion process from two-dimensional image space to three-dimensional point cloud space in existing technologies is often accompanied by severe information loss and spatial distortion, lacking a mechanism to convert detection errors into "differential propagation signals" and feed them back to the underlying feature space for adaptive optimization. This open-loop structure results in poor robustness of the system when facing occlusion, extreme lighting, or sensor noise, making it difficult to maintain long-term temporal consistency of perception results. In summary, developing an autonomous driving perception system that can deeply integrate spatiotemporal features, possess motion attitude decoupling capabilities, and introduce a closed-loop differential propagation correction mechanism has become a critical technical problem urgently needing to be solved in this field.

[0089] In the existing technological path of autonomous driving perception, although the introduction of deep learning models has greatly improved the target recognition capability in complex environments, the following bottlenecks still urgently need to be addressed in practical engineering applications: 1. Lack of Temporal Dynamic Feature Modeling: Most existing perception algorithms focus on spatial feature extraction from single-frame images. Even when multi-frame processing is involved, they often employ simple channel stitching or implicit fusion using recurrent neural networks (RNNs). This approach lacks explicit modeling of pixel-level and semantic-level "changes" between adjacent frames. Under high-speed driving or complex dynamic backgrounds, the system struggles to extract minute target motion features from massive amounts of background data, leading to delayed predictions of obstacle movement trends and susceptibility to environmental noise interference.

[0090] 2. Decoupling of Geometric Constraints and Pose Awareness: Traditional depth estimation networks and vehicle pose awareness are usually separate branches. However, during vehicle acceleration, braking, or cornering, the camera's pitch and roll angles shift instantaneously due to suspension vibrations. Existing technologies often ignore the impact of this physical pose change on visual depth features, resulting in significant vertical positioning errors in the generated pseudo-point cloud data, severely affecting the accuracy of subsequent target fusion.

[0091] 3. Error Accumulation Due to Open-Loop Execution: Current perception workflows generally present a unidirectional topology, i.e., from image input to feature extraction, and then to detection result output. This "open-loop" architecture lacks effective error correction feedback. When the system generates a perception error at time t due to occlusion or drastic changes in lighting, this error cannot be converted into a correction signal and fed back to the feature layer. This not only causes "flickering" and jumps in the perception results on the time axis, but also prevents the system from using known error experience for adaptive enhancement in subsequent frame processing.

[0092] 4. Weighting blind spots in multimodal feature fusion: In the process of converting 2D image features into 3D point cloud features, existing technologies lack a dynamic evaluation mechanism for feature confidence. Because the error contribution from different branches (such as the depth branch and the detection branch) cannot be quantified, the final target fusion module often resorts to simple heuristic weighting when dealing with conflicting data (such as depth estimation indicating a wall ahead, but the detection network not detecting the target), making it difficult to make optimal decisions.

[0093] This application provides an autonomous driving perception fusion system based on spatiotemporal feature difference and multipath error propagation, such as Figure 2 As shown, it includes: feature difference space 21, depth estimation network 22, motion pose network 23, feature detection error 24, difference propagation 25, target detection (point cloud) 26, target detection 27, target fusion 28, and target detection error 29, among which, The depth estimation network 22 (corresponding output: stable depth information) is located in the upper green box in the figure.

[0094] Input: The input consists of the feature difference information output from the feature difference space 21 at time t, and the vehicle distance and attitude information.

[0095] Function: Calculate the distance between the object and the vehicle based on image features.

[0096] When the vehicle bumps or turns, the footage captured by the camera will show vertical shaking or side tilting. Without introducing "vehicle attitude parameters," the model will mistakenly interpret this as the road surface or objects bouncing up and down. Through correction, the camera's own motion interference can be eliminated, and accurate relative depth can be output.

[0097] Motion Pose Network 23 (corresponding output: spatial coordinates / pseudo-point cloud) is located in the blue middle box in the figure.

[0098] Input: Vehicle distance and attitude.

[0099] Function: To determine the precise state of a vehicle in three-dimensional space.

[0100] The motion pose network 23 does not work in isolation; it receives input in two dimensions: Dimension A: Physical Priors (from the "Vehicle Distance and Attitude" module on the left) Data content: Raw data (such as acceleration, angular velocity, and steering angle) provided by the inertial measurement unit (IMU) and wheel speedometer.

[0101] Processing method: The above data is used as "initial values" or "constraints" and input into the motion attitude network to determine the approximate displacement of the vehicle. This helps the motion attitude network maintain the reasonableness of motion estimation even when visual features are blurred (such as strong light or untextured road surfaces).

[0102] Dimension B: Visual observation (from "feature difference space") Data content: residual or optical flow information between the feature maps at time t and t-1.

[0103] Processing method: The motion pose network 23 analyzes the overall displacement of pixels / features in the image through convolutional layers. If all points in the background are moving to the left, the motion pose network can infer that the vehicle is turning to the right.

[0104] The motion pose network 23 can decouple the above input and decompose it into two parts: 1. Vehicle compensation: Identify which changes are "pseudo-motions" caused by vehicle bumps and steering.

[0105] 2. Target Extraction: After eliminating the motion of the vehicle, identify which changes are caused by real dynamic targets (such as pedestrians crossing the road).

[0106] The final output of the motion attitude network 23 is the real-time calculation of the target's motion vector, heading angle, and a set of transformation matrices describing the changes of the vehicle from time t-1 to time t, which is usually called 6-DoF Pose (six degrees of freedom displacement parameters).

[0107] This application provides an autonomous driving perception fusion method based on spatiotemporal feature difference and multipath error propagation, aiming to comprehensively improve the robustness and accuracy of perception through a closed-loop feedback mechanism and explicit motion modeling. It can be implemented through the following five stages: Phase 1: Image feature extraction and temporal state management.

[0108] 1. Multi-view image acquisition: The system acquires multiple raw image data (image space) at the current time [t] through the vehicle-mounted sensor array.

[0109] 2. Multi-scale feature extraction: The backbone network is used to perform convolution operations on the original image to extract high-dimensional semantic features.

[0110] 3. Temporal feature storage: After feature fusion, the extracted features are stored as the feature space H(t) / W(t) of the current frame, and the feature space H(t-1) / W(t-1) of the previous time step is also saved to provide a basis for subsequent temporal comparison.

[0111] Phase 2: Spatiotemporal feature difference and dynamic modeling.

[0112] 4. Feature Alignment and Difference Calculation: Operator operations are performed on the features of frame t-1 and frame t to construct a feature difference space21. This space aims to explicitly highlight the residual information of moving objects in the scene.

[0113] 5. Multi-source data input: The physical data of "vehicle distance and attitude" synchronously collected by the vehicle sensors is input into the depth estimation network 22 and the motion attitude network 23 as geometric constraints.

[0114] Phase 3: Parallel Perception Task Inference.

[0115] 6. Depth estimation network collaboration: The depth estimation network 22 simultaneously receives "feature difference space" information and "vehicle distance and attitude" information, and outputs a dense environmental depth map to achieve high-precision ranging.

[0116] The information processing procedure of the depth estimation network 22 is as follows: Input stage: Multi-source information fusion.

[0117] The feature space at time t provides semantic information for the current frame, including the object's category, shape, and position P(i) on the image plane.

[0118] Characteristic difference space: Provides the change between time t and t-1. This difference signal includes the parallax caused by vehicle movement.

[0119] Motion posture network signals: from Figure 2 As can be seen, a line connecting the collected vehicle distance and attitude information points to the depth estimation network 22. This indicates that the vehicle's real-time attitude (bumps, steering) is used to correct features, ensuring that the depth estimation network 22 does not mistake the vehicle's swaying for a change in depth.

[0120] Processing stage: geometric modeling and parameter mapping.

[0121] The depth estimation network 22 internally simulates the geometry of pinhole imaging: Spatial projection calculation: The depth estimation network learns the mathematical mapping between image plane coordinates X(i) and camera center O(c) and focal length f through convolutional layers, such as... Figure 3 As shown.

[0122] Feature Differential Correction: The differential propagation signal (orange line at the bottom of the figure) is injected into the network to enhance attention to depth-sensitive features through weight adjustment. For example, during turbulence, the differential propagation signal can guide the network to refer more to the static features at time t, rather than the differential features that may cause bias, thereby outputting stable depth information.

[0123] Output stage: Generate depth map and point cloud base.

[0124] After processing, the information output by the network directly drives subsequent modules: Depth information output: Provides the Z-axis distance for each feature point.

[0125] Generating a pseudo-point cloud: The depth output is combined with the features at time t and input into the object detection (point cloud) module. At this point, as shown... Figure 3 As shown, the point P(i) on the image plane is projected back to the spatial point P(c) to construct the three-dimensional spatial coordinates.

[0126] "Pseudo-point clouds" essentially project 2D image features into 3D space. Without compensation when the vehicle turns, the generated point cloud will become distorted. This network uses real-time pose parameters to perform spatial geometric correction on the feature map (similar to a coordinate transformation), ensuring that the generated point cloud is spatially logically continuous and stable.

[0127] 7. Motion Attitude Analysis: The motion attitude network 23 calculates the target's motion vector, heading angle, and the vehicle's 6-DOF attitude in real time based on continuous feature changes.

[0128] 8. Target Detection (Point Cloud Projection): Path A (point cloud detection), i.e. Figure 2 The target detection (point cloud) 26 shown: Map the features of the environment depth map to the three-dimensional space to generate a pseudo point cloud and perform target detection based on the point cloud.

[0129] Path B (traditional detection), i.e. Figure 2 The target detection shown in Figure 27: Perform conventional obstacle bounding box regression and classification on a two-dimensional feature map (environment depth map).

[0130] Phase 4: Error detection and closed-loop differential propagation (core innovative steps).

[0131] 9. Multidimensional error analysis: like Figure 2 Feature detection error 24 shown: Compare the predicted and actual feature distributions in the feature difference space to identify perceived noise.

[0132] Figure 2 The “feature detection error” module in the architecture is quantified by the implicit logic represented by the following formula (1): Errorfeat = ||Feature [t]–Warp(Feature[t-1], Pose, Depth)||(1); Here, Errorfeat represents the feature detection error, which is a measure of difference in the feature space and indicates the degree of mismatch between the model's predicted features and the true features.

[0133] Feature[t] represents the feature of frame t.

[0134] Warp(Feature[t-1], Pose, Depth) represents transforming the features (Feature[t-1]) of the previous frame to their spatial position in the current frame using pose (Pose) and depth (Depth), achieving spatiotemporal alignment. Pose can be obtained by combining vehicle pose information with feature difference information.

[0135] ||·|| is a mathematical tool for measuring feature differences, usually using the L2 norm (Euclidean distance), which calculates the square root of the sum of the squares of the feature vectors.

[0136] In the implementation process, in order to calculate the feature detection error, the feature detection error module needs to obtain the original [t] frame feature map without differential processing as the "true value" and perform precise pixel-level or feature point-level residual comparison.

[0137] Quantization logic: The features from the previous frame are projected onto the current frame using "motion pose" and "depth information". If the projected features highly overlap with the features actually extracted in the current frame, it indicates that the contributions of the depth and pose branches are accurate.

[0138] Contribution Decomposition: The "Differential Propagation" module automatically identifies which branch (whether it's inaccurate depth prediction or pose calculation deviation) caused the final fusion conflict by calculating the gradient derror / dweight. This is an automated, gradient-based "contribution metric." Here, derror is short for "derivative of error," and dweight is short for "derivative of weight," which actually refers to the gradient of the error with respect to the weights. Calculating the gradient derror / dweight is equivalent to calculating the derivative of the error with respect to the weights (gradient), which is crucial in the backpropagation algorithm. According to the principle of gradient descent, adjusting the weights in the opposite direction of the gradient (because the gradient direction is where the error increases the fastest, while the opposite direction is where the error decreases the fastest) can gradually reduce the error, making the neural network's predictions closer to the true values, thus optimizing the network's training.

[0139] Cross-modal consistency verification: When the detection network does not detect a target, but the depth estimation shows a wall: the system can view the "Target Detection Error" module. If the point cloud detection branch detects an obstacle, but the image branch does not, differential propagation can generate an extremely high penalty term. The magnitude of this penalty term directly corresponds to the "contribution" of that branch to the total error.

[0140] like Figure 2 The target detection error 29 is shown: compare the consistency of results from different detection paths (path A vs path B) and calculate the confidence residual.

[0141] 10. Differential Propagation Mechanism: The calculated feature detection error and target detection error are injected into the "differential propagation" module 25, and the weight distribution of the "feature difference space" is readjusted through the feedback path. This closed-loop mechanism ensures that the system can correct perception biases caused by occlusion, sudden changes in illumination, etc. in real time.

[0142] Figure 2 The orange dashed line represents the following: Using the "error signal" generated by the backend, parameters of all key frontend modules are collaboratively corrected across multiple levels, as follows: A. Front-end feature extraction and fusion module. Dashed line placement: Between image space and feature extraction.

[0143] Correction logic: If the backend finds that the feature detection error is large, it means that the extracted features are "not stable enough".

[0144] Correction objective: Adjust the weights of the convolutional kernels to make them more robust to changes in lighting and motion blur. Force the network to learn higher-level semantic features that are more "translation-invariant" along the time axis [t-1, t].

[0145] B. Depth estimation network. Dashed line points to the green box at the top center of the flowchart.

[0146] Correction logic: Depth information is the bridge connecting the image (2D) and the physical world (3D). If the 3D bounding box of the object detection is in the wrong position, it is usually because the depth prediction is off.

[0147] Correction objective: By using "object detection error" to inversely constrain depth weights, the sensitivity of depth to object scale and distance is optimized, achieving self-supervised depth refinement.

[0148] C. Motion Attitude Network. Dashed line landing point: the blue box in the lower middle of the flowchart.

[0149] Correction logic: The pose network is responsible for predicting the vehicle's R (rotation) and t (translation). If the features are not aligned, it is likely that the pose matrix has been calculated incorrectly.

[0150] Correction objective: To adjust network parameters to reduce the residual between the "predicted pose" and the "visually observed pose", thereby improving the accuracy of vehicle localization.

[0151] During implementation, features can be evaluated using dynamic masks or confidence maps. Error reversal evaluation method: The “Feature Detection Error” module outputs not just a numerical value, but usually an error heatmap of the same size as the feature map.

[0152] Low error region: High feature consistency, giving high confidence.

[0153] High error region: identified as sensor noise, occlusion or motion blur, and assigned a low confidence level.

[0154] Target-level confidence closed loop: When handling conflicts, the "Target Fusion" module can refer to the statistical characteristics fed back by the "Target Detection Error" module. If the error of a certain branch fluctuates drastically in the temporal performance of the past N frames, the system can automatically reduce the weight of that branch during fusion through differential propagation.

[0155] Phase 5: Multimodal target fusion output.

[0156] 11. For example Figure 2 The target fusion 28 shown: The fusion module collects various target data (range information) from point cloud detection (path A), traditional detection (path B) and error correction.

[0157] 12. Final target generation: Through weighted confidence fusion and temporal filtering, the final perception results (including the target's precise position, velocity, attitude, category, etc.) are output for use by the downstream planning and control module.

[0158] This application provides an autonomous driving perception fusion system and method based on spatiotemporal feature difference and multipath error propagation, aiming to comprehensively improve the robustness and accuracy of perception through closed-loop feedback mechanism and explicit motion modeling.

[0159] Specifically, the technical problem to be solved and the technical effects achieved by the present invention are as follows: 1. Establish an explicit feature difference modeling mechanism.

[0160] This application aims to address the problem of insensitivity to dynamic feature capture in existing technologies. By constructing a dedicated "feature difference space," pixel-level comparison and residual extraction are performed between the feature spaces at time t and time t-1. This mechanism aims to filter out redundant static backgrounds in the environment and significantly amplify the feature responses of dynamic obstacles, thereby achieving high-sensitivity capture of moving targets in complex backgrounds.

[0161] 2. Achieve deep coupling between vehicle motion posture and visual features.

[0162] To address ranging errors caused by attitude fluctuations during vehicle movement, the proposed technical challenge is how to utilize the vehicle's motion and pose network to provide dynamic calibration for the feature space. By incorporating real-time vehicle distance and attitude parameters, spatial geometric correction is performed on the feature map, ensuring that the system can still output stable depth information and spatial coordinates even when the vehicle is bumping or turning, thereby improving the spatial consistency of pseudo-point cloud generation.

[0163] 3. Construct a closed-loop correction system for perception based on differential propagation.

[0164] To address the error accumulation problem in the perception process, a monitoring module for "feature detection error" and "target detection error" is designed to capture output conflicts of each perception branch in real time. These error signals are encoded as "differential propagation signals" and injected back into the feature difference space and depth pose estimation network. This enables the system to have "self-learning and temporal compensation" capabilities, ensuring that failed detections in the previous frame can become optimization constraints in the next frame.

[0165] 4. Optimize high-fidelity conversion from image features to point cloud targets.

[0166] This paper aims to address the problem of information loss during multi-task fusion. By introducing a point cloud-based target detection path and combining it with a high-precision depth field output from a depth estimation network, two-dimensional feature maps are converted into voxel or point cloud representations with three-dimensional geometric attributes. Through a multi-dimensional target fusion module, visual semantic information and three-dimensional spatial geometric information are complemented, eliminating perceptual blind spots and improving the success rate of detecting occluded targets.

[0167] 5. Improve the temporal smoothness and anti-interference ability of the perception results.

[0168] By combining differential propagation and multipath error detection, detection fluctuations caused by sensor noise or momentary occlusion are eliminated. By maintaining a dynamically updated feature cache in time, the system can use spatiotemporal coherence to logically complete the data when faced with missing information in a single frame, ensuring that the target data output to the execution layer (such as the planning and control module) has extremely high stability and reliability.

[0169] The ultimate goal is to change the traditional "one-way, passive, and static" processing mode of autonomous driving perception systems. By explicitly defining feature differences, geometricizing attitude constraints, and closing the error propagation loop, an intelligent perception framework capable of sensing motion, understanding attitude, and self-correcting errors can be constructed. This not only significantly reduces the safety risks of autonomous vehicles in complex urban intersections, bumpy elevated roads, and scenarios with multiple obstacles, but also provides a more confident environmental model for subsequent decision-making and planning.

[0170] This application provides a target perception method apparatus based on feature difference and error propagation. Please refer to [link to relevant documentation]. Figure 4 The device 400 includes: The feature difference module 410 is used to perform pixel-level comparison and residual extraction on the features of frame t and frame t-1 using the feature difference space to obtain feature difference information, where t is an integer greater than or equal to 1. The first detection module 420 is used to use vehicle attitude information as geometric constraints, and to use a depth estimation network to determine the distance of the target in the image based on the feature difference information to obtain a two-dimensional environment depth map, and to perform target detection based on the two-dimensional environment depth map to obtain a first detection result. The second detection module 430 is used to map the two-dimensional environment depth map to three-dimensional space to generate a pseudo point cloud map, and to perform target detection on the pseudo point cloud map to obtain a second detection result. The result fusion module 440 is used to fuse the first detection result and the second detection result using weighted confidence to determine the target detection result, wherein the target detection result includes at least the target's position, velocity, attitude and category information; The weight adjustment module 450 is used to adjust the weight distribution of the feature difference space and the depth estimation network based on the target detection error and the feature detection error during the target detection process. The target detection error is obtained by comparing the first detection result and the second detection result, and the feature detection error is obtained by comparing the predicted feature distribution and the actual feature distribution. The predicted feature distribution is the predicted feature distribution of the t-th frame image based on the two-dimensional environment depth map, vehicle attitude information and the features of the t-1 frame image, and the features of the t-th frame image are determined as the actual feature distribution.

[0171] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. For example, as shown... Figure 5As shown, the computer device 500 includes: a memory 501, a processor 502, and a computer program 503 stored in the memory 501 and running on the processor 502, wherein when the processor 502 executes the computer program 503, the computer device can execute any of the target perception methods based on feature difference and error propagation described above.

[0172] Furthermore, this application also protects a control device, which may include a memory and a processor. The memory stores executable program code, and the processor is used to call and execute the executable program code to perform a target perception method based on feature difference and error propagation provided in this application. This application can divide the control device into functional modules based on the above method example. For example, each module can correspond to a specific function, or two or more functions can be integrated into a processing module. The integrated module can be implemented in hardware. It should be noted that the module division in this application is illustrative and only represents a logical functional division; other division methods may exist in actual implementation. It should also be noted that all relevant content of each step involved in the above method embodiment can be referenced to the functional description of the corresponding functional module, and will not be repeated here. It should be understood that the control device provided in this application is used to execute the above-mentioned target perception method based on feature difference and error propagation, and therefore can achieve the same effect as the above-described implementation method. When using integrated units, the control device may include a processing module and a storage module. When the control device is applied to a block device, the processing module can be used to control and manage the actions of the block device. The storage module can be used to support block devices in executing mutual program code, etc. The processing module can be a processor or controller, which can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination of functions that implement computing capabilities, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and microprocessors, etc., and the storage module can be a memory.

[0173] Furthermore, the control device provided in the embodiments of this application may specifically be a chip, component, or module. The chip may include a connected processor and a memory. The memory stores instructions, and when the processor calls and executes the instructions, the chip can execute the target perception method based on feature difference and error propagation provided in the above embodiments. The embodiments of this application also provide a computer-readable storage medium storing computer program code. When the computer program code is run on a computer, the computer executes the aforementioned method steps to implement the target perception method based on feature difference and error propagation provided in the above embodiments.

[0174] This application also provides a computer program product. When the computer program product is run on a computer, it causes the computer to perform the aforementioned related steps to realize the target perception method based on feature difference and error propagation provided in the above embodiments. The control device, computer-readable storage medium, computer program product, or chip provided in this application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here. Through the description of the above embodiments, those skilled in the art can understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the control device can be divided into different functional modules to complete all or part of the functions described above. In the embodiments provided in this application, it should be understood that the disclosed control device and method can be implemented in other ways. For example, the control device embodiments described above are merely illustrative. For example, the division of modules or units is merely a logical functional division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another control device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, control device or unit, and can be electrical, mechanical or other forms.

[0175] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multiple task processing and parallel processing are possible or may be advantageous. The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. The above content is only a specific implementation of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

Claims

1. A target perception method based on feature difference and error propagation, characterized in that, The method includes: The feature difference space is used to perform pixel-level comparison and residual extraction on the features of frame t and frame t-1 to obtain feature difference information, where t is an integer greater than or equal to 1; Using vehicle attitude information as geometric constraints, a depth estimation network is used to determine the distance of targets in the image based on the feature difference information, resulting in a two-dimensional environmental depth map. Target detection is then performed based on the two-dimensional environmental depth map to obtain a first detection result. The two-dimensional environmental depth map is mapped to three-dimensional space to generate a pseudo point cloud map, and the target detection is performed on the pseudo point cloud map to obtain a second detection result; The first detection result and the second detection result are fused using a weighted confidence score to determine the target detection result, wherein the target detection result includes at least the target's position, velocity, attitude, and category information; During the target detection process, the weight distribution of the feature difference space and the depth estimation network is adjusted based on the target detection error and the feature detection error. The target detection error is obtained by comparing the first detection result and the second detection result, and the feature detection error is obtained by comparing the predicted feature distribution and the actual feature distribution. The predicted feature distribution is the predicted feature distribution of the t-th frame image based on the two-dimensional environment depth map, vehicle attitude information and the features of the t-1 frame image. The features of the t-th frame image are then determined as the actual feature distribution.

2. The method as described in claim 1, characterized in that, The method further includes: The vehicle-mounted camera system acquires raw image data of the current moment. The original image is convolutionally processed using a backbone network to extract high-dimensional semantic features. The high-dimensional semantic features are fused and stored as the image features of frame t and the image features of frame t-1, respectively.

3. The method as described in claim 1, characterized in that, The vehicle attitude information includes vehicle distance data and vehicle attitude data, and the method further includes: The vehicle distance data is obtained using an onboard active radar. The vehicle's attitude data is acquired using an onboard posture sensor.

4. The method as described in claim 3, characterized in that, The vehicle-mounted posture sensor includes an inertial measurement unit and a wheel speedometer; the acquisition of the vehicle's posture data using the vehicle-mounted posture sensor includes: The inertial measurement unit and the wheel speed meter are used to acquire the vehicle's acceleration, speed and steering angle data.

5. The method as described in claim 1, characterized in that, The method further includes: Compare the consistency between the first and second detection results; Calculate the confidence residual based on the consistency of the results; The confidence residual is used as the target detection error.

6. The method as described in claim 1, characterized in that, The method further includes: Using the predicted depth of the two-dimensional environment depth map and the vehicle attitude information, the features of the t-1 frame image are transformed to the predicted feature distribution of the t frame image to obtain the predicted feature distribution; The features of the t-frame image are determined as the actual feature distribution; Compare the predicted feature distribution with the actual feature distribution to identify perceived noise; The perceived noise is used as the feature detection error.

7. The method as described in claim 1, characterized in that, The adjustment of the weight distribution of the feature difference space and the depth estimation network based on the target detection error and the feature detection error includes: The target detection error and the feature detection error are input into a differential propagation network for encoding to obtain a dynamic differential propagation signal; The parameters of the feature difference space and the depth estimation network are corrected in reverse using dynamic differential propagation signals.

8. A target sensing device based on feature difference and error propagation, characterized in that, The device includes: The feature difference module is used to perform pixel-level comparison and residual extraction of the features of frame t and frame t-1 using the feature difference space to obtain feature difference information, where t is an integer greater than or equal to 1; The first detection module is used to use vehicle attitude information as geometric constraints, and to use a depth estimation network to determine the distance of the target in the image based on the feature difference information to obtain a two-dimensional environment depth map, and to perform target detection based on the two-dimensional environment depth map to obtain a first detection result. The second detection module is used to map the two-dimensional environment depth map to three-dimensional space to generate a pseudo point cloud map, and to perform target detection on the pseudo point cloud map to obtain a second detection result. The result fusion module is used to fuse the first detection result and the second detection result using weighted confidence to determine the target detection result, wherein the target detection result includes at least the target's position, velocity, attitude and category information; The weight adjustment module is used to adjust the weight distribution of the feature difference space and the depth estimation network based on the target detection error and the feature detection error during the target detection process. The target detection error is obtained by comparing the first detection result and the second detection result, and the feature detection error is obtained by comparing the predicted feature distribution and the actual feature distribution. The predicted feature distribution is the predicted feature distribution of the t-th frame image based on the two-dimensional environment depth map, vehicle pose information and the features of the t-1 frame image, and the features of the t-th frame image are determined as the actual feature distribution.

9. An electronic device, characterized in that, The electronic device includes: a memory and at least one processor, wherein the memory stores instructions; The at least one processor invokes the instructions in the memory to cause the electronic device to execute the target perception method based on feature difference and error propagation as described in any one of claims 1 to 7.

Images