A multi-resolution end-to-end deep perceptual method with online parameter update
By using a multi-resolution end-to-end parallax inference network and an asynchronous online learning mechanism, the problems of high computational complexity and insufficient stability of deep perception methods in autonomous driving are solved, and real-time adaptive optimization and high-precision perception in complex environments are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SOUTHEAST UNIV
- Filing Date
- 2026-05-09
- Publication Date
- 2026-06-26
AI Technical Summary
Existing deep perception methods suffer from high computational complexity and inflexible resource adjustment in autonomous driving. Offline training lacks generalization ability, online learning affects system stability, cross-modal supervision increases system complexity, and it is difficult to achieve real-time adaptive optimization in complex environments.
A multi-resolution end-to-end parallax inference network is adopted. By decoupling progressive computation and asynchronous online learning, a sparse cross-modal supervision signal is constructed. Quality gating and backoff mechanisms are introduced to achieve decoupling and stability of inference and update, and online adaptive optimization is supported.
While ensuring real-time performance, it improves perception accuracy and robustness in complex environments, reduces the system's dependence on depth sensors, and enhances engineering deployability and system stability.
Smart Images
Figure CN122289252A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of autonomous driving and computer vision technology, specifically relating to a multi-resolution end-to-end and online parameter update depth perception method. Background Technology
[0002] In autonomous driving systems, environmental perception is the core foundation for safe driving and path planning, with the acquisition of three-dimensional spatial information being particularly crucial. Current mainstream depth perception methods mainly include two technical approaches: those based on LiDAR and those based on visual perception.
[0003] LiDAR can provide high-precision 3D point cloud information, but it is costly, energy-intensive, and has limitations in complex environments. In contrast, vision-based methods, due to their low cost and ease of deployment, are gradually becoming an important development direction for autonomous driving perception systems.
[0004] In visual methods, stereo vision calculates disparity by matching binocular images to recover scene depth information, which is a depth estimation method with explicit geometric constraints. However, traditional stereo matching methods rely on manually designed features, which are prone to matching errors in weakly textured regions, regions with repetitive textures, and occluded regions, leading to a decrease in depth estimation accuracy.
[0005] With the development of deep learning, stereo matching methods based on convolutional neural networks have achieved significant improvements in accuracy by constructing cost volumes and performing end-to-end learning. However, these methods still have the following problems:
[0006] (1) The computational complexity is high, making it difficult to flexibly adjust computing resources in real-time systems;
[0007] (2) The model relies on offline training and has insufficient generalization ability in complex environments;
[0008] (3) Online learning methods are usually coupled with the reasoning process, which affects the stability of the system;
[0009] (4) Cross-modal supervision (such as depth information) usually directly participates in inference input, increasing system complexity;
[0010] Therefore, it is necessary to propose an environment perception method that can integrate multi-view visual information and combine it with an asynchronous online learning mechanism to achieve online adaptive optimization of the model while ensuring real-time performance, and also has good engineering deployability. Summary of the Invention
[0011] To address the aforementioned issues, this invention discloses a deep perception method with multi-resolution end-to-end and online parameter updates. By decoupling inference and learning, progressive computation, and a quality-gated update mechanism, it achieves continuous adaptive optimization of the model while ensuring real-time performance, significantly improving perception accuracy and robustness in complex environments.
[0012] To achieve the above objectives, the technical solution of the present invention is as follows:
[0013] A multi-resolution end-to-end and online parameter update depth perception method includes the following steps:
[0014] (1) Construct a multi-view vision acquisition system to acquire left and right view images and depth information, and establish spatial mapping relationship between each sensor through calibration method to achieve multi-source data alignment;
[0015] (2) Input the left and right view images into a progressive multi-resolution disparity inference network for staged disparity prediction to obtain the disparity distribution of the scene, wherein:
[0016] The disparity prediction process consists of multiple progressive stages, which complete the disparity estimation step by step at different resolutions, from low resolution to high resolution. Each stage outputs an intermediate disparity result and gradually approximates the true disparity through a progressive residual refinement mechanism. It also supports terminating the output of the disparity result at the current resolution at any stage in advance.
[0017] (3) Perform coordinate transformation and disparity mapping on the depth information to construct a sparse disparity supervision signal. This supervision signal is only used for model optimization and does not participate in inference input.
[0018] (4) Construct an asynchronous online adaptive mechanism in which the reasoning process and the update process are independent of each other, wherein:
[0019] The inference process outputs disparity results in real time based on the left and right view images. The update process updates the disparity inference network with low-frequency parameters based on the sparse supervision signal. The update process is executed in the background and does not affect the real-time execution of the inference process.
[0020] (5) Construct a quality assessment mechanism for supervision signals, screen effective supervision data, and trigger model parameter updates when the supervision signal meets the preset quality requirements; otherwise, skip the update process.
[0021] (6) Based on the disparity results, calculate and convert them into depth information using a formula, and output the scene depth map as the final output.
[0022] The depth information is only used in the construction of the supervision signal in step (3) and is not used in the disparity prediction process in step (2).
[0023] As a supplement to the present invention, the progressive multi-resolution disparity inference network in step (2) adopts a multi-stage progressive structure, performs disparity estimation at different resolutions, and refines the disparity results step by step through residuals, including:
[0024] (1) Multi-scale feature coding submodule, used to extract features from left and right view images at different resolutions;
[0025] (2) A phased cost body construction submodule is used to construct the matching cost body within the local disparity search range at each scale;
[0026] (3) A three-dimensional feature aggregation submodule, used to perform joint modeling of the cost volume in terms of spatial and disparity dimensions;
[0027] (4) Initial disparity generation submodule, used to obtain coarse-scale disparity results;
[0028] (5) Progressive residual refinement submodule, which corrects parallax step by step in high-resolution space;
[0029] The process involves information exchange between stages through upsampling and residual propagation, forming a coarse-to-fine disparity estimation process. A dynamic disparity search range constraint mechanism is employed in each stage to adaptively determine the search range of the current stage based on the disparity prediction results of the previous stage.
[0030] As a supplement to the present invention, the progressive multi-resolution disparity inference network structure adopts a modular decomposition design, which enables the feature extraction submodule and the disparity estimation submodule to be separately adjusted in terms of parameters during the online update process.
[0031] As a supplement to the present invention, the inference process and the update process in step (4) are decoupled in time dimension, wherein the inference process continuously performs forward calculation to output disparity results, and the update process performs parameter optimization in the background and does not affect the inference delay.
[0032] As a supplement to the present invention, the model parameter update in step (5) is performed in a low-frequency triggering mode, and the update operation is performed when there is effective depth information. Furthermore, the parameters are optimized by constructing training samples based on historical data cache, so as to reduce the impact of the update process on the real-time performance of the system.
[0033] As a supplement to the present invention, in the supervision signal quality evaluation mechanism described in step (5), when the effective pixel ratio of the supervision signal meets the preset quality requirement threshold, the model parameter update is triggered and executed; otherwise, the update process is skipped.
[0034] As a supplement to the present invention, the update process in step (4) adopts a sparse supervised loss function based on the effective pixel region, and calculates the error between the predicted disparity and the supervised disparity only at the location with effective depth information.
[0035] As a supplement to the present invention, step (5) freezes the normalized layer statistical parameters during the parameter update process and adopts a single-step gradient update strategy to reduce the instability caused by online updates.
[0036] As a supplement to the present invention, this method also includes a model performance rollback mechanism: after the model is updated, the model performance is evaluated, and when the performance is lower than a preset threshold, the model parameters are restored to those before the update.
[0037] The present invention also provides an autonomous driving environment perception system, comprising:
[0038] The system comprises a multi-view vision acquisition module, a disparity inference module, a supervision signal construction module, an asynchronous online update module, and a depth map reconstruction module, among which each module works together to execute the aforementioned depth perception method.
[0039] The beneficial effects of this invention are as follows:
[0040] (1) Realize incremental calculation of disparity prediction and support dynamic trade-off between real-time performance and accuracy;
[0041] (2) Decouple the reasoning and learning processes to improve system stability;
[0042] (3) Online model optimization is achieved using sparse cross-modal supervision;
[0043] (4) Introduce quality gating and rollback mechanisms to improve the reliability of online learning;
[0044] (5) Reduce reliance on depth sensors and improve system deployability. Attached Figure Description
[0045] Figure 1 It is the overall technology roadmap.
[0046] Figure 2 This is a schematic diagram of the hardware platform and the asynchronous online update process.
[0047] Figure 3 This is a diagram of the multi-resolution end-to-end progressive disparity estimation network structure.
[0048] Figure 4 This is an example diagram of multi-resolution end-to-end disparity estimation (showing stepwise disparity estimation from low resolution to high resolution). Detailed Implementation
[0049] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.
[0050] like Figure 1 As shown, the depth perception method with multi-resolution end-to-end and online parameter update according to the present invention includes:
[0051] Step 1: Multi-view vision data acquisition and alignment
[0052] The system receives environmental perception data from the multi-view vision acquisition unit, including left and right view images. and depth information, wherein the depth information is at the pixel level The value at point is denoted as The overall hardware architecture is shown below. Figure 2 As shown, this hardware architecture is for experimental example only and is not intended as a hardware limitation. The left and right images are used for disparity calculation, and depth information is used to construct the supervision signal.
[0053] First, the intrinsic parameter matrix is obtained through camera calibration. and extrinsic parameter matrix Establish a unified spatial coordinate system to achieve multi-source data alignment. It is a 3×3 camera intrinsic parameter matrix, containing focal length and principal point coordinates; It is a 3×3 rotation matrix. It is a 3×1 translation vector, and together the two constitute the camera extrinsic parameters, which are used to describe the transformation relationship between the camera coordinate system and the world coordinate system.
[0054] For pixels depth value The camera imaging model is mapped to three-dimensional coordinates:
[0055] ;
[0056] in Represents pixels The corresponding 3D coordinates in the camera coordinate system Represents pixels The depth value, Represents the camera intrinsic parameter matrix The inverse matrix, For pixels Homogeneous coordinates.
[0057] This enables all sensor data to be compared and processed within a unified coordinate system.
[0058] Step 2: Progressive Multi-Resolution End-to-End Parallax Inference
[0059] The left and right view images are input into BatchNet (a progressive multi-resolution stereo matching network) for disparity estimation, such as... Figure 3 As shown, disparity estimation can be progressively refined from low resolution (1 / 16) to the original resolution, and can output estimation results according to real-time accuracy requirements, which facilitates online updates and computational control.
[0060] (1) Multi-scale feature extraction
[0061] At each resolution scale The features extracted from the left and right images are denoted as follows: :
[0062] ;
[0063] in, For convolutional coding submodule, Images are presented from both left and right perspectives.
[0064] (2) Construction of matching cost body
[0065] Define the parallax search range at each scale ,in Based on the disparity prediction results from the previous stage, the cost volume is adaptively determined and constructed as follows:
[0066] ;
[0067] in, Representing scale Next position In candidate parallax The corresponding matching cost; This represents the candidate disparity value or disparity index at the current scale; This indicates the maximum disparity search range at the current scale; This indicates a concatenation operation of the left and right feature vectors along the channel dimension; Indicates the left feature map in The eigenvector at that location; This indicates that the right feature map is offset in the horizontal direction. The eigenvectors after that.
[0068] (3) Three-dimensional feature aggregation
[0069] Perform 3D convolution aggregation on the cost volume:
[0070] ;
[0071] The cost volume after three-dimensional feature aggregation is represented at scale. ,Location Candidate parallax The value of the next generation; This represents a 3D convolutional neural network module, which consists of multiple stacked 3D convolutional layers, batch normalization layers, and activation function layers. It is used for joint feature modeling and information aggregation in both spatial and disparity dimensions to improve matching accuracy and robustness.
[0072] (4) Initial disparity estimation
[0073] Coarse-scale parallax is obtained using probabilistic regression:
[0074] ;
[0075] in, Representing scale Next position Initial disparity estimation results at the location; This represents the candidate disparity value or disparity index at the current scale, and its value range is... ; This indicates that normalization is performed along the disparity dimension; The cost volume after three-dimensional feature aggregation is represented at scale. ,Location Candidate parallax The value of the next generation.
[0076] (5) Progressive residual refinement
[0077] At high resolution scales, parallax is corrected step-by-step through a local search window:
[0078] ;
[0079] in, Indicates the first Sub-refinement time scale Next position Disparity estimation results at the location; This indicates the corresponding residual correction amount; This represents the updated disparity estimation result.
[0080] The progressive multi-resolution end-to-end disparity estimation network supports progressive steps from 1 / 16 → 1 / 8 → 1 / 4 → 1 / 2 → original resolution, ensuring the refinement of edges and thin structures, while providing intermediate output stages for early stopping or real-time decision-making.
[0081] Step 3: Construction of cross-modal sparse supervision signals
[0082] To improve disparity prediction accuracy, depth information is converted into disparity form, and a cross-modal supervision signal is constructed:
[0083] ;
[0084] in, Represents the pixels obtained by converting depth information. Supervisory parallax value at the location; Indicates focal length; Indicates the binocular baseline; Represents pixels The depth value.
[0085] Filter the set of valid depth pixels:
[0086] ;
[0087] in, Represents the set of effective depth pixels; This indicates the preset maximum effective parallax range.
[0088] The supervision signal is constructed only within the effective pixel area and is used only for subsequent model optimization. It does not participate in the disparity inference input in step two, thereby achieving weak coupling utilization of cross-modal information.
[0089] Step 4: Constructing an Asynchronous Online Adaptive Update Mechanism
[0090] Construct an asynchronous execution mechanism that decouples the inference thread from the update thread, such as... Figure 2 As shown.
[0091] Inference thread: based on current model parameters By performing forward calculations on the left and right view images, the predicted disparity result at the current time step is obtained. .
[0092] in, Indicates time Time pixel Predicted disparity value at; This represents the set of learnable parameters of the model at the current time step. Indicates the current moment.
[0093] Update thread: When the update interval condition is met Valid depth data exists and the supervision quality meets the requirements. Parameter updates are performed periodically, and the loss function is:
[0094] ;
[0095] in, Indicates time Sparse monitoring loss; Indicates the number of effective depth pixels; Represents the set of effective depth pixels; Represents the pixels obtained by converting depth information. Supervisory parallax value at the location; Indicates the update interval. Parameter update:
[0096] ;
[0097] in, This represents the updated set of model parameters; Indicates the learning rate; Represents the loss function Gradients to model parameters. Updates are performed infrequently and do not block the inference thread, ensuring system real-time performance.
[0098] Step 5: Monitor quality assessment and update control
[0099] Establish a monitoring signal quality assessment mechanism to screen effective monitoring data and control update trigger conditions.
[0100] Define the supervised quality function:
[0101] ;
[0102] in, Indicates the quality score of supervision; Indicates the number of effective depth pixels; Indicates the image height; Indicates the image width; This represents the total number of pixels in the image.
[0103] When the following conditions are met: When the monitoring signal meets the quality requirements, it triggers a model parameter update; otherwise, the update is skipped to avoid low-quality or anomalous data interfering with the model. This indicates the preset quality threshold.
[0104] Meanwhile, to ensure the stability of online updates, the following strategies are adopted:
[0105] (1) Freeze the statistical parameters of the normalization layer to avoid distribution drift caused by small batches of data;
[0106] (2) Single-step gradient update strategy to control the magnitude of parameter change each time;
[0107] (3) Limit update frequency and set update interval ,when Valid depth data exists and the supervision quality meets the requirements. Parameter updates are performed only when necessary;
[0108] (4) Only in the effective supervision area Internal calculation of loss.
[0109] The above measures together ensure the stability of the disparity estimation network during the online update process.
[0110] In addition, the present invention introduces a model performance rollback mechanism: after the model is updated, the model performance is evaluated, and when the performance is lower than a preset threshold, the model parameters are restored to those before the update, further ensuring system reliability.
[0111] Step Six: Deep Reconstruction and Result Output
[0112] Based on the final predicted disparity results Recovery depth:
[0113] ;
[0114] in, Represents pixels The final predicted disparity value at the location; This represents the output depth value obtained from the reconstruction; Indicates focal length; This represents the binocular baseline. The final depth map inference result is used as the environmental perception output of the autonomous driving system, such as... Figure 4 As shown.
[0115] In the above implementation method:
[0116] (1) Depth information is only used for constructing supervisory signals and does not participate in inference input, thus avoiding sensor coupling;
[0117] (2) The inference thread and the update thread run independently to ensure real-time performance;
[0118] (3) The disparity estimation network supports progressive multi-resolution output and online parameter updates;
[0119] (4) This invention is not limited to specific network structure implementations;
[0120] (5) The progressive parallax inference structure can be implemented by different neural networks;
[0121] (6) The asynchronous update mechanism can be implemented through different parallel computing methods;
[0122] (7) The quality assessment function can take different forms.
[0123] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.
Claims
1. A deep perceptual method with multi-resolution end-to-end and online parameter update, characterized in that, The method comprises the following steps: (1) constructing a multi-view visual acquisition system, acquiring left and right view images and depth information, and establishing a spatial mapping relationship between sensors through a calibration method to realize multi-source data alignment; (2) inputting the left and right view images into a progressive multi-resolution disparity inference network for staged disparity prediction to obtain the disparity distribution of the scene, wherein: The disparity prediction process is composed of multiple progressive stages, and the disparity estimation is completed step by step at different resolutions, gradually increasing from low resolution to high resolution. Each stage outputs an intermediate disparity result, and gradually approaches the true disparity through a progressive residual refinement mechanism, and supports early termination of the output of the current resolution disparity result at any stage; (3) performing coordinate conversion and disparity mapping on the depth information to construct a sparse disparity supervision signal, which is only used for model optimization and does not participate in inference input; (4) constructing an asynchronous online adaptive mechanism independent of the inference process and the update process, wherein: The inference process outputs the disparity result in real time based on the left and right view images, and the update process updates the low-frequency parameters of the disparity inference network based on the sparse disparity supervision signal, and the update process is executed in the background and does not affect the real-time execution of the inference process; (5) constructing a supervision signal quality evaluation mechanism to filter effective supervision data, and triggering model parameter update when the supervision signal meets the preset quality requirement condition, otherwise skipping the update process; (6) converting the depth information into depth information according to the disparity result through a formula to output a scene depth map as the final output; Wherein, the depth information only participates in the construction of the supervision signal in step (3) and does not participate in the disparity prediction process in step (2).
2. The method of claim 1, wherein: The progressive multi-resolution disparity inference network of step (2) adopts a multi-stage progressive structure to perform disparity estimation at different resolutions and gradually refine the disparity result through a residual method, comprising: (1) a multi-scale feature encoding submodule for extracting features of left and right view images at different resolutions; (2) a staged cost volume construction submodule for constructing a matching cost volume within a local disparity search range at each scale; (3) a three-dimensional feature aggregation submodule for jointly modeling the cost volume in space and disparity dimension; (4) an initial disparity generation submodule for obtaining a coarse-scale disparity result; (5) a progressive residual refinement submodule for step-by-step correction of disparity in a high-resolution space; Wherein, the stages are connected through upsampling and residual transmission to form a coarse-to-fine disparity estimation process, and a dynamic disparity search range constraint mechanism is adopted in each stage to adaptively determine the search interval of the current stage based on the disparity prediction result of the previous stage.
3. The method of claim 2, wherein: The progressive multi-resolution disparity inference network structure adopts a modular decomposition design, so that the feature extraction submodule and the disparity estimation submodule can be adjusted separately in the online update process.
4. The method of claim 1, wherein: The inference process and the update process described in step (4) are decoupled in time dimension. The inference process continuously performs forward calculations to output disparity results, while the update process performs parameter optimization in the background and does not affect the inference delay.
5. The method according to claim 1, characterized in that: The model parameter update in step (5) is performed using a low-frequency triggering method, and the update operation is performed when there is valid depth information. Furthermore, the parameters are optimized by constructing training samples based on historical data cache to reduce the impact of the update process on the real-time performance of the system.
6. The method according to claim 1, characterized in that: In the supervision signal quality assessment mechanism described in step (5), when the effective pixel ratio of the supervision signal meets the preset quality requirement threshold, the model parameter update is triggered and executed; otherwise, the update process is skipped.
7. The method according to claim 1 or 5, characterized in that: The update process in step (4) uses a sparse supervised loss function based on the effective pixel region, and calculates the error between the predicted disparity and the supervised disparity only at the location with effective depth information.
8. The method according to claim 1, characterized in that: Step (5) freezes the normalized layer statistical parameters during the parameter update process and adopts a single-step gradient update strategy to reduce the instability caused by online updates.
9. The method of claim 1, wherein: It also includes a model performance rollback mechanism: after the model is updated, the model performance is evaluated, and when the performance is lower than a preset threshold, the model parameters are restored to those before the update.
10. An autonomous driving environment perception system, characterized in that, include: The system comprises a multi-view vision acquisition module, a parallax inference module, a supervision signal construction module, an asynchronous online update module, and a depth map reconstruction module, wherein each module works in concert to perform the method described in any one of claims 1 to 9.