Adas intelligent control method based on full vehicle camera perception

Abstract

This invention provides an ADAS intelligent control method based on full-vehicle camera perception, comprising the following steps: S1: Rigid body spatiotemporal synchronization and online joint calibration of the full-vehicle surround-view camera group based on Lie group and Lie algebra; S2: Pixel-level alignment and full-domain static-dynamic bi-branch semantic mask generation; S3: Hierarchical confidence perception and target full attribute decoupling extraction based on variational Bayesian inference; S4: Spatiotemporal coupled risk field modeling based on lattice Boltzmann machine; S5: Multi-constraint hierarchical decision-making and expected behavior sequence generation; S6: Actuator control quantity calculation and feedforward compensation. This invention breaks down the independent barriers between each link in traditional schemes by deeply integrating the Lie group and Lie algebra online calibration algorithm, the variational Bayesian decoupled perception algorithm, and the lattice Boltzmann machine risk field modeling algorithm, achieving end-to-end collaborative optimization and closed-loop iteration from front-end calibration to end-to-end control.

Description

Technical Field

[0001] This invention relates to the field of intelligent vehicle driver assistance technology, and in particular to an ADAS intelligent control method based on full-vehicle camera perception. Background Technology

[0002] Advanced Driver Assistance Systems (ADAS) are a core safety feature of intelligent vehicles. ADAS solutions based on full-vehicle vision perception have become the mainstream technology for large-scale deployment in passenger vehicles due to their low hardware cost, ease of deployment, and rich semantic information. However, existing pure vision-based ADAS solutions suffer from the following core technical shortcomings in engineering implementation and practical applications: The multi-camera calibration process has inherent shortcomings. Existing solutions generally adopt offline factory calibration. In actual vehicle use, factors such as vehicle body deformation, suspension vibration, ambient temperature drift, and lens aging can directly lead to inaccurate calibration parameters, causing a chain of problems such as subsequent perception and positioning deviations, stitching ghosting, and decision-making misjudgments. A few online calibration solutions only aim to minimize global reprojection error, which cannot match the high-precision requirements of ADAS systems in high-risk areas. The calibration process is completely disconnected from backend safety decision-making and cannot dynamically adjust the calibration optimization direction based on real-time driving risks.

[0003] The accuracy and reliability of the environmental perception stage are insufficient. Conventional target detection algorithms only output the overall target detection confidence score, failing to decouple the independent errors of core attributes such as target spatial position, motion state, semantic category, and contour boundary. In scenarios such as target occlusion, distant small targets, complex lighting, and severe weather, false negatives and missed detections are prone to occur. Furthermore, the perception stage and the front-end calibration stage are independent of each other. The pose uncertainty of the calibration stage cannot be passed to the perception stage, making it impossible to improve perception accuracy through prior calibration constraints, and the perception results cannot be used to optimize calibration parameters.

[0004] The risk assessment and decision-making processes are poorly adapted. Existing solutions mostly use discrete threshold-based risk judgments based on distance and relative speed, which cannot characterize the continuous distribution and diffusion characteristics of driving risks in the spatiotemporal dimensions, nor do they consider the uncertainties in the attributes of the perception process. This easily leads to problems such as delayed risk assessment, overly conservative approaches, or insufficient safety redundancy. At the same time, the decision-making process and the front-end perception and calibration processes are transmitted in a one-way open loop, and the safety requirements of the decision cannot guide the front-end calibration and perception optimization in reverse.

[0005] The end-to-end architecture has systemic flaws. The calibration, perception, decision-making, and control stages of the existing ADAS solution are unidirectional serial architectures, with each stage independent of the others. They only realize the one-way transmission of the output of the preceding stage to the following stage, without forming a two-way interaction and closed-loop correction throughout the entire link. The errors of each stage will accumulate step by step, which can easily lead to a decrease in system robustness and even cause decision-making and control errors under complex urban roads and high-speed dynamic conditions. Summary of the Invention

[0006] This invention provides an ADAS intelligent control method based on full-vehicle camera perception, which overcomes the technical defects of existing pure vision ADAS solutions and solves the core pain points of existing solutions, such as easy calibration inaccuracy, insufficient perception accuracy, discrete risk modeling, open-loop independence of each link, and easy accumulation of errors. It realizes deep integration and closed-loop iterative optimization of the entire link of calibration-perception-decision-control, and comprehensively improves the driving safety, scenario adaptability and robustness of ADAS system under all working conditions.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: The ADAS intelligent control method based on full-vehicle camera perception includes the following steps: S1: Combining the vehicle synchronization benchmark of the body domain controller and the real-time dynamic parameters collected by the vehicle CAN bus, the spatiotemporal synchronization and online joint calibration of the multiple original images collected by the vehicle-mounted surround view camera group are completed to obtain time-aligned multiple images and global spatial mapping parameters. S2: Based on global spatial mapping parameters, complete the whole-domain planar pixel-level alignment and stitching of multi-path images to generate a panoramic stitched image. Perform static-dynamic dual-branch semantic segmentation on the panoramic stitched image to obtain the whole-domain semantic mask and the division results of static and dynamic regions. S3: Combining panoramic stitched images, global semantic masks, static and dynamic region segmentation results, and global spatial mapping parameters, the target candidate region is selected, the target full attribute decoupled perception and inter-frame tracking are completed, and the global target set and the uncertainty distribution of target attributes are obtained. S4: Based on the global semantic mask, the global target set and the uncertainty distribution of target attributes, combined with the real-time vehicle dynamics parameters, a global spatiotemporal coupling risk field covering multiple prediction time domains is constructed, and the boundary of the vehicle dynamics feasible domain is determined simultaneously. S5: Using the spatiotemporal coupling risk field of the vehicle body and the boundary of the feasible domain of vehicle dynamics as constraints, combined with the driver's operation intention signal collected by the vehicle CAN bus, candidate behavior sequences are generated through multi-constraint hierarchical optimization, and the optimal expected behavior sequence is obtained after safety verification. S6: Based on the optimal expected behavior sequence and combined with the real-time dynamic parameters of the vehicle, construct the longitudinal and lateral dynamic inverse model of the vehicle, complete the control quantity calculation and feedforward compensation, and generate the final vehicle actuator control command after boundary constraints and human-machine arbitration.

[0008] In this specification, the online joint calibration of S1 is implemented using a calibration model based on Lie group and Lie algebra. The rigid body pose transformation of each camera in the vehicle surround view camera group relative to the center of mass of the vehicle body is modeled on a special Euclidean group SE(3). The Lie algebra elements corresponding to this Lie group are used to represent the rotation and translation parameters of the camera, avoiding the gimbaled singularity problem inherent in the Euler angle representation method. Based on the Lie algebra left-multiplication perturbation model, a linearized solution equation for the reprojection error is constructed. The globally optimal internal and external pose parameters of the camera are obtained through iterative optimization, and the corresponding global spatial mapping parameters are generated.

[0009] In this specification, the optimization objective of online joint calibration of S1 introduces the risk field gradient constraint corresponding to the spatiotemporal coupled risk field of the whole vehicle body; based on the risk field gradient, high-risk areas within the whole vehicle body are located, and higher optimization weights are given to the reprojection error of image feature points in high-risk areas. While ensuring the global calibration accuracy, the spatial mapping accuracy of high-risk areas is improved first.

[0010] In this specification, the target full-attribute decoupled perception of S3 is implemented using a detection model based on variational Bayesian inference. The core attributes of the target are decoupled into four categories: spatial location attributes, motion state attributes, semantic category attributes, and contour boundary attributes. A Bayesian network model with multi-attribute association is constructed. A decomposable variational distribution is introduced to approximate the true posterior distribution of the target attributes. Iterative solution is completed by maximizing the lower bound of evidence. The probability distribution and confidence of each attribute category are estimated independently, realizing the decoupled estimation and uncertainty quantification of the target attributes.

[0011] In this specification, the S3 variational Bayesian inference detection model uses the camera pose parameters and calibration uncertainty obtained from online joint calibration as the spatial prior distribution of the target's spatial position attributes; the pose accuracy information of the calibration process is directly integrated into the estimation process of the target attributes, realizing the positive transfer of calibration accuracy to perception accuracy.

[0012] In this specification, the S4 vehicle body spatiotemporal coupled risk field is constructed using a lattice Boltzmann machine model. A two-dimensional nine-speed lattice model is used to divide the vehicle body plane and multiple prediction time domains into spatiotemporal dimensions. The static environmental risk corresponding to the global semantic mask, the dynamic target risk corresponding to the global target set, and the dynamic constraint risk corresponding to the real-time dynamic parameters of the vehicle are integrated to construct the equilibrium distribution function of the model. The spatiotemporal evolution of the risk field is completed through collision and migration steps to obtain a continuous risk distribution covering the entire spatial domain and multiple prediction time domains, which characterizes the spatial diffusion and temporal evolution characteristics of driving risks.

[0013] In this specification, during the construction of the equilibrium distribution function of S4, the uncertainty distribution of target attributes is incorporated into the calculation of dynamic target risk; based on the probability distribution of target position and motion state, the mathematical expectation of dynamic risk is solved, so that the risk field modeling fully covers the uncertainty of the perception link.

[0014] In this specification, the multi-constraint hierarchical optimization of S5 divides decision constraints into safety hard constraints, comfort soft constraints, and driving intention matching constraints according to their priority from high to low. Among them, safety hard constraints are the highest priority constraints that cannot be broken. All candidate behavior sequences must meet the hard requirements of risk threshold, dynamic feasible region, and safe distance throughout the entire process. The weight coefficients of soft constraints are dynamically adjusted based on the current driving scenario recognition results to ensure driving safety while taking into account the matching degree between driving comfort and driver's operating intention.

[0015] In this specification, the feedforward compensation of S6 includes jerk feedforward compensation and slope resistance compensation in the longitudinal control dimension, and angular change rate feedforward compensation, steering friction feedforward compensation, and road curvature feedforward compensation in the lateral control dimension, which are used to compensate for the response lag, nonlinear characteristics and external road disturbances of the vehicle's actuator system.

[0016] In this specification, the boundary constraint of S6 is the physical limit boundary constraint of the actuator, which is used to ensure that the output control command is within the range of the maximum stroke and maximum response rate of the vehicle actuator; the human-machine arbitration is used to prioritize the response to the driver's operation command when the driver's active operation input exceeds the preset threshold.

[0017] In summary, the present invention has at least the following beneficial effects: By using Lie groups and Lie algebras to achieve online joint calibration of multiple cameras, it can adapt to parameter deviations caused by vehicle body deformation, temperature drift and vibration. At the same time, it integrates risk field gradients to dynamically optimize the calibration target, prioritizes the spatial mapping accuracy of high-risk areas, improves the reliability of perception and decision-making from the source, and solves the problems of inaccuracy and disconnection from safety requirements in traditional offline calibration. Variational Bayesian inference is used to achieve decoupled perception and confidence quantification of all target attributes. It can independently evaluate the estimation errors of position, motion, category and contour. Combined with the prior constraints of front-end calibration, it significantly reduces the probability of missed detection and false detection in scenarios such as occlusion and small distant targets, and improves the stability and reliability of perception results. Based on the lattice Boltzmann mechanism, a spatiotemporally coupled continuous risk field is constructed, which integrates the static environment, dynamic target uncertainty and vehicle dynamic limits into a model. Compared with the traditional discrete risk assessment, it can more comprehensively and continuously characterize the distribution and diffusion of driving risks, and provide more accurate safety boundaries for decision-making. It achieves bidirectional interaction between the three core algorithms of calibration, perception, and risk modeling, and, together with the full-domain verification and closed-loop correction of the end-point execution effect, breaks the barrier of the traditional unidirectional serial architecture, effectively suppresses the accumulation of errors in each link, and greatly improves the robustness of the system under complex road conditions and dynamic working conditions. Employing multi-constraint hierarchical decision-making and dynamic inverse model control with feedforward compensation, it balances driving comfort and driver's operational intentions while strictly ensuring driving safety, and improves the response speed and tracking accuracy of control commands, thereby optimizing the overall driving experience and scenario adaptability. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the ADAS intelligent control method based on full-vehicle camera perception involved in this invention.

[0020] Figure 2 This is a schematic diagram of the sensing and online calibration process involved in this invention.

[0021] Figure 3 This is a schematic diagram of the target full attribute decoupling perception process involved in this invention.

[0022] Figure 4 This is a schematic diagram of the process for modeling the spatiotemporal coupled risk field involved in this invention. Detailed Implementation

[0023] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the embodiments of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0024] The following disclosure provides many different implementations or examples for carrying out different structures of the embodiments of the present invention. To simplify the disclosure of the embodiments of the present invention, specific examples of components and arrangements are described below. Of course, these are merely examples and are not intended to limit the embodiments of the present invention. Furthermore, reference numerals and / or reference letters may be repeated in different examples of the embodiments of the present invention; such repetition is for simplification and clarity and does not in itself indicate a relationship between the various implementations and / or arrangements discussed.

[0025] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0026] like Figure 1 As shown, this embodiment provides an ADAS intelligent control method based on full-vehicle camera perception, including: S1: Rigid body spatiotemporal synchronization and online joint calibration of a full-vehicle surround-view camera group based on Lie groups and Lie algebras This step's input sources include the vehicle's hardware synchronization time reference, the original image frames and hardware trigger timestamps collected by the vehicle's surround-view camera group, the real-time dynamic parameters of the vehicle body transmitted via the vehicle's CAN bus, the calibration parameter correction values ​​output from the previous frame iteration, the initial intrinsic and extrinsic parameters calibrated at the vehicle's factory settings during the first execution, and the feedback risk field gradient matrix. This provides a unique spatiotemporal reference and spatial mapping reference for the entire solution, addressing three core pain points commonly found in existing multi-camera vehicle calibration solutions: offline calibration inaccuracies, singularities in Euler angle representation, and a disconnect between calibration accuracy and ADAS safety requirements. The perception and online calibration process is as follows: Figure 2 As shown.

[0027] 1.1 Nanosecond-level spatiotemporal synchronization and motion compensation Based on the vehicle's hardware synchronization time reference, the eight original image frames captured by the eight cameras in the vehicle surround view camera group are synchronized using hardware-triggered timestamp matching and inter-frame motion interpolation to ensure that the exposure time difference of the eight image frames does not exceed 100ns. Simultaneously, based on the vehicle's real-time dynamic parameters, pixel-level compensation is performed on the rigid body displacement of the vehicle caused by the time difference, eliminating inter-frame pixel offsets caused by the vehicle's own motion, and generating eight synchronized image frames.

[0028] 1.2 Construction of a calibration model based on Lie groups and Lie algebras The pose transformation of a single camera relative to the vehicle's center of mass is modeled on a special Euclidean group SE(3). This group can represent rigid body rotation and translation in three-dimensional space without singularity, completely avoiding the gimbal lock problem that occurs in conventional Euler angle representation methods when the pitch angle is close to 90 degrees. For the eight cameras in the vehicle surround view camera group, a unique index number is assigned to each camera, with the number ranging from 1 to 8. Let the SE(3) group element corresponding to the i-th camera be denoted as . This element fully represents the rotation and translation transformation of the i-th camera relative to the coordinate system of the vehicle's center of mass. (Similar to Lie group elements) The corresponding Lie algebra element is denoted as This element belongs to the Lie algebra corresponding to a special Euclidean group. The conversion between Lie groups and Lie algebras is achieved through exponential mapping, and the corresponding conversion formula is: ,in Lie algebra elements The antisymmetric matrix representation, It is composed of translation vector and rotation vector, and its specific form is as follows: , Let be the translation vector of the i-th camera relative to the center of mass of the vehicle body. Let be the rotation vector of the i-th camera relative to the center of mass of the vehicle body. The specific matrix form is as follows: ,in Rotation vector The corresponding 3×3 antisymmetric matrix, , , These are rotation vectors. Components on the X, Y, and Z axes The specific form is as follows: Meanwhile, the intrinsic parameter matrix of the i-th camera is defined as follows: The intrinsic parameter matrix fully characterizes the internal optical parameters of the camera, specifically in the form of: ;in , Let be the focal lengths of the i-th camera along the X and Y axes of the image, respectively. , Let be the coordinates of the principal point of the i-th camera in the image plane.

[0029] 1.3 Construction of the Optimization Objective Function by Integrating Risk Field Gradients Conventional calibration algorithms, which only optimize by minimizing global reprojection error, result in inconsistent calibration accuracy between low-risk open areas around the vehicle and high-risk obstacle and drivable areas, failing to meet the high-precision perception requirements of ADAS systems for high-risk areas. Therefore, by incorporating the risk field gradient matrix into the optimization objective and assigning higher weight to the reprojection error in high-risk areas, the calibration optimization process is forced to converge towards improving the perception accuracy of high-risk areas, achieving deep collaboration between front-end calibration and back-end safety decision-making.

[0030] For the 8 aligned image frames after time synchronization, extract the ORB feature points of each image frame. Let the pixel coordinates of the k-th feature point extracted from the j-th frame of the i-th camera be denoted as . The coordinates of the three-dimensional spatial point corresponding to this feature point in the vehicle's centroid coordinate system are: Based on the pose representation using Lie group Lie algebras, the reprojection error of this feature point is defined as: ,in This is a perspective projection function that projects three-dimensional homogeneous coordinates onto a two-dimensional pixel plane. The specific operation rules are as follows: .

[0031] Introducing the risk field gradient matrix This matrix represents the spatial rate of change of the risk field around the vehicle body, enabling precise location and extent of high-risk areas, and constructing an optimization objective function that incorporates safety constraints: ,in The number of consecutive frames used for calibration. The number of valid feature points extracted in each frame of the image. This is the risk constraint weight coefficient, used to balance the weights of global reprojection error and high-risk area constraints. The initial value is set to 0.1, and it can be adaptively adjusted according to the level of risk in the scenario. This optimization objective function ensures the overall accuracy of global calibration while prioritizing the improvement of calibration accuracy in high-risk areas that are crucial to driving safety.

[0032] 1.4 Optimization Solution Based on Lie Algebra Left Multiplication Perturbation Model Lie groups belong to non-Euclidean spaces, making it impossible to directly perform derivative operations on the elements of a Lie group. Therefore, a Lie algebra left-multiplication perturbation model is adopted to transform the optimization problem of the Lie group into an optimization problem of small perturbations of the Lie algebra, which greatly simplifies the calculation process of differentiation and iterative solution and meets the real-time requirements of the vehicle system.

[0033] Lie group elements Multiply by a tiny perturbation transformation on the left. ,in For the Lie algebra elements corresponding to small perturbations, the reprojection error is related to Perform a linearized approximation: ,in For the left-multiplication perturbation operator on the Lie group, it means that the perturbation transformation is performed first and then the original pose transformation is performed. Let be the Jacobian matrix of the reprojection error with respect to a small Lie algebraic perturbation. This matrix fully characterizes the effect of small pose changes on the reprojection error, and its specific form is as follows: ;in , representing a point in three-dimensional space After pose transformation, the homogeneous coordinates in the coordinate system of the i-th camera.

[0034] Based on the linearized reprojection error, the Gauss-Newton method is used for iterative solution. Each iteration updates the Lie algebra elements and intrinsic parameter matrix until the decrease in the objective function is less than a preset threshold, at which point iterative convergence is achieved. After convergence, the optimal Lie algebra elements are obtained. and intrinsic parameter matrix Then, the optimal Lie group pose is obtained through exponential mapping. Based on the optimized intrinsic and extrinsic parameters, the homography matrix and projection transformation matrix from the pixel coordinate system to the two-dimensional plane coordinate system and the three-dimensional coordinate system of the vehicle's centroid are generated for each image frame. These are collectively referred to as the global projection parameters. .

[0035] Model Iteration and Application Description: The optimization process of this algorithm is an online iterative optimization process, requiring no offline pre-training. During vehicle operation, image frames from each control cycle participate in the optimization solution. Through continuous frame iteration, the internal and external participating pose parameters are continuously corrected to adapt to parameter shifts caused by changes in vehicle body deformation, vibration, temperature drift, and other operating conditions. Initially, the vehicle's factory-calibrated internal and external participating pose parameters are used as the initial values ​​for iteration.

[0036] The output of this step specifically includes: 8 time-synchronized aligned image frames and an optimized global intrinsic parameter matrix. With the optimal Lie group pose Global projection parameters Optimal Lie algebra elements .

[0037] S2: Pixel-level alignment and global static-dynamic bi-branch semantic mask generation This step follows the time-synchronized aligned image frames output from step S1 and the global projection parameters involving both internal and external elements. The core objective is to achieve global stitching of multiple images and accurate separation of environmental semantics. This solves the problems of stitching ghosting, false detection caused by coupling of static and dynamic environments, and excessive invalid computation in existing multi-camera fusion solutions. It provides accurate regional constraints and semantic priors, significantly compresses the invalid computation area of ​​target detection, and improves the real-time performance and accuracy of the perception process.

[0038] 2.1 Single-frame image preprocessing First, for each aligned image frame output by S1, a distortion correction mapping table is generated based on the distortion coefficients in the intrinsic parameter matrix obtained through synchronous optimization of S1. Pixel-level distortion correction is then performed on each image based on this mapping table, eliminating radial and tangential distortion caused by the wide-angle lens of the camera, generating distortion-free image frames. Subsequently, an adaptive bilateral filtering algorithm is applied to the distortion-free image frames for denoising. This algorithm can suppress image sensor noise while fully preserving key features such as edges and corners in the image, avoiding edge blurring caused by conventional Gaussian filtering. This provides high-quality image input for subsequent feature extraction, image stitching, and semantic segmentation, generating preprocessed image frames.

[0039] 2.2 Global pixel-level alignment and stitching of 8-channel images Global projection parameters based on the output of step S1 Each pixel in the eight preprocessed image frames is projected onto the vehicle's global plane. The global plane, with the vehicle's centroid as the origin, the longitudinal direction of movement as the positive X-axis, and the leftward direction as the positive Y-axis, covers a rectangular area of ​​30 meters in front and behind the vehicle and 15 meters to the left and right. This plane is the sole spatial reference plane in the entire process. Through projection transformation, a bird's-eye view projection frame is generated for each image. These eight bird's-eye view projection frames collectively cover the entire global plane, with 15% to 30% overlap between the fields of view of adjacent cameras. For pixel fusion in the overlapping areas, a pixel gradient weight-based fusion strategy is adopted. Higher gradient values ​​are assigned to edge and texture pixels, while lower gradient values ​​are assigned to smoother pixels. This effectively eliminates ghosting and blurring issues caused by parallax in the overlapping areas, ultimately generating a panoramic stitched image covering the entire vehicle's global plane.

[0040] 2.3 Static-Dynamic Dual-Branch Separation of Semantic Masks To avoid perceptual errors caused by the feature coupling between static environment and dynamic target, a dual-branch parallel processing architecture is adopted to generate static semantic masks and dynamic semantic masks respectively, thereby achieving accurate separation of environmental semantics.

[0041] First, the static branch is executed. Based on the prior static semantic mask output from the previous frame, and combined with the grayscale, gradient, and texture features of the current frame's panoramic stitched image, the static branch first uses inter-frame difference combined with dense optical flow to extract static regions within the vehicle's global plane that do not undergo relative motion over time, eliminating interference from dynamic targets. Then, a lightweight semantic segmentation network is used to perform pixel-level semantic classification on the extracted static regions, generating a static semantic mask. The semantic categories included in the static semantic mask are: drivable road surface, lane lines, curbs, guardrails, traffic signs, static obstacles, green belts, and building walls. Each pixel corresponds to a unique semantic category.

[0042] Then, dynamic branching is performed. First, in the panoramic stitched image, all areas already covered by the static semantic mask are removed, resulting in dynamic candidate regions containing only potential dynamic targets. This significantly reduces the detection range of dynamic targets and minimizes unnecessary computation. Next, inter-frame dense optical flow is applied to the dynamic candidate regions, combined with compensation based on the vehicle's self-motion calculated from real-time vehicle dynamic parameters. This eliminates global pixel displacement caused by the vehicle's own motion, extracting pixel regions with relative motion to the vehicle body, and generating a dynamic semantic mask. The dynamic semantic mask contains semantic categories specifically for motor vehicles, non-motor vehicles, pedestrians, animals, and movable obstacles, with each pixel corresponding to a unique semantic category.

[0043] 2.4 Assigning Confidence Values ​​to Global Semantic Masks Static and dynamic semantic masks are superimposed to generate a global semantic mask covering the entire vehicle body plane, ensuring that every pixel in the global plane has a corresponding semantic category. Simultaneously, for each pixel within the mask, an initial confidence level between 0 and 1 is assigned based on its corresponding semantic category, inter-frame feature matching degree, and projection-reprojection error. The closer the confidence level is to 1, the more reliable the semantic classification result for that pixel.

[0044] The output of this step specifically includes: a panoramic stitched image of the entire vehicle body plane, a global semantic mask, and the division results of static and dynamic regions.

[0045] S3: Hierarchical confidence perception and decoupled extraction of target full attributes based on variational Bayesian inference This step builds upon the panoramic stitched image, global semantic mask, and dynamic / static region segmentation results output from step S2. It also introduces the optimal Lie algebra element output from step S1 as a spatial prior. The core objective is to achieve decoupled perception of all target attributes and uncertainty quantification. This addresses the core pain points of existing vehicle-mounted target detection solutions, such as the inability of overall confidence to characterize single-attribute errors, high false negative and false positive rates in occluded and small target scenes, and a disconnect between perception results and calibration accuracy. Furthermore, it provides target attribute input with uncertainty quantification, enabling deep collaboration between perception and risk decision-making. The target full-attribute decoupled perception process is as follows: Figure 3 As shown.

[0046] 3.1 Sensing Region Hierarchy and Precise Candidate Region Selection Based on the global semantic mask and dynamic / static region segmentation results output from step S2, a three-level perception region division is performed on the entire vehicle plane. Different regions correspond to different perception priorities and computational resource allocation strategies. This significantly reduces the overall computational load while ensuring no missed detections in high-risk areas, meeting the real-time requirements of in-vehicle systems. The three-level perception regions are specifically high-priority, medium-priority, and low-priority regions. The high-priority region includes all dynamic candidate regions within the drivable road surface and the static region within 5 meters in front of and behind the vehicle. This region is directly related to vehicle driving safety, so the lowest candidate region screening threshold is used to ensure that no potential targets are missed. The medium-priority region includes dynamic candidate regions within 1 meter outside the drivable road surface and the static region within 5 to 15 meters of the vehicle. This region has potential risk intrusion, so a medium screening threshold is used to balance detection accuracy and computational load. The low-priority region includes regions more than 1 meter outside the drivable road surface and the static region more than 15 meters outside the vehicle. This region has minimal impact on the current vehicle driving safety, so a higher screening threshold is used to filter out invalid candidate regions and reduce unnecessary computation. Based on the three-level region segmentation results, the region of interest of the corresponding region is extracted from the panoramic stitched image to generate a set of target candidate regions, providing an accurate input range for subsequent target attribute detection.

[0047] 3.2 Construction of a Decoupling Model for Target Attributes Based on Variational Bayesian Inference Conventional target detection algorithms only output the bounding box of the target and an overall confidence score, failing to distinguish between errors in different attributes such as target position, category, and motion state. When the target is occluded or at a distance, the overall confidence score cannot provide reliable uncertainty information for subsequent decisions, easily leading to decision-making errors. Therefore, a fully attribute-decoupled Bayesian network model is adopted, decoupling the core attributes of the target into four interrelated but independently estimated attributes: spatial location, motion state, semantic category, and contour boundary. A unique index number is assigned to each detected target, and the four attributes of the m-th target are denoted as random variables. , , , ,in Corresponding spatial location attributes, Corresponding motion state attributes, Corresponding semantic category attributes, Corresponding to the contour boundary attributes. Simultaneously, let the image observation values ​​corresponding to the four attributes of the m-th target be respectively... , , , The observations are extracted from candidate regions of the panoramic stitched image by the attribute decoupling detection network.

[0048] To improve the estimation accuracy of the target spatial location attributes, the optimal Lie algebra element output in step S1 is introduced. As prior distribution parameters for spatial location attributes, the camera pose accuracy information obtained from the front-end calibration stage is directly integrated into the target attribute estimation process, achieving deep fusion of calibration and perception. This avoids the problems of independent calibration and perception, and the inability to propagate and correct errors, found in conventional solutions. Based on this, a Bayesian network model fusing spatial priors is constructed. The joint probability distribution formula of this model is: ;in The prior distribution of the spatial location attribute, which models the impact of camera pose uncertainty on target spatial location estimation, is defined as a Gaussian distribution, specifically in the form: ;in The mean of the prior distribution is the optimal Lie algebra element obtained by optimization in step S1. Calculated through exponential mapping and projection transformation; The covariance matrix of the prior distribution is given by the reprojection error Jacobian matrix from step S1. The calculations fully characterize the uncertainty of camera pose calibration. The lower the uncertainty of camera calibration, the smaller the covariance of the prior distribution, the stronger the constraint on spatial position estimation, and the higher the accuracy of target position estimation, thus realizing the direct transfer from calibration accuracy to perception accuracy.

[0049] in the formula The conditional probability distribution represents the relationships between attributes. There are natural dependencies among the four attribute classes. The spatial location of the target affects the estimation of its motion state, and the semantic category affects the estimation of the contour boundary. Therefore, modeling the dependencies between attributes using conditional probability distributions enables mutual constraints and collaborative optimization among multiple attributes, improving the estimation accuracy of each attribute. In the formula... To observe the likelihood distribution, the error distribution between image observations and the true attributes of the target was modeled, which was obtained from the output of the attribute decoupling detection network.

[0050] 3.3 Variational Inference and Model Solving The true posterior distribution of Bayesian networks is often difficult to solve directly. Therefore, variational Bayesian inference is employed, which introduces an easily tractable variational distribution to approximate the true posterior distribution. This transforms the complex integral problem into a simple optimization problem, meeting the real-time requirements of in-vehicle systems. The introduced variational distribution is denoted as... Used to approximate the true posterior distribution Using the mean-field assumption, the variational distribution is decomposed into a product of four independent attribute variational distributions, specifically in the form of: The core objective of variational Bayesian inference is to minimize the KL divergence between the variational distribution and the true posterior distribution. This process is equivalent to maximizing the lower bound of evidence, ELBO. The specific formula for the lower bound of evidence is: ,in For variational distribution The mathematical expectation of the formula is the expectation of the logarithmic joint probability distribution with respect to the variational distribution. The first term is the expectation of the logarithmic joint probability distribution with respect to the variational distribution, and the second term is the information entropy of the variational distribution.

[0051] Iterative optimization is performed using the coordinate ascent method. In each iteration, the variational distributions of the other three attributes are fixed, while the variational distribution of one attribute is optimized. This process is repeated until the lower bound of evidence converges. The convergence criterion is that the difference between the lower bounds of evidence in two consecutive iterations is less than a preset threshold. After convergence, the mean of the variational distribution for each attribute is the optimal estimate of that target attribute; the variance of the variational distribution represents the uncertainty of the estimation result for that attribute, corresponding to the confidence level of the attribute. The smaller the variance, the higher the confidence level.

[0052] 3.4 Stratified Confidence Verification and Target Screening Based on the confidence levels of the four attribute categories obtained through variational inference, a hierarchical verification rule is established to filter detected targets, eliminate invalid targets, and distinguish between high-confidence and low-confidence targets, providing a basis for subsequent tracking and completion. The hierarchical verification rule specifically consists of hard constraint rules and soft constraint rules. The hard constraint rule is the highest priority filtering rule; if the confidence level corresponding to the semantic category attribute of a target is lower than 0.3, the target is directly determined as an invalid target and eliminated, not proceeding to the next processing stage. The soft constraint rule is a secondary filtering rule; for targets that pass the hard constraint rule, if any two of the confidence levels corresponding to their spatial location attribute, motion state attribute, and contour boundary attribute are lower than 0.5, the target is marked as a low-confidence target and proceeds to the subsequent tracking and verification stage; if all three confidence levels are higher than 0.5, the target is marked as a high-confidence target and directly retained.

[0053] 3.5 Multi-frame correlation target tracking and attribute completion Based on the target tracking trajectory and historical attribute values ​​output from the previous frame, a joint probabilistic data association algorithm is used to perform inter-frame trajectory association between high-confidence and low-confidence targets in the current frame, achieving continuous target tracking and attribute completion. For successfully associated low-confidence targets, a Kalman filter algorithm is used, combining the target's historical attribute values ​​and trajectory information to complete and correct the attribute estimates in the current frame, while updating the confidence of the corresponding attributes to reduce attribute estimation errors caused by occlusion and blurring. For new targets that fail to associate, a unique tracking ID is assigned, and the corresponding trajectory sequence and historical attribute values ​​are initialized. For targets that fail to associate for three consecutive frames, the target is determined to be a missing target, removed, and no longer included in subsequent processing.

[0054] 3.6 Model Training Process The training process of this algorithm consists of two stages: offline pre-training and online fine-tuning. In the offline pre-training stage, large-scale publicly available autonomous driving datasets and a self-built multi-scenario vehicle dataset are used to train the parameters of the attribute decoupling detection network, including the parameters of the observation likelihood distribution and the parameters of the conditional probability distribution between attributes. This enables the network to stably extract target attribute observations under different scenarios and model accurate probability distributions. In the online fine-tuning stage, during actual vehicle driving, based on real-time acquired image data and observations, the variational distribution parameters are continuously fine-tuned by maximizing the lower bound of evidence. This allows the model to adapt to the current driving scenario, lighting conditions, and environmental characteristics, continuously improving the estimation accuracy and reliability of target attributes.

[0055] The output of this step specifically includes: a global target set, containing the tracking ID of each retained target, the best estimated values ​​of the four types of attributes, and the corresponding confidence levels; a target tracking trajectory sequence, containing the historical trajectory and motion state changes of each target; a list of low-confidence targets and their corresponding area markers; and the variational distribution of target attributes. .

[0056] S4: Spatiotemporal Coupled Risk Field Modeling Based on Lattice Boltzmann Machine This step builds upon the global target set, target tracking trajectory sequence, and target attribute variational distribution output from step S3, and combines this with the global semantic mask output from step S2. The core objective is to construct a coupled risk field covering the entire spatiotemporal domain, addressing the common pain points of existing ADAS risk assessment schemes, such as discretized risk modeling, inability to characterize the spatiotemporal diffusion characteristics of risk, failure to consider the uncertainty of target attributes, and disconnection from the front-end perception and calibration stages. Simultaneously, the risk field gradient matrix output from this step will be fed back to the calibration optimization objective function of step S1, achieving reverse collaboration between the decision-making and calibration stages, and completing the bidirectional closed-loop fusion of the three core algorithms. The spatiotemporal coupled risk field modeling process is as follows: Figure 4 As shown.

[0057] 4.1 Spatiotemporal Dimensions and Grid Division of the Risk Field Using the vehicle's entire surface as the sole spatial reference, a spatiotemporally coupled risk field model is constructed. The spatial dimension completely covers the entire surface of the vehicle, while the temporal dimension is divided into four consecutive time slices: the current moment, the short-term prediction domain, the medium-term prediction domain, and the long-term prediction domain. The current moment is denoted as... The short-term forecast time domain is to ,in equal Add 0.5 seconds, the intermediate prediction time domain is to ,in equal Add 3 seconds, long-term prediction time domain is to ,in equal Add 5 seconds. The four time slices fully cover the decision-making needs of the ADAS system from emergency braking to path planning.

[0058] To achieve continuous modeling and efficient solution of the risk field, a lattice Boltzmann machine model is adopted. The spatial dimension is divided into uniformly distributed two-dimensional lattices, with each lattice having a step size of 0.1 meters in both the X and Y axes. This ensures that the spatial resolution of the risk field meets the accuracy requirements of vehicle-mounted decision-making. Each time slice in the time dimension corresponds to one time step of the lattice Boltzmann machine, with a time step size of 0.1 seconds. This ensures that the temporal resolution of the risk field meets the changing requirements of dynamic scenarios.

[0059] A two-dimensional, nine-speed D2Q9 lattice model is adopted, which achieves an optimal balance between computational efficiency and accuracy, fully meeting the real-time requirements of the vehicle system. For any lattice in space, its coordinates are: Corresponding time step The distribution function is denoted as ,in The D2Q9 model contains 9 discrete velocity directions, indexed by the discrete velocity directions. The value range is from 0 to 8, and the discrete velocity vector corresponding to each index is denoted as . The specific discrete velocity is defined as: when When equal to 0, for , corresponding to the stationary direction; when When the values ​​are 1, 2, 3, or 4, They are respectively , , , , corresponding to the four positive directions; when When the values ​​are 5, 6, 7, or 8, They are respectively , , , This corresponds to four diagonal directions. The lattice velocity is equal to the lattice space step size divided by the time step size. The value is 1 meter per second.

[0060] 4.2 Construction of Equilibrium Distribution Function for Integrating Multi-Source Information The core of the evolution of the lattice Boltzmann machine is the equilibrium distribution function, which determines the steady-state distribution of the risk field. By integrating three types of information—static environmental semantics, the attributes and uncertainties of dynamic targets, and the vehicle's own dynamic limits—into the equilibrium distribution function, a deep coupling of multi-source information is achieved, constructing a risk field model that comprehensively characterizes driving risks. First, the core representative quantity of the risk field is defined as the total risk potential function, which is obtained by fusing three parts: the static risk potential function, the dynamic risk potential function, and the dynamic constraint potential function. Each part corresponds to a type of risk source.

[0061] First, define the static risk potential function, denoted as . This function characterizes the driving risks posed by the static environment. Its value is determined based on the static semantic mask output in step S2. The assignment rules for different semantic categories are as follows: For non-drivable areas, including areas corresponding to guardrails, building walls, green belts, and curbs, The value is 1.0; for the region corresponding to a static obstacle, The value is 0.95 multiplied by the negative power of the natural exponential function, where the power is the distance from the grid to the nearest static obstacle divided by 5. The closer the distance, the higher the value of the static risk potential function. For the constraint area corresponding to traffic signs, including the road surface area corresponding to stop lines and speed limit signs, The value is 0.6; for areas outside the vehicle's lane within the drivable road surface, The value is 0.3 multiplied by the negative power of the natural exponential function, where the power is the distance from the grid cell to the center line of the vehicle's lane divided by 10; for the vehicle's lane area within the drivable road surface, The value is 0.05. The static risk potential function ranges from 0 to 1; the higher the value, the higher the static driving risk in that area.

[0062] The dynamic risk potential function is then defined, denoted as... This function characterizes the driving risk posed by dynamic targets, with its core input being the variational distribution of target attributes output from step S3. Conventional dynamic risk modeling only uses point estimates of target attributes, ignoring the uncertainties in the perception stage, which can easily lead to biased risk assessment. By directly incorporating the probability distributions of target position and motion state output from step S3 into the dynamic risk potential function, the perceptual uncertainty is transferred to the risk assessment throughout the entire process. For the m-th dynamic target, its corresponding dynamic risk sub-function is constructed. This function is the expectation based on the probability distribution of the target's position and motion state, and its specific form is: ;in and For the m-th target at time step The location coordinates are derived from the variational distribution of the spatial location attributes output by step S3. get; The risk diffusion coefficient is the variational distribution of the motion state attributes output from step S3. The mean value is calculated, and the faster the target moves, the larger the risk diffusion coefficient and the wider the corresponding risk impact range. The expectation operation is based on the variational distribution of the target's position and motion state, which can directly incorporate the uncertainty of target attributes into the risk value calculation. The higher the estimated uncertainty of the target attributes, the larger the risk diffusion range and the more conservative the risk assessment result, effectively avoiding decision-making risks caused by perception errors. The dynamic risk potential function of the entire domain is obtained by superimposing the dynamic risk sub-functions corresponding to all dynamic targets. .

[0063] Next, we define the dynamic constraint potential function, denoted as . This function characterizes the driving constraints imposed by the vehicle's own dynamic limits, preventing the decision-making process from outputting behavioral commands that exceed the vehicle's physical limits. Based on the vehicle's real-time dynamic parameters and combined with the vehicle's nonlinear two-degree-of-freedom dynamic model, the function calculates the vehicle's maximum steering capability, maximum braking capability, and maximum acceleration capability at the current vehicle speed and road adhesion coefficient. The road adhesion coefficient is estimated based on the drivable road surface semantic type output in step S2. Based on the vehicle's dynamic limits, a dynamically feasible region is generated within the prediction time domain, i.e., the area the vehicle can reach within its physical limits. For areas outside the dynamically feasible region... The value of is 1.0, representing an absolutely infeasible region; for regions within the dynamically feasible region, The value of is 0.2 multiplied by the negative power of the natural exponential function, where the power is the distance from the lattice to the boundary of the dynamic feasible region divided by 2. The closer to the boundary of the feasible region, the higher the value of the dynamic constraint potential function, reminding the decision-making process to avoid approaching the dynamic limit of the vehicle.

[0064] Finally, the three types of risk potential functions are fused to obtain the total risk potential function. The specific form is as follows: The use of maximum value fusion ensures that any type of risk is fully represented, preventing any risk omission. Based on the total risk potential function, the density function of the risk field is defined. Its value is 1 minus The density function ranges from 0 to 1; the smaller the value, the higher the driving risk corresponding to that grid cell. Simultaneously, the velocity field of the risk field is defined. The velocity field is obtained by weighting the mean of the probability distribution of the motion states of all dynamic targets, which characterizes the overall flow trend of the risk field and provides a basis for the spatiotemporal diffusion of risk.

[0065] Based on the density function and velocity field defined above, the equilibrium distribution function corresponding to the D2Q9 model is constructed, with the following specific form: ;in These are the weight coefficients corresponding to the D2Q9 model, specifically taking the following values: when When equal to 0, Equals 4 / 9; when When the values ​​are 1, 2, 3, and 4, the corresponding weighting coefficient is 1 / 9; when... When the values ​​are 5, 6, 7, and 8, the corresponding weighting coefficient is 1 / 36. The speed of sound in the lattice is given by the value of lattice velocity. Divide by the square root of 3. The equilibrium distribution function fully integrates information from the density function and the velocity field, and can accurately characterize the steady-state distribution of the risk field.

[0066] 4.3 Lattice Boltzmann Evolution Equation and Risk Field Gradient Calculation The evolution process of the lattice Boltzmann machine is divided into a collision step and a migration step. By iteratively executing these two steps, the risk field is evolved and diffused in the spatiotemporal dimension, resulting in the risk field distribution of different time slices.

[0067] First, a collision step is performed using the BGK collision operator. This operator significantly simplifies the collision process calculation and meets the real-time requirements of in-vehicle systems. The specific formula for the collision step is as follows: ;in The distribution function after the collision. The relaxation time is used to control the rate at which the risk field converges to the equilibrium state in this scheme. The initial value is 0.8. The smaller the relaxation time, the faster the risk field converges to the equilibrium state.

[0068] After the collision step is completed, the migration step is executed. The migration step moves the post-collision distribution function to adjacent cells according to the corresponding discrete velocity direction, realizing the spatial diffusion of the risk field. The specific formula for the migration step is as follows: ;in and Discrete velocity vectors Components on the X and Y axes For time step.

[0069] By iteratively executing the collision and migration steps for each of the four time slices, the density function for each cell within all time slices can be obtained. Then, the risk field value corresponding to each cell is calculated. The formula for calculating the risk field value is: The risk field value ranges from 0 to 1. The closer the value is to 1, the higher the driving risk at that location and time.

[0070] To achieve reverse collaboration between the backend decision-making stage and the frontend calibration stage, the risk field gradient matrix is ​​calculated based on the calculated risk field value at the current moment. This matrix represents the spatial rate of change of the risk field around the vehicle body, reflecting the location and extent of high-risk areas. The specific form of the risk field gradient matrix is: ;in , These are the first-order partial derivatives of the risk field value along the X and Y axes, respectively. The second-order mixed partial derivatives of the risk field value are given; all partial derivatives are calculated using the central difference method. The calculated risk field gradient matrix is ​​then obtained. This will be directly fed back to the calibration optimization objective function in step S1, giving higher weight to the feature point reprojection error in high-risk areas. This will encourage the calibration process to prioritize improving the perception accuracy of high-risk areas, achieving full-link reverse optimization from decision-making safety requirements to front-end calibration, and completing the bidirectional closed-loop fusion of the three core algorithms.

[0071] 4.4 Model Adaptive Adjustment Process The parameter adjustment of this algorithm is an online adaptive process, requiring no offline pre-training. During actual vehicle operation, the relaxation time is adaptively adjusted based on the current driving scenario, vehicle speed, number of targets, and motion state. With risk diffusion coefficient The value of is determined by factors such as vehicle speed and risk diffusion coefficient. For example, in high-speed cruising scenarios, the vehicle speed is higher and the impact range of the risk is wider. Therefore, the relaxation time and risk diffusion coefficient will be increased appropriately to make the risk field spread more widely and reserve more safety margin for the decision-making process. In urban low-speed following scenarios, the vehicle speed is lower and the environment changes more frequently. Therefore, the relaxation time will be reduced appropriately to enable the risk field to respond more quickly to the dynamic changes in the environment.

[0072] The output of this step specifically includes: a spatiotemporally coupled risk field, containing the risk field values ​​of all grid cells in the vehicle's global plane within four time slices; the boundary of the vehicle dynamics feasible domain; the probability distribution of the dynamic target trajectory for each time slice; and the risk field gradient matrix. This feedback is directly fed back to the calibration and optimization stage in step S1, achieving bidirectional collaborative optimization across the entire link.

[0073] S5: Multi-Constraint Hierarchical Decision Making and Desired Behavior Sequence Generation This step follows the output of step S4, which includes the spatiotemporal coupled risk field, the boundary of the vehicle dynamics feasible domain, and the probability distribution of the dynamic target trajectory. The core objective is to generate the optimal expected behavior sequence that is safe, comfortable, and in line with the driver's intentions under multiple constraints, thereby solving the problems of discretization of existing ADAS decision-making logic, poor scenario adaptability, and insufficient balance between safety and comfort.

[0074] 5.1 Hierarchical Definition of Decision Constraints The constraints that the decision-making process needs to meet are divided into three levels according to their priority from high to low: safety hard constraints, comfort soft constraints, and driving intention matching constraints. This ensures that the decision-making process always prioritizes driving safety, while also taking into account passenger comfort and the driver's operating intentions.

[0075] Safety hard constraints are the highest priority and cannot be broken. All expected behavior sequences of decision outputs must meet the requirements of safety hard constraints throughout the entire process. Specifically, these include: the risk field value corresponding to the vehicle's driving trajectory for the expected behavior sequence must be lower than 0.3 for all trajectory points in the entire prediction time domain; the driving trajectory must be within the boundary of the vehicle's dynamic feasible domain throughout the entire process; it must not enter the drivable area with a static risk value of 1.0; and the minimum distance to all dynamic targets must not be less than the safe distance dynamically calculated based on the relative speed and relative acceleration between the vehicle and the target.

[0076] Comfort soft constraints are medium-priority optimizable constraints. Under the premise of meeting safety hard constraints, they aim to minimize vehicle acceleration, deceleration, and steering impact to improve ride comfort. Specifically, these constraints include: the absolute value of the vehicle's longitudinal acceleration corresponding to the desired behavior sequence must not exceed 4 m / s², the absolute value of the longitudinal jerk must not exceed 5 m / s³, the absolute value of the lateral acceleration must not exceed 2.5 m / s², the absolute value of the lateral jerk must not exceed 3 m / s³, and the rate of change of the steering wheel angle must not exceed 30 degrees per second.

[0077] The driving intention matching constraint is a low-priority adaptable constraint. Under the premise of satisfying the first two types of constraints, the expected behavior sequence needs to match the driver's operation intention to the greatest extent and minimize the deviation between the expected behavior and the driver's input. The driver's operation intention signal includes the driver's steering wheel angle input, accelerator pedal input, brake pedal input, and turn signal.

[0078] 5.2 Driving Scene Recognition and Decision Weight Adaptation In different driving scenarios, drivers have significantly different needs for safety, comfort, and driving intention. Therefore, it is necessary to first identify the current driving scenario and then dynamically adjust the weights of constraints to improve the scenario adaptability of decision-making. Based on the global semantic mask output from step S2, the global target set output from step S3, and the real-time dynamic parameters of the vehicle, the current driving scenario is identified. The specific scenario categories are: urban road following scenario, urban road intersection scenario, highway cruising scenario, highway lane changing scenario, low-speed parking scenario, and emergency obstacle avoidance scenario. For different scenarios, the weights of comfort soft constraints and driving intention matching constraints are dynamically adjusted. For example, in emergency obstacle avoidance scenarios, the weight of comfort soft constraints is reduced to the minimum to prioritize driving safety; in highway cruising scenarios, the weight of comfort soft constraints is appropriately increased to ensure a comfortable ride on long journeys; when the driver activates the turn signal to prepare for a lane change, the weight of driving intention matching constraints is increased, while the risk verification of the lane change direction is strengthened.

[0079] 5.3 Expected Behavior Sequence Sampling and Rolling Optimization A model predictive control framework is employed to achieve rolling optimization of the desired behavior sequence. The control time domain of model predictive control is 1 second, and the prediction time domain is 5 seconds, perfectly matching the risk field time slice in step S4. Using the current vehicle motion state as the initial value, multiple sets of uniform samples are performed on the two core control variables—longitudinal acceleration and front wheel angle—within the prediction time domain to generate multiple candidate control sequences. Each candidate control sequence corresponds to a candidate driving trajectory and a set of candidate behavior sequences. For each set of candidate behavior sequences, a corresponding cost function is constructed based on three levels of constraints. The cost function is a weighted sum of safety constraint cost, comfort cost, and driving intention matching cost, with weight coefficients dynamically determined by the currently identified driving scenario. Through rolling optimization, the candidate behavior sequence with the minimum cost function value is selected as the optimal desired behavior sequence.

[0080] 5.4 Decoupling of Expected Behavioral Sequences The selected optimal expected behavior sequence is decoupled into two independent but coordinated continuous time series: a longitudinal expected behavior sequence and a lateral expected behavior sequence. The longitudinal expected behavior sequence covers the entire prediction time domain and includes the expected vehicle speed, expected longitudinal acceleration, expected braking pressure, and expected throttle opening, and is used for subsequent longitudinal actuator control. The lateral expected behavior sequence covers the entire prediction time domain and includes the expected front wheel angle, expected steering wheel angle, expected lateral offset, and expected yaw rate, and is used for subsequent lateral actuator control.

[0081] 5.5 Expected Behavioral Sequence Security Secondary Verification To avoid constraint omissions during optimization, the decoupled longitudinal and lateral expected behavior sequences undergo a full-time-domain safety hard constraint check to confirm that the corresponding driving trajectories fully meet all constraints regarding risk field value, safe distance, and dynamic feasible region. If the check fails, the sequence is discarded, and the sampling optimization process is restarted until an expected behavior sequence that fully complies with the safety hard constraints is generated, ensuring the absolute safety of the decision output.

[0082] The output of this step specifically includes: the optimal expected behavior sequence, which includes the decoupled longitudinal and lateral expected behavior sequences; the expected driving trajectory corresponding to the decision; the scene recognition result and constraint weight parameters.

[0083] S6: Actuator control quantity calculation and feedforward compensation This step follows the optimal desired behavior sequence output by step S5. The core objective is to convert the desired vehicle motion state into control commands that the vehicle actuators can directly execute. At the same time, through multi-dimensional feedforward compensation, it solves the problems of low tracking accuracy, slow response, and poor anti-interference ability in conventional ADAS control loops, ensuring that the vehicle can accurately track the desired behavior sequence.

[0084] 6.1 Construction of Inverse Vehicle Dynamics Model To convert the desired motion state into actuator control commands, a precise inverse vehicle dynamics model needs to be constructed. Separate longitudinal and lateral inverse dynamics models are built to achieve independent calculation and coordinated control of longitudinal and lateral control quantities. The longitudinal inverse dynamics model is built based on the vehicle's longitudinal dynamics equations, incorporating the pressure-deceleration mapping relationship of the vehicle's braking system and the torque-acceleration mapping relationship of the powertrain. The model's inputs are the desired longitudinal acceleration and the current real-time vehicle speed, and its outputs are the desired braking pressure and desired drive torque required to achieve that desired acceleration. The lateral inverse dynamics model is built based on the vehicle's nonlinear two-degree-of-freedom dynamics model, incorporating the transmission ratio characteristics of the vehicle's steering system. The model's inputs are the desired front wheel steering angle, the current real-time vehicle speed, and the current yaw rate, and its output is the desired steering wheel angle command required to achieve that desired front wheel steering angle.

[0085] 6.2 Longitudinal control variable calculation and multi-dimensional feedforward compensation First, based on the expected longitudinal acceleration and expected vehicle speed in the expected longitudinal behavior sequence, combined with the current vehicle speed and current longitudinal acceleration in the real-time vehicle dynamics parameters, the basic expected braking pressure and expected driving torque are calculated using a longitudinal dynamics inverse model. To compensate for the response lag of the vehicle's braking and power systems and improve the response speed and tracking accuracy of longitudinal control, a multi-dimensional feedforward compensation term is constructed to correct the basic control quantities. First, there is a feedforward compensation term based on acceleration. Based on the expected acceleration in the expected longitudinal behavior sequence and the future trend of expected acceleration changes, the control quantity is corrected in advance to compensate for the response lag of the execution system and avoid acceleration tracking delays and overshoot. Second, there is a slope resistance compensation term. Based on the difference between the longitudinal acceleration and the rate of change of vehicle speed in the real-time vehicle dynamics parameters, the slope of the current driving surface is estimated, and then the compensation torque or compensation pressure corresponding to the slope resistance is calculated to offset the influence of slope resistance on longitudinal acceleration and improve control accuracy in slope scenarios. Finally, by superimposing the basic control quantities and the feedforward compensation term, the final longitudinal actuator control commands are generated, including braking system control commands and power system control commands.

[0086] 6.3 Lateral Control Variable Calculation and Multi-Dimensional Feedforward Compensation First, based on the expected front wheel steering angle and expected yaw rate in the lateral expected behavior sequence, combined with the current steering wheel angle, current yaw rate, and current real-time vehicle speed in the vehicle's real-time dynamic parameters, the basic expected steering wheel angle command is calculated using a lateral dynamics inverse model. To compensate for steering system response lag, friction nonlinearity, and road surface disturbances, and to improve the tracking accuracy and response speed of lateral control, a multi-dimensional feedforward compensation term is constructed to correct the basic steering angle command. First, a feedforward compensation term based on the steering angle change rate, based on the steering wheel angle change rate in the lateral expected behavior sequence and the expected steering angle change trend at future moments, corrects the steering angle command in advance to compensate for steering system response lag and avoid steering tracking delay. Second, a friction feedforward compensation term, based on the friction characteristics of the steering system, compensates for frictional resistance during steering initiation and reversal phases, eliminating dead zones and hysteresis effects in the steering system. Finally, a road surface curvature feedforward compensation term, based on the lane line curvature in the static semantic mask output from step S2, corrects the steering command in advance, improving lateral tracking accuracy in curve scenarios. By superimposing the basic control quantity and the feedforward compensation term, the final lateral actuator control command, i.e. the electric power steering system control command, is finally generated.

[0087] 6.4 Control Command Boundary Constraints and Human-Machine Arbitration First, physical limit constraints are set for the generated longitudinal and lateral actuator control commands to ensure that the control commands are within the actuator's maximum stroke and maximum response rate range, preventing commands exceeding the actuator's capabilities and causing control failure. Then, human-machine arbitration logic is performed to prioritize the driver's operational intention signals and ADAS control commands. When the driver's active input exceeds a preset threshold, it is determined that the driver is taking over the vehicle, and the driver's operational commands are prioritized, suppressing ADAS control commands to ensure a reasonable allocation of driving authority and guarantee driving safety.

[0088] The output of this step specifically includes: longitudinal actuator control commands, including braking system control commands and power system control commands, which are directly output to the corresponding actuators of the vehicle; lateral actuator control commands, namely electric power steering system control commands, which are directly output to the vehicle steering actuators; and the desired vehicle state sequence corresponding to the control commands.

[0089] S7: Full-domain visual verification of execution results and closed-loop correction of end-to-end parameters This step is the core of the closed-loop optimization of the entire solution. It takes the output results of all steps S1 to S6, and quantifies the error of each link in the entire chain through full-domain visual verification of the execution effect and feedback of the actual vehicle status. It adaptively corrects the core parameters of all previous steps, solves the problems of open-loop operation of perception-decision-control in conventional ADAS systems, continuous accumulation of errors, and poor scenario adaptability, and realizes continuous iterative optimization of the entire solution, forming a complete closed loop of data flow and control flow.

[0090] 7.1 Obtaining the actual state of vehicle execution effect First, based on real-time vehicle dynamic parameters acquired via the vehicle's CAN bus, the actual motion state of the vehicle after the actuator executes control commands is obtained, including core motion parameters such as actual vehicle speed, actual longitudinal acceleration, actual lateral acceleration, actual steering wheel angle, and actual yaw rate. Simultaneously, based on the current frame image data newly acquired by the all-around camera group, a panoramic stitched image of the current frame is generated through the spatiotemporal synchronization and calibration logic in step S1. This extracts the vehicle's actual driving trajectory within the vehicle's entire horizontal plane, providing a benchmark for subsequent execution error quantification.

[0091] 7.2 Quantification of Control Execution Error and Correction of Control Model Parameters The acquired actual vehicle motion state and trajectory are compared with the desired trajectory output in step S5 and the desired vehicle state sequence output in step S6 to quantify core control error indicators such as longitudinal tracking error, lateral tracking error, and response lag time. Based on the quantified control errors, the recursive least squares method is used to correct the parameters of the inverse vehicle dynamics model and the coefficients of the feedforward compensation term in step S6 online. This compensates for model errors caused by changes in vehicle parameters, actuator aging, and road conditions, continuously improving the tracking accuracy and response speed of subsequent control stages.

[0092] 7.3 Decision Logic Verification and Constraint Weight Adjustment Based on the vehicle's actual driving trajectory, the risk value of that trajectory in the spatiotemporally coupled risk field output in step S4 is traced back to verify the safety and rationality of the expected behavior sequence generated in step S5. If the risk value of the actual driving trajectory exceeds expectations, or the trajectory tracking error exceeds a preset threshold, the weights of the three-level constraints and the coefficients of the cost function in step S5 are adaptively corrected based on the current scene recognition results to optimize the output of subsequent decision-making stages, making the decision logic more adaptable to the current scene and vehicle state.

[0093] 7.4 Perception effect verification and perception parameter correction Based on the panoramic stitched image and the global target set of the current frame, the target trajectory predicted in step S3 of the previous frame is compared with the actual position of the target in the current frame to quantify the target motion prediction error. For the regions corresponding to the low-confidence target list output by step S3, the priority of the perception region, the candidate region selection threshold, and the hierarchical confidence verification rules in step S3 are adjusted to improve the detection accuracy and attribute estimation reliability of low-confidence targets. Simultaneously, based on the inter-frame feature matching error of static regions, the static semantic mask prior in step S2 is updated to improve the accuracy and stability of subsequent semantic mask generation.

[0094] 7.3 Iterative Optimization of Calibration Parameters Based on the reprojection error of static feature points between the current frame and the previous frame, and combined with the actual motion state of the vehicle, the global intrinsic parameter matrix and global projection parameters in step S1 are iteratively optimized to generate calibration parameter correction values, which are fed back to step S1 of the next frame. This enables continuous online optimization of calibration parameters, continuously adapts to parameter changes caused by vehicle body deformation, temperature drift, and vibration, and ensures the spatial mapping accuracy of the entire process.

[0095] The correction parameters generated in each of the above steps are categorized and organized according to their corresponding preceding steps to generate all the input parameters required for the next frame iteration, ensuring that all correction parameters can be directly called by the corresponding steps in the next frame, thus completing the closed-loop iteration of the entire link.

[0096] All outputs of this step are fed back to the next frame iteration of the preceding step, specifically including: calibration parameter correction values, fed back to step S1 of the next frame for iterative optimization of calibration parameters; static semantic mask prior, fed back to step S2 of the next frame for generation of static semantic mask; target tracking trajectory and attribute history values, and perception parameter correction values, fed back to step S3 of the next frame for target tracking and attribute estimation; and decision constraint weight correction values, fed back to step S5 of the next frame for calculation of the cost function for decision optimization. The corrected values ​​of the dynamic inverse model parameters and the corrected values ​​of the feedforward compensation coefficients are fed back to step S6 of the next frame for the calculation and compensation of control quantities.

[0097] In this solution, steps S1 to S7 are executed cyclically, taking the vehicle's ADAS control cycle as the unit. All correction parameters output from step S7 of the previous frame are directly used as inputs for the corresponding steps in the next frame. Simultaneously, the algorithms for the three core steps S1, S3, and S4 achieve bidirectional interaction and deep fusion between each other through formulas, forming a complete data flow loop from front-end calibration to environmental perception, risk decision-making, execution control, and closed-loop correction. This ensures the robustness, accuracy, and adaptability of the solution across all scenarios and operating conditions.

Claims

1. An ADAS intelligent control method based on full-vehicle camera perception, characterized in that, Includes the following steps: S1: Combining the vehicle synchronization benchmark and the real-time dynamic parameters acquired by the vehicle CAN bus, the acquired multi-channel original images are spatiotemporally synchronized and jointly calibrated online to obtain time-aligned multi-channel images and global spatial mapping parameters. S2: Based on global spatial mapping parameters, complete the whole-domain planar pixel-level alignment and stitching of multi-path images to generate a panoramic stitched image. Perform static-dynamic dual-branch semantic segmentation on the panoramic stitched image to obtain the whole-domain semantic mask and the division results of static and dynamic regions. S3: Combining panoramic stitched images, global semantic masks, static and dynamic region segmentation results, and global spatial mapping parameters, the target candidate region is selected, the target full attribute decoupled perception and inter-frame tracking are completed, and the global target set and the uncertainty distribution of target attributes are obtained. S4: Based on the global semantic mask, the global target set and the uncertainty distribution of target attributes, combined with the real-time vehicle dynamics parameters, a global spatiotemporal coupling risk field covering multiple prediction time domains is constructed, and the boundary of the vehicle dynamics feasible domain is determined simultaneously. S5: Using the spatiotemporal coupling risk field of the vehicle body and the boundary of the feasible domain of vehicle dynamics as constraints, combined with the driver's operation intention signal collected by the vehicle CAN bus, candidate behavior sequences are generated through multi-constraint hierarchical optimization, and the optimal expected behavior sequence is obtained after safety verification. S6: Based on the optimal expected behavior sequence and combined with the real-time dynamic parameters of the vehicle, construct the longitudinal and lateral dynamic inverse model of the vehicle, complete the control quantity calculation and feedforward compensation, and generate the final vehicle actuator control command after boundary constraints and human-machine arbitration.

2. The ADAS intelligent control method based on full-vehicle camera perception according to claim 1, characterized in that, The online joint calibration of S1 is achieved using a calibration model based on Lie group and Lie algebra. The rigid body pose transformation of the camera relative to the center of mass of the vehicle body is modeled on a special Euclidean group SE (3). The rotation and translation parameters of the camera are represented by the Lie algebra elements corresponding to the special Euclidean group SE (3), thus avoiding the gimbaled singularity problem inherent in the Euler angle representation method. The linearized solution equation of the reprojection error is constructed based on the Lie algebra left-multiplied perturbation model. The globally optimal internal and external pose parameters of the camera are obtained through iterative optimization, and the corresponding global spatial mapping parameters are generated.

3. The ADAS intelligent control method based on full-vehicle camera perception according to claim 2, characterized in that, In the optimization objective of S1's online joint calibration, the risk field gradient constraint corresponding to the spatiotemporal coupled risk field of the entire vehicle body is introduced; based on the risk field gradient, the high-risk area in the entire vehicle body is located, and the image feature point reprojection error in the high-risk area is given a higher optimization weight. While ensuring the global calibration accuracy, the spatial mapping accuracy of the high-risk area is improved first.

4. The ADAS intelligent control method based on full-vehicle camera perception according to claim 1, characterized in that, S3's target full-attribute decoupled perception is achieved using a detection model based on variational Bayesian inference. The core attributes of the target are decoupled into four categories: spatial location attributes, motion state attributes, semantic category attributes, and contour boundary attributes. A Bayesian network model with multi-attribute association is constructed. A decomposable variational distribution is introduced to approximate the true posterior distribution of the target attributes. Iterative solutions are completed by maximizing the lower bound of evidence. The probability distribution and confidence of each attribute category are estimated independently, realizing the decoupled estimation and uncertainty quantification of target attributes.

5. The ADAS intelligent control method based on full-vehicle camera perception according to claim 4, characterized in that, In the S3 variational Bayesian inference detection model, the camera pose parameters and calibration uncertainty obtained from online joint calibration are used as the spatial prior distribution of the target's spatial location attributes; the pose accuracy information of the calibration process is directly integrated into the estimation process of the target attributes, realizing the positive transfer of calibration accuracy to perception accuracy.

6. The ADAS intelligent control method based on full-vehicle camera perception according to claim 1, characterized in that, The S4's full-domain spatiotemporal coupled risk field is constructed using a lattice Boltzmann machine model. A two-dimensional nine-speed lattice model is used to divide the full-domain plane and multiple prediction time domains into spatiotemporal lattice dimensions. The model's equilibrium distribution function is constructed by integrating the static environmental risk corresponding to the full-domain semantic mask, the dynamic target risk corresponding to the full-domain target set, and the dynamic constraint risk corresponding to the vehicle's real-time dynamic parameters. The spatiotemporal evolution of the risk field is completed through collision and migration steps, resulting in a continuous risk distribution covering the full-domain space and multiple prediction time domains, which characterizes the spatial diffusion and temporal evolution characteristics of driving risks.

7. The ADAS intelligent control method based on full-vehicle camera perception according to claim 6, characterized in that, In the process of constructing the equilibrium distribution function of S4, the uncertainty distribution of target attributes is incorporated into the calculation of dynamic target risk; based on the probability distribution of target position and motion state, the mathematical expectation of dynamic risk is solved, so that the risk field modeling fully covers the uncertainty of the perception link.

8. The ADAS intelligent control method based on full-vehicle camera perception according to claim 1, characterized in that, In the multi-constraint hierarchical optimization of S5, decision constraints are divided into safety hard constraints, comfort soft constraints, and driving intention matching constraints according to their priority from high to low. Among them, safety hard constraints are the highest priority constraints that cannot be broken. All candidate behavior sequences must meet the hard requirements of risk threshold, dynamic feasible region, and safe distance throughout the process. The weight coefficients of soft constraints are dynamically adjusted based on the current driving scenario recognition results to ensure driving safety while taking into account the matching degree between driving comfort and driver operation intention.

9. The ADAS intelligent control method based on full-vehicle camera perception according to claim 1, characterized in that, The S6's feedforward compensation includes jerk feedforward compensation and slope resistance compensation in the longitudinal control dimension, as well as angular change rate feedforward compensation, steering friction feedforward compensation, and road curvature feedforward compensation in the lateral control dimension. These are used to compensate for the response lag, nonlinear characteristics, and external road disturbances of the vehicle's actuator system.

10. The ADAS intelligent control method based on full-vehicle camera perception according to claim 1, characterized in that, The boundary constraints of S6 are physical limit boundary constraints for the actuator, used to ensure that the output control commands are within the range of the maximum stroke and maximum response rate of the vehicle actuator; Human-machine arbitration is used to prioritize responding to driver operation commands when the driver's active input exceeds a preset threshold.

Images