A binocular vision road width measurement method and system oriented to vehicle-road cloud cooperation

Abstract

The application discloses a binocular vision road width measurement method and system facing vehicle-road cloud cooperation and belongs to the technical field of intelligent transportation. The method comprises the following steps: collecting road images based on a vehicle-mounted binocular camera; updating the external parameter of the camera based on roadside absolute coordinate information when a first preset condition is met; extracting features from the road images based on the external parameter of the vehicle-mounted binocular camera to generate a semantic segmentation graph and a depth graph of the road images; calculating the original measurement value of the road width based on the semantic segmentation graph and the depth graph; evaluating the reliability of current visual perception to generate an environment confidence coefficient; dynamically adjusting the fusion weight of the original width value based on the environment confidence coefficient; fusing the original width value and a road width prior value based on the fusion weight to generate the final measurement value of the road width; and uploading abnormal data containing the final measurement value to the cloud for cluster analysis when a second preset condition is met. The application can output a high-robustness final measurement value of the road width.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation technology, and in particular relates to a binocular vision road width measurement method and system for vehicle-road-cloud collaboration. Background Technology

[0002] In the vehicle-road-cloud collaborative autonomous driving system, the static reference elements of the dynamic high-precision map (such as road width and lane line position) are the core foundation for ensuring the safe passage of vehicles. Among them, the measurement accuracy of road width directly affects the accuracy of lane-level path planning and the decision-making safety in scenarios such as meeting oncoming traffic on narrow roads and overtaking on curves.

[0003] Currently, road width measurement mainly relies on technologies such as manual surveying, lidar scanning, and binocular vision measurement. Among these, binocular vision technology has become one of the mainstream solutions for roadside and vehicle-side perception due to its advantages such as low cost and flexible deployment.

[0004] However, existing road width measurement technologies based on binocular vision have the following drawbacks:

[0005] (1) Poor adaptability to complex scenes: Traditional technologies rely on grayscale threshold segmentation or edge detection algorithms to extract edge features such as curbs and lane markings. However, in scenes such as rainy weather with water reflection, nighttime backlight, vegetation occlusion, or dense vehicle occlusion, the feature point mismatch rate is high, resulting in an overestimation of the width measurement value.

[0006] (2) Long-term operation calibration drift is serious: The intrinsic parameters (focal length, distortion coefficient) and extrinsic parameters (relative position, attitude) of the binocular camera are easily affected by environmental factors and may shift—vehicle vibration during vehicle deployment can cause the camera mounting bracket to loosen. Traditional offline calibration methods require periodic shutdowns to disassemble the equipment for calibration, which not only has high maintenance costs but also cannot meet the needs of continuous operation of the vehicle-road-cloud system, resulting in the accumulation of measurement errors over time.

[0007] (3) Blind spots exist in the measurement range: The coverage radius of a single fixed binocular camera is only 50-80 meters. Blind spots are easily formed in areas where the line of sight is blocked, such as tunnel entrances and sharp bends, and continuous road width data cannot be provided.

[0008] (4) Disconnection with cloud map integration: Most existing binocular vision measurement systems are independent working units, and the measurement data is directly stored or output locally, without forming a synergy with the "static benchmark-dynamic update" architecture of dynamic high-precision maps. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a binocular vision road width measurement method and system for vehicle-road-cloud collaboration. It can output a highly robust final measurement value of road width and trigger cloud map updates based on deviations, providing high-precision data support for autonomous vehicle path planning and roadside traffic collaborative management.

[0010] This invention is achieved through the following technical solution:

[0011] The first aspect of this invention discloses a binocular vision road width measurement method for vehicle-road-cloud cooperation, comprising:

[0012] Road images are acquired and corrected using vehicle-mounted binocular cameras;

[0013] When the first preset condition is met, the extrinsic parameters of the vehicle-mounted binocular camera are calibrated and updated based on the acquired roadside absolute coordinate information;

[0014] Based on the extrinsic parameters of the vehicle-mounted binocular camera, features are extracted from the road image using a semantic-geometric joint perception algorithm based on Swin Transformer, and semantic segmentation map and depth map of the road image are generated.

[0015] Based on the semantic segmentation map and depth map, the original measurement value of the road width is obtained through three-dimensional back projection and spatial geometric calculation;

[0016] Assess the reliability of current visual perception and generate an environmental confidence coefficient;

[0017] Based on the environmental confidence coefficient, the fusion weight of the original width value is dynamically adjusted;

[0018] Based on the fusion weights, the original width value and the prior road width value in the vehicle map are fused to generate the final measured value of the road width;

[0019] When the second preset condition is met, the abnormal data containing the final measurement value will be uploaded to the cloud so that the cloud can perform cluster analysis on the abnormal data, and when the cluster analysis result meets the third preset condition, the cloud will send out the vehicle map update data.

[0020] Furthermore, road images are acquired and corrected based on vehicle-mounted binocular cameras, including:

[0021] Road images are acquired using an onboard binocular camera, and the road images include a left view and a right view;

[0022] The distortion correction of the left and right views was performed using the Zhang Zhengyou calibration method.

[0023] The corrected left and right views are then subjected to grayscale normalization.

[0024] Furthermore, when the first preset condition is met, the extrinsic parameters of the vehicle-mounted binocular camera are calibrated and updated based on the acquired roadside absolute coordinate information, including:

[0025] When a vehicle enters the coverage area of ​​a roadside unit, it receives and stores the world coordinate set of multiple absolute anchor points broadcast by the roadside unit, and records the receiving timestamp.

[0026] The pre-trained YOLOv8 detection model identifies the physical landmarks in the left view that correspond to the anchor points broadcast by the roadside unit.

[0027] Detect the anchor point pixel coordinates corresponding to physical landmarks, and filter out the anchor point pixel coordinates with a detection confidence greater than the first threshold to form a pixel coordinate set corresponding to the world coordinate set;

[0028] Based on the pinhole camera model, a mapping formula from world coordinates to pixel coordinates is established;

[0029] The mapping formula is transformed to obtain the projection function;

[0030] A nonlinear least squares optimization objective function is constructed based on the reprojection error, wherein the reprojection error is the Euclidean distance between the observed pixel coordinates and the projected pixel coordinates;

[0031] The rotation matrix is ​​mapped to the Lie algebra space and expressed as a perturbation, and an extrinsic parameter update model is established by representing the perturbation.

[0032] With the goal of minimizing the reprojection error, the Levenberg-Marquardt algorithm is used to iteratively optimize the perturbation until the convergence condition is met, and the optimized perturbation is obtained.

[0033] Based on the optimized perturbation, the updated extrinsic parameter matrix is ​​calculated using the extrinsic parameter update model, and the validity of the updated extrinsic parameters is verified. If valid, the extrinsic parameter update is performed; otherwise, the original extrinsic parameters are maintained and the anomaly is recorded.

[0034] Furthermore, based on the extrinsic parameters of the vehicle-mounted binocular camera, features are extracted from the road image using a semantic-geometric joint perception algorithm based on the Swin Transformer, and semantic segmentation map and depth map of the road image are generated, including:

[0035] Input the left view into the pre-trained Swin Transformer network;

[0036] The Swin Transformer network divides the input left view into multiple image patches;

[0037] Each image patch is converted into a feature vector through linear projection, and location information is added to form an initial feature map;

[0038] The initial feature map is transformed by a feature extraction structure comprising multiple consecutive processing stages to generate and output an image feature map; wherein, at least one processing stage employs a moving window self-attention mechanism to process the feature map input to that processing stage;

[0039] Global average pooling is performed on the image feature map to obtain channel-level feature vectors;

[0040] The channel-level feature vectors are nonlinearly transformed through a multilayer perceptron, and the output is an initial weight coefficient with a dimension equal to the number of channels in the image feature map.

[0041] Calculate the vertical gradient value of each pixel in the image feature map;

[0042] The initial weight coefficients are corrected based on the vertical gradient graph;

[0043] The corrected weight coefficients are multiplied element-wise with each channel of the image feature map to obtain the enhanced feature map.

[0044] Furthermore, based on the semantic segmentation map and depth map, the original measurement value of the road width is obtained through three-dimensional back projection and spatial geometric calculation, including:

[0045] Extract edge pixels from the semantic segmentation map, and filter out the left edge pixel set and the right edge pixel set respectively;

[0046] Based on the depth map, the road edge pixels are back-projected from the image coordinate system to the camera's 3D coordinate system to obtain the left road edge point cloud set and the right road edge point cloud set;

[0047] Based on the left-side point cloud set and the right-side point cloud set, the RANSAC algorithm is used to fit the left-side spatial line and the right-side spatial line respectively.

[0048] Define the vehicle front axle normal plane: with the center of the vehicle front axle as the origin, the vehicle's driving direction as the X-axis, the horizontal direction perpendicular to the driving direction as the Y-axis, and the direction perpendicular to the ground as the Z-axis, construct the vehicle coordinate system, and the vehicle front axle normal plane is the YZ plane;

[0049] Calculate the projected straight lines on the vehicle's front axle plane along the spatial straight lines of the left and right roads, respectively.

[0050] Calculate the Euclidean distance between the two projected lines; this Euclidean distance is the original width of the road.

[0051] Furthermore, the reliability of the current visual perception is assessed, and an environmental confidence coefficient is generated, including:

[0052] The left view is converted to grayscale to obtain a grayscale image;

[0053] Statistical histogram distribution of grayscale images;

[0054] Calculate the probability distribution of the histogram based on the histogram distribution of the grayscale image;

[0055] The brightness histogram entropy of an image is calculated based on the probability distribution of the histogram.

[0056] Normalize the brightness histogram entropy of the image to the range [0,1];

[0057] Based on the left and right views acquired by the binocular camera, calculate the forward disparity map from the left view to the right view and the reverse disparity map from the right view to the left view, respectively.

[0058] For the corresponding pixel positions in the forward disparity map and the reverse disparity map, extract their forward disparity value and reverse disparity value, and calculate the absolute value of the difference between the two. If the absolute value is less than the preset consistency threshold, the disparity consistency verification of the corresponding pixel is determined to be passed.

[0059] Calculate the parallax consistency pass rate;

[0060] The environmental confidence coefficient is obtained by weighted summation of luminance histogram entropy and disparity consistency pass rate.

[0061] Furthermore, based on the environmental confidence coefficient, the fusion weights of the original width value are dynamically adjusted, including:

[0062] The vehicle-end observation noise variance is dynamically constructed based on the environmental confidence coefficient. The formula for calculating the vehicle-end observation noise variance is as follows:

[0063]

[0064] In the formula, Based on the variance, λ is the adjustment coefficient, and λ is the environmental confidence coefficient. Variance of noise observed at the vehicle end;

[0065] Based on the observed noise variance at the vehicle end, the fusion weights are obtained using the Kalman gain calculation formula. The formula for calculating the fusion weights is as follows:

[0066]

[0067] In the formula, To integrate weights, To predict covariance, The variance of the noise observed at the vehicle end.

[0068] Furthermore, the formula for calculating the final measured value is as follows:

[0069]

[0070] In the formula, For the final measurement value, This is the original width value. The road width is a priori value. For weight fusion.

[0071] Furthermore, the cloud performs cluster analysis on the abnormal data, and when the cluster analysis results meet the third preset condition, the cloud sends out updated vehicle map data, including:

[0072] Grouping abnormal data from multiple vehicles according to road segment ID and timestamp;

[0073] Invalid and abnormal data are filtered based on preset rules;

[0074] Cluster analysis is performed on abnormal information of the same road segment within a set time window to determine the true width of the road segment.

[0075] When the difference between the actual road width and the prior road width is greater than the sixth threshold, the prior road width is updated to the actual road width, and the update timestamp and data source are recorded.

[0076] The system sends map update commands to vehicles within the preset range of the road section and surrounding areas via OTA, so that the vehicles can update the locally cached map data in real time after receiving the commands.

[0077] The clustering analysis results and map update status are fed back to the vehicles that participated in uploading abnormal data, so that the vehicles can adjust the weight of the prior prediction values ​​when calculating the final measurement value based on the feedback results.

[0078] A second aspect of the present invention discloses a binocular vision road width measurement system for vehicle-road-cloud cooperation, comprising:

[0079] The image acquisition module is used to acquire and correct road images based on an onboard binocular camera;

[0080] The extrinsic parameter update module is used to calibrate and update the extrinsic parameters of the vehicle-mounted binocular camera based on the acquired roadside absolute coordinate information when the first preset condition is met.

[0081] The joint perception module is used to extract features from the road image based on the extrinsic parameters of the vehicle-mounted binocular camera and the semantic-geometric joint perception algorithm based on the Win Transformer, and generate a semantic segmentation map and a depth map of the road image.

[0082] The road width calculation module is used to obtain the original measured value of the road width based on the semantic segmentation map and depth map through three-dimensional back projection and spatial geometric calculation;

[0083] The confidence assessment module is used to evaluate the reliability of the current visual perception and generate an environmental confidence coefficient.

[0084] The weight adjustment module is used to dynamically adjust the fusion weight of the original width value based on the environmental confidence coefficient.

[0085] The data fusion module is used to fuse the original width value and the prior value of road width in the vehicle map based on the fusion weight to generate the final measured value of road width.

[0086] The cloud update module is used to upload abnormal data containing the final measurement value to the cloud when the second preset condition is met, so that the cloud can perform cluster analysis on the abnormal data, and send the vehicle map update data to the cloud when the cluster analysis result meets the third preset condition.

[0087] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0088] (1) The edge detection success rate of traditional CNN-based methods is only about 60%, while this invention captures global features through SwinTransformer and combines vertical gradient weighting to form a "semantic-geometric" joint perception mechanism, which can suppress horizontal texture interference and get rid of the dependence of traditional pixel-level edge detection on texture. The edge detection success rate in rainy, backlit, and occluded scenarios is increased to about 90%, which enhances the perception robustness in complex scenarios.

[0089] (2) This invention uses roadside absolute anchor points and the LM algorithm (Lievenberg-Marquardt algorithm) to automatically calibrate the camera's extrinsic parameters online in real time to eliminate physical drift. It can continuously correct the extrinsic parameters without stopping disassembly, maintain high precision for long-term operation, and the error can still be controlled within 5cm after 6 months of continuous operation.

[0090] (3) The coverage radius of the roadside unit (RSU) in this invention is 300-500 meters. By supplementing the blind spot at the vehicle end with roadside data, the data interruption in areas such as tunnel entrances and sharp bends is effectively avoided, and continuous measurement without blind spots is achieved.

[0091] (4) This invention dynamically adjusts the fusion weights by environmental entropy, so that the effective data utilization rate under severe weather conditions can reach more than 95%;

[0092] (5) The present invention uses the DBSCAN clustering algorithm in the cloud to process crowdsourced data and update high-precision maps in real time, which can shorten the map update delay to less than 5 minutes, thereby constructing an efficient data processing and optimization closed loop of "real-time perception-intelligent fusion-rapid update". Attached Figure Description

[0093] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0094] Figure 1 This is a schematic flowchart of a binocular vision road width measurement method in this invention;

[0095] Figure 2 This is a block diagram of one component of the binocular vision road width measurement system in this invention. Detailed Implementation

[0096] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0097] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0098] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "inner," and "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the accompanying drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0099] In the description of this invention, unless otherwise explicitly specified and limited, the term "connection" or similar designation indicating a connection between components should be interpreted broadly. For example, it can refer to a fixed connection, a detachable connection, or an integral part; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can refer to the internal communication between two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0100] like Figures 1 to 2 As shown in the figure, this embodiment discloses a binocular vision road width measurement method and system for vehicle-road-cloud collaboration.

[0101] The first aspect of this embodiment discloses a binocular vision method for measuring road width in a vehicle-road-cloud cooperative manner, such as... Figure 1 As shown, the binocular vision road width measurement method includes steps S100 to S800.

[0102] Step S100. Acquire and correct road images based on vehicle-mounted binocular cameras.

[0103] In some embodiments of this example, road images are acquired and corrected based on vehicle-mounted binocular cameras, including steps S110 to S130.

[0104] Step S110. Acquire road images based on the vehicle-mounted binocular camera, the road images including a left view and a right view.

[0105] For example, the left and right views are captured simultaneously at a resolution of 1920×1080 and a frame rate of 30fps to ensure that the exposure parameters (such as shutter speed, ISO, etc.) of the left and right cameras are consistent.

[0106] Step S120. Perform distortion correction on the left and right views using the Zhang Zhengyou calibration method.

[0107] For example, based on the intrinsic parameter matrix K (including focal length) of the binocular camera as specified by the manufacturer. Principal point coordinates radial distortion coefficient (etc.), using the distortion correction formula of Zhang Zhengyou calibration method to eliminate radial and tangential distortion in the left and right views.

[0108] The formula for radial distortion correction is:

[0109]

[0110] In the formula, These are the corrected pixel coordinates; These are the coordinates of the distorted pixels; ; is the radial distortion coefficient.

[0111] The formula for tangential distortion correction is:

[0112]

[0113] In the formula, These are the corrected pixel coordinates; These are the coordinates of the distorted pixels; ; denoted as the tangential distortion coefficient.

[0114] Step S130. Perform grayscale normalization on the corrected left and right views.

[0115] In these embodiments, by performing grayscale normalization on the left and right views, the pixel values ​​of the left and right views are mapped to the range of [0, 255], thus eliminating pixel value fluctuations caused by differences in illumination intensity.

[0116] Step S200. When the first preset condition is met, the extrinsic parameters of the vehicle-mounted binocular camera are calibrated and updated based on the acquired roadside absolute coordinate information.

[0117] In some embodiments of this example, when the first preset condition is met, the extrinsic parameters of the vehicle-mounted binocular camera are calibrated and updated based on the obtained roadside absolute coordinate information, including steps S210 to S290.

[0118] Step S210. When the vehicle enters the coverage area of ​​the roadside unit, receive and store the world coordinate set of multiple absolute anchor points broadcast by the roadside unit, and record the receiving timestamp.

[0119] The first preset condition is that the vehicle enters the coverage area of ​​the roadside unit, that is, when the vehicle enters the coverage area of ​​the roadside unit, the first preset condition is met.

[0120] For example, the vehicle-mounted C-V2X OBU module receives wireless signals broadcast by roadside units (RSUs) in real time and determines whether the vehicle has entered the coverage area of ​​the roadside unit (signal strength ≥ -85dBm, coverage radius 300-500 meters) through signal strength detection (RSSI). The coverage radius of the wireless signals broadcast by the roadside unit (RSU) can reach 300-500 meters. When the signal strength received by the vehicle-mounted C-V2X OBU module is greater than the corresponding threshold (e.g., -85dBm), it is considered that the vehicle has entered the coverage area of ​​the roadside unit.

[0121] After the roadside unit is powered on, it broadcasts a set of world coordinates for n absolute anchor points at a frequency of 10Hz via the PC5 interface (direct communication interface). Anchor points are selected from fixed landmarks such as the center of the manhole cover and the corner of the stop line. The coordinates of the anchor points are calibrated by professional surveying equipment. The coordinate accuracy of the anchor points is ≤2cm. The world coordinate system adopted is the WGS-84 coordinate system.

[0122] Step S220. Identify the physical landmarks in the left view that correspond to the anchor points broadcast by the roadside unit based on the pre-trained YOLOv8 detection model.

[0123] These implementations can also be based on the right view, and the left view in subsequent related steps can be modified accordingly to the right view.

[0124] The YOLOv8 detection model has an input image size of 640×640 and a confidence threshold of 0.7.

[0125] The physical landmarks include the center of the manhole cover and the corner points of the stop line, etc.

[0126] Step S230. Detect the anchor point pixel coordinates corresponding to the physical landmarks, and filter out the anchor point pixel coordinates with a detection confidence greater than the first threshold to form a pixel coordinate set corresponding to the world coordinate set.

[0127] For the center anchor point of the manhole cover: the Hough circle detection algorithm is used to fit the edge of the manhole cover, and the center coordinates are the pixel coordinates of the anchor point.

[0128] For the corner anchor points of the stop line: the Canny edge detection algorithm is used to extract the edge of the stop line, and then the corner coordinates are located by the corner detection algorithm (Shi-Tomasi algorithm), which are the pixel coordinates of the anchor points.

[0129] For example, anchor pixel coordinates with a detection confidence level ≥ 0.8 are selected to form a pixel coordinate set. , This represents the set of detected anchor point pixel coordinates, where u represents the pixel column coordinate and v represents the pixel row coordinate, ensuring that the pixel coordinates are consistent with the world coordinate set P. w One-to-one correspondence (matched by anchor ID).

[0130] Step S240. Based on the pinhole camera model, establish a mapping formula from world coordinates to pixel coordinates.

[0131] The mapping formula is:

[0132]

[0133] In the formula, s is the scale factor (unknown). pixel coordinate vector K is the camera intrinsic parameter matrix, R is the rotation matrix (3*3 size), and T is the translation vector (3×1 size). Homogeneous vector of world coordinates u represents the column coordinate of a pixel, and v represents the row coordinate of a pixel. These represent the horizontal, vertical, and height coordinates in a Gaussian coordinate system, respectively; T represents transpose, and its core function is to exchange the row and column dimensions of the vector.

[0134] Step S250. Transform the mapping formula to obtain the projection function.

[0135]

[0136] In the formula, These are elements in the rotation matrix R. For elements in the translation vector T, The coordinates of the main point.

[0137] Step S260. Construct a nonlinear least squares optimization objective function based on the reprojection error, wherein the reprojection error is the Euclidean distance between the observed pixel coordinates and the projected pixel coordinates.

[0138] The formula for calculating the reprojection error is:

[0139]

[0140] In the formula, For reprojection error, These are the projected pixel coordinates. The coordinates of the observed pixels.

[0141] The nonlinear least squares optimization objective function is:

[0142]

[0143] In the formula, ε=1e-6 is the smoothing parameter of the robust loss function.

[0144] The objective of the nonlinear least squares optimization function is to minimize F(R,T) and solve for the optimal rotation matrix R and translation vector T.

[0145] Step S270. Map the rotation matrix to the Lie algebra space and represent it as a perturbation, and establish an extrinsic parameter update model represented by the perturbation.

[0146] The disturbance is a 6-dimensional vector, where the first 3 dimensions are rotation vectors and the last 3 dimensions are translation vectors.

[0147] The extrinsic parameter update model is as follows:

[0148]

[0149] In the formula, exp (ξ∧) Let be the Lie algebraic exponential mapping, ∧ denote the transformation from a vector to an antisymmetric matrix, and ξ be the perturbation. , The extrinsic parameter matrix of the current camera (including the current rotation matrix) and the current translation vector The current rotation matrix is ​​3×3 in size and is used to describe the camera pose. The optimized external parameters.

[0150] Step S280. With the goal of minimizing the reprojection error, the Levenberg-Marquardt algorithm (LM algorithm) is used to iteratively optimize the perturbation until the convergence condition is met, and the optimized perturbation is obtained.

[0151] The process of iteratively optimizing the perturbation using the Levenburg-Marquardt algorithm includes steps S281 to S289.

[0152] Step S281. Initialize the iteration parameters, which include the disturbance amount, damping coefficient, number of iterations, maximum number of iterations, and error convergence threshold.

[0153] For example, initialize the iteration parameters: disturbance ξ0=0, damping coefficient =1e-3, iteration number k=0, maximum iteration number=50, error convergence threshold=1e-5.

[0154] Step S282. Calculate the Jacobian matrix for the current iteration.

[0155] The Jacobian matrix is ​​6×2n dimensional, and each row corresponds to the partial derivative of an external parameter perturbation with respect to a projection error.

[0156] Step S283. Calculate the error vector.

[0157] Step S284. Solve the incremental equation: In the formula, I is the identity matrix. The damping coefficient is... This represents the increment of the perturbation in the k-th iteration. For Jacobian matrices, This is the error vector.

[0158] Step S285. Update the perturbation: And calculate the new extrinsic parameters. and calculating new errors ,in, This represents the optimized reprojection error. This represents the optimized camera rotation matrix. This represents the optimized camera translation vector.

[0159] Step S286. If Then accept the increment, let At the same time, reduce the damping coefficient (e.g., divide the damping coefficient by 10); if If the increment is rejected, the damping coefficient is increased (e.g., the damping coefficient is multiplied by 10).

[0160] Step S287. Repeat steps S281 to S286 until the difference between the current error and the new error is less than the error convergence threshold or the maximum number of iterations is reached.

[0161] Step S290. Based on the optimized perturbation amount, calculate the updated extrinsic parameter matrix through the extrinsic parameter update model, and verify the validity of the updated extrinsic parameters. If valid, perform extrinsic parameter update; otherwise, maintain the original extrinsic parameters and record the anomaly.

[0162] The method for validating the updated extrinsic parameters is as follows: perform a security check on the optimized extrinsic parameters, calculate the average reprojection error of all anchor points, and if the average error is below the second threshold (e.g., the second threshold is 3cm), then the calibration is deemed valid and the camera extrinsic parameter matrix in the vehicle system is overwritten in real time; if the average error is greater than the second threshold, then the calibration is deemed invalid, the original extrinsic parameters are retained and a calibration anomaly log (including timestamp, vehicle position, and error value) is recorded.

[0163] Step S300. Based on the extrinsic parameters of the vehicle-mounted binocular camera, extract features from the road image using a semantic-geometric joint perception algorithm based on Swin Transformer, and generate a semantic segmentation map and a depth map of the road image.

[0164] In some embodiments of this example, features are extracted from the road image using a semantic-geometric joint perception algorithm based on Swin Transformer, including steps S310 to S320.

[0165] Step S310. Input the left view into the pre-trained Swin Transformer network for feature extraction and generate an image feature map.

[0166] The process of the Swin Transformer network extracting features from the input image (such as the left view) includes steps S311 to S313.

[0167] Step S311. Divide the input image into multiple image blocks.

[0168] For example, the input image can be segmented into non-overlapping image patches of size 4×4 pixels.

[0169] Step S312. Convert each image patch into a feature vector through linear projection, and add position information to form an initial feature map.

[0170] For example, each 4×4 pixel image patch is converted into a 96-dimensional feature vector; a learnable 16-dimensional positional encoding vector is concatenated with the 96-dimensional feature vector of each 4×4 image patch; the positional encoding vector is iteratively optimized as the network is trained, and finally fused with the feature vector to form an initial feature map.

[0171] In these implementations, by adding positional information to the feature vectors, the model can perceive the spatial positional relationship of pixels, avoid the loss of positional information due to image block segmentation, and improve the accuracy of global feature capture.

[0172] Step S313. Transform the initial feature map using a feature extraction structure that includes multiple consecutive processing stages to generate and output an image feature map; wherein, at least one processing stage employs a moving window self-attention mechanism to process the feature map input to that processing stage.

[0173] For example, the feature extraction structure includes four processing stages, referred to as the first processing stage, the second processing stage, the third processing stage, and the fourth processing stage, respectively. The first processing stage, the second processing stage, the third processing stage, and the fourth processing stage contain 2-layer, 2-layer, 18-layer, and 2-layer network structures, respectively. The number of output feature channels of the first processing stage, the second processing stage, the third processing stage, and the fourth processing stage are 96, 192, 384, and 768, respectively.

[0174] In the feature extraction structure, the feature map output from the previous processing stage serves as the input to the next processing stage. For example, the input to the first processing stage is the initial feature map, and the output of the first processing stage is the first feature map; the input to the second processing stage is the first feature map, and the output of the second processing stage is the second feature map; the input to the third processing stage is the second feature map, and the output of the third processing stage is the third feature map; the input to the fourth processing stage is the third feature map, and the output of the fourth processing stage is the image feature map.

[0175] The process of processing feature maps using a moving window attention mechanism includes dividing the feature map into multiple windows and calculating self-attention within each window.

[0176] For example, the feature map can be divided into 7×7 windows.

[0177] The formula for calculating self-attention is:

[0178]

[0179] In the formula, Q represents the query matrix, K represents the key matrix, and V represents the value matrix; The feature dimension is represented by 96 / 192 / 384 / 768, corresponding to different stages; B represents the relative position deviation (pre-calculated based on the relative coordinates of pixels within the window).

[0180] In some implementations of this embodiment, cross-window feature interaction is achieved by changing the window's partition position between at least two adjacent processing stages.

[0181] Specifically, cross-window feature interaction is achieved by changing the window division position, including: dividing the feature map of the current stage into fixed 7×7 windows, and calculating self-attention (local feature extraction) in each window; between adjacent stages, moving the window 3.5 pixels along the width of the feature map (step size is half the window size), and calculating self-attention again (cross-window feature interaction); concatenating the results of the two self-attention calculations along the channel dimension to obtain a feature map that integrates local and global information.

[0182] In these implementations, cross-window feature interaction is achieved by changing the window's division position, taking into account both local texture details and global context information, and avoiding the omission of cross-region feature associations by traditional fixed windows; compared with global self-attention, this method increases the amount of computation less, and balances accuracy and real-time performance.

[0183] Step S320. Perform adaptive channel weighting processing based on channel statistics and spatial gradient features on the image feature map to obtain an enhanced feature map.

[0184] In some embodiments of this example, adaptive channel weighting processing based on channel statistics and spatial gradient features is performed on the image feature map to obtain an enhanced feature map, including steps S321 to S325.

[0185] Step S321. Perform global average pooling on the image feature map to obtain channel-level feature vectors.

[0186] Step S322. The channel-level feature vector is nonlinearly transformed through a multilayer perceptron to output initial weight coefficients with dimensions equal to the number of channels in the image feature map.

[0187] For example, the multilayer perceptron includes two fully connected networks, where the number of neurons in the hidden layer is 1 / 4 of the number of feature channels.

[0188] Step S323. Calculate the vertical gradient value of each pixel in the image feature map.

[0189] For example, the vertical gradient value is calculated using the Sobel operator, where the horizontal convolution kernel of the Sobel operator is... The vertical convolution kernel of the Sobel operator is .

[0190] The formula for calculating the vertical gradient value is as follows:

[0191]

[0192] In the formula, G v This represents the vertical gradient value.

[0193] Step S324. Correct the initial weight coefficients based on the vertical gradient graph.

[0194] For example, attention masks can be constructed based on vertical gradient values, and the weight coefficients can be adjusted so that channels with larger vertical gradient values ​​have larger weight coefficients.

[0195] The correction formula for the initial weight coefficients is as follows:

[0196]

[0197] In the formula, w c G represents the initial weighting coefficients. v (c) represents the average vertical gradient value of the c-th channel, w' c These are the corrected weighting coefficients.

[0198] Step S325. Multiply the corrected weight coefficients element-wise with each channel of the image feature map to obtain the enhanced feature map.

[0199] In this embodiment, by enhancing the image feature map, the feature channels with large vertical gradients, such as curbs and guardrails, are strengthened, which can effectively suppress horizontal texture interference.

[0200] In some implementations of this embodiment, generating semantic segmentation maps and depth maps of road images includes: inputting enhanced feature maps into a dual-branch decoding network to generate semantic segmentation maps and depth maps.

[0201] The dual-branch decoding network includes a semantic segmentation branch and a depth branch.

[0202] The semantic segmentation branch is used to generate a semantic segmentation map based on the feature enhancement map. Specifically, the semantic segmentation branch uses transposed convolution to upsample the enhanced feature map (sampling rates of 2, 4, and 8, respectively), gradually restoring it to the resolution of the input image. Finally, it outputs a pixel-level semantic segmentation map Sseg through the Softmax activation function. The semantic segmentation map contains three semantic labels: "road surface", "curb", and "obstacle", with label values ​​of 0, 1, and 2, respectively.

[0203] The depth branch is used to generate a depth map based on the feature enhancement map. Specifically, the depth branch is based on the disparity calculation principle of binocular vision. It inputs the feature maps of the left and right views into a semi-global matching network, calculates the disparity value d for each pixel (disparity = column coordinate of the left image pixel - column coordinate of the corresponding pixel in the right image), and then generates a depth map D using the disparity-depth conversion formula. map .

[0204] The parallax-depth conversion formula is:

[0205]

[0206] In the formula, Z is the depth value in meters; f is the focal length of the stereo camera (obtained from the intrinsic parameter matrix K); and b is the baseline length of the stereo camera (a fixed value, such as 120mm). This represents the disparity value, expressed in pixels.

[0207] Step S400. Based on the semantic segmentation map and depth map, the original measurement value of the road width is obtained through three-dimensional back projection and spatial geometric calculation.

[0208] In some embodiments of this example, based on the semantic segmentation map and depth map, the original measurement value of the road width is obtained through three-dimensional back projection and spatial geometric calculation, including steps S410 to S460.

[0209] Step S410. Extract the edge pixels from the semantic segmentation map, and filter out the left edge pixel set and the right edge pixel set respectively.

[0210] For example, if the label of the road edge in the semantic segmentation graph is 1, then from the semantic segmentation graph S seg Extract pixels with a semantic label of 1 (roadside pixels), and then filter out the set of left roadside pixels. (The set of pixels along the roadside on the left and right sides of the image) (The roadside pixels on the right side of the image), where, The column coordinates of the left-side pixel set. Let these be the row coordinates of the pixel set along the left path. The column coordinates of the left-side pixel set. The left-side pixel set has row coordinates.

[0211] Step S420. Based on the depth map, back-project the road edge pixels from the image coordinate system to the camera's 3D coordinate system to obtain the left road edge point cloud set and the right road edge point cloud set.

[0212] The formula for transforming from the image coordinate system to the camera's 3D coordinate system is:

[0213]

[0214] In the formula, (X,Y,Z) are the three-dimensional coordinates in the camera coordinate system. Principal point coordinates Focal length This represents the depth value corresponding to pixel (u,v).

[0215] Step S430. Based on the left-side point cloud set and the right-side point cloud set, the RANSAC algorithm is used to fit the left-side spatial line and the right-side spatial line respectively.

[0216] In some implementations of this embodiment, the RANSAC algorithm is used to fit the spatial straight lines of the left-side point cloud and the right-side point cloud respectively, specifically including steps S431 to S434.

[0217] Step S431. Randomly select 3 point cloud data as initial samples and construct a straight line equation.

[0218] The equation of the straight line is:

[0219]

[0220] In the formula, (X,Y,Z) are the three-dimensional coordinates in the camera coordinate system. Let be a point on the straight line. It is the direction vector of the line.

[0221] Step S432. Calculate the first distance from all left-side point clouds to the straight line.

[0222] The formula for calculating the first distance is:

[0223]

[0224] In the formula, d is the first distance, and (X,Y,Z) are the three-dimensional coordinates in the camera coordinate system. Let be a point on the straight line. It is the direction vector of the line.

[0225] Step S433. Count the number of interior points in the left edge point cloud set that satisfy the first distance less than the threshold.

[0226] Step S434. Repeat steps S531 to S533 a preset number of times, and select the line with the most interior points as the left-side spatial line.

[0227] The fitting method for the right path along the spatial line is the same as the fitting method for the left path along the spatial line.

[0228] Step S440. Define the vehicle front axle normal plane: With the center of the vehicle front axle as the origin, the vehicle's driving direction as the X-axis, the horizontal direction perpendicular to the driving direction as the Y-axis, and the direction perpendicular to the ground as the Z-axis, construct the vehicle coordinate system. The vehicle front axle normal plane is the YZ plane (X=0).

[0229] Step S450. Calculate the projected lines on the vehicle front axle plane along the spatial straight line of the left road and the spatial straight line of the right road, respectively.

[0230] Step S460. Calculate the Euclidean distance between the two projected lines, which is the original width value of the road.

[0231] The unit of the original width value is meters.

[0232] Step S500. Evaluate the reliability of the current visual perception and generate an environmental confidence coefficient.

[0233] In some embodiments of this example, the reliability of the current visual perception is evaluated and an environmental confidence coefficient is generated, including steps S510 to S530.

[0234] Step S510. Calculate the brightness histogram entropy of the left view.

[0235] In some embodiments of this example, the brightness histogram entropy of the left view is calculated, including steps S510 to S515.

[0236] Step S511. Perform grayscale processing on the corrected left view to obtain a grayscale image.

[0237] Step S512. Statistically analyze the histogram distribution of the grayscale image. , denoted as the number of pixels at gray level i.

[0238] Step S513. Calculate the probability distribution of the histogram. , where M×N is the total number of pixels in the image.

[0239] Step S514. Calculate the brightness histogram entropy of the image.

[0240] The formula for calculating the entropy of the brightness histogram is:

[0241] .

[0242] Step S515. Normalize the brightness histogram entropy of the image to the range [0,1].

[0243] The higher the entropy value of the brightness histogram, the more stable the illumination and the richer the image information.

[0244] Step S520. Calculate the disparity consistency pass rate of the binocular disparity map.

[0245] In some implementations of this embodiment, binocular parallax is calculated. Figure 1 Consistency pass rate, including steps S521 to S523.

[0246] Step S521. Based on the left and right views acquired by the binocular camera, calculate the forward disparity map from the left view to the right view respectively. Reverse parallax diagram from right view to left view .

[0247] Step S522. For the corresponding pixel positions in the forward disparity map and the reverse disparity map, extract their forward disparity value and reverse disparity value, and calculate the absolute value of the difference between the two. If the absolute value is less than the preset consistency threshold, the disparity consistency verification of the corresponding pixel is determined to be passed.

[0248] Specifically, for each pixel (u, v) in the left and right views, verify the parallax consistency: if If the disparity consistency verification of that pixel is passed, then the disparity consistency verification is deemed successful. This represents the disparity value from the left view pixel (u,v) to the right view. Represents the pixels of the right view The disparity value to the left view.

[0249] Step S523. Calculate the parallax consistency pass rate.

[0250] The formula for calculating the parallax consistency pass rate is as follows:

[0251]

[0252] In the formula, M×N represents the number of pixels with consistent disparity matching; M×N represents the total number of pixels in the image. The disparity consistency pass rate is denoted by , which ranges from [0,1]. The higher the value, the more reliable the disparity matching.

[0253] Step S530. Obtain the environmental confidence coefficient by weighted summation based on the luminance histogram entropy and disparity consistency pass rate.

[0254] The formula for calculating the environmental confidence coefficient is as follows:

[0255]

[0256] In the formula, w1=0.6 and w2=0.4 are weighting coefficients (calibrated through experiments to balance the influence of illumination stability and matching reliability); λ is the environmental confidence coefficient, which ranges from [0,1]. The higher the value of λ, the stronger the visual perception reliability.

[0257] Step S600. Based on the environmental confidence coefficient, dynamically adjust the fusion weight of the original width value.

[0258] In some implementations of this embodiment, the fusion weight of the original width value is dynamically adjusted based on the environmental confidence coefficient, including steps S610 to S620.

[0259] Step S610. Dynamically construct the vehicle-end observation noise variance based on the environmental confidence coefficient.

[0260] The formula for calculating the variance of the vehicle-end observation noise is as follows:

[0261]

[0262] In the formula, Based on the variance, λ is the adjustment coefficient, and λ is the environmental confidence coefficient. The variance of the noise observed at the vehicle end.

[0263] For example, basic variance (Based on measurement error statistics under clear daytime conditions with no interference), adjustment coefficient (Control the rate of change of variance).

[0264] When the environment is harsh, the environmental confidence coefficient decreases, the noise variance of the vehicle-end observation increases exponentially, and the reliability of the vehicle-end measurement values ​​decreases.

[0265] Step S620. Based on the vehicle-end observation noise variance, the fusion weights are obtained using the Kalman gain calculation formula.

[0266] The formula for calculating the fusion weight is:

[0267]

[0268] In the formula, To integrate weights, To predict covariance (based on the precision statistics of map data, such as a value of 0.05). The variance of the noise observed at the vehicle end.

[0269] The greater the fusion weight, the higher the weight of the vehicle-side data.

[0270] Step S700. Based on the fusion weight, fuse the original width value and the prior value of road width in the vehicle map to generate the final measured value of road width.

[0271] The formula for calculating the final measured value is:

[0272]

[0273] In the formula, For the final measurement value, This is the original width value. The road width is a priori value. For weight fusion.

[0274] Step S800. When the second preset condition is met, the abnormal data containing the final measurement value is uploaded to the cloud so that the cloud can perform cluster analysis on the abnormal data, and when the cluster analysis result meets the third preset condition, the cloud sends out the vehicle map update data.

[0275] The second preset condition is: the difference between the final measured value and the prior value of the road width is greater than the third threshold, and the duration is greater than the fourth threshold.

[0276] That is, if the difference between the final measured value and the prior value of the road width is greater than the third threshold and the duration is greater than the fourth threshold, then the second preset condition is met. At this time, the road width is judged to be abnormal, and the vehicle-mounted C-V2X OBU module uploads the abnormal data to the cloud through the 5G network; otherwise, it is judged to be a normal scenario, and the fused data is only stored locally and not actively uploaded.

[0277] The abnormal data includes final measurement values, environmental confidence coefficients, vehicle location (GPS positioning data), timestamps, and camera extrinsic parameters.

[0278] In some implementations of this embodiment, the cloud performs cluster analysis on the abnormal data, and sends out vehicle map update data when the cluster analysis result meets a third preset condition, including steps S810 to S860.

[0279] Step S810. Group the abnormal data from multiple vehicles according to the road segment ID and timestamp.

[0280] For example, data within the same road segment and group 5 are considered as one group.

[0281] Step S820. Filter invalid abnormal data based on preset rules.

[0282] For example, data with λ less than 0.1 (extremely low reliability of visual perception), data with vehicle positioning error greater than 1 meter, and data whose final measurement value exceeds the reasonable width range (±1 meter) of the road segment are removed.

[0283] Step S830. Perform cluster analysis on abnormal information of the same road segment within a set time window to determine the true width value of the road segment.

[0284] For example, using the DBSCAN clustering algorithm, the specific clustering analysis process is as follows: Preprocessed valid data is clustered using DBSCAN, with the following clustering parameters set: cluster radius ε = 0.05 meters (5 cm, based on measurement accuracy requirements), minimum sample size min_samples = 5 (data from at least 5 vehicles is aggregated); the density of each cluster center (the number of samples within an ε range around the cluster center) is calculated, and cluster centers with densities exceeding a fifth threshold (e.g., the fifth threshold is 5) are selected. The width value corresponding to this cluster center is the true width value of the road segment.

[0285] Step S840. When the difference between the actual road width value and the prior road width value is greater than the sixth threshold, update the prior road width value to the actual width value and record the update timestamp and data source.

[0286] For example, if This triggers a map update, updating the prior road width value to the actual width value, where... This is the actual width value. This is the prior value for road width.

[0287] The data source refers to the uploaded vehicle IDs of the abnormal data participating in the clustering.

[0288] Step S850. Send map update instructions to vehicles in the road segment and surrounding preset range via OTA (Over-The-Air) technology so that vehicles can update the locally cached map data in real time after receiving the instructions.

[0289] Step S860. Feed back the clustering analysis results and map update status to the vehicles that participated in uploading abnormal data, so that the vehicles can adjust their local fusion strategies based on the feedback results (such as increasing the map prior value weight of the road segment).

[0290] For example, the weight of the prior value of the road width for a road segment can be dynamically adjusted based on feedback results: if the cloud indicates that the map data for that road segment is reliable (e.g., it has just been updated), the weight can be reduced. As the Kalman gain Kk decreases, the weight of the prior value of road width is indirectly increased; when the feedback data is unreliable (such as in construction sections), it is increased. This increases the weight of vehicle-side measurement values. Specifically, it increases the prediction covariance. The smaller the value, the higher the reliability of the prior value of road width, and the greater the weight.

[0291] Table 1 shows the hardware deployment options for the method in this embodiment in one example.

[0292] Table 1

[0293] Deployment level hardware module Technical parameters Reasons for selection vehicle end Binocular vision module Baseline length 120mm, resolution 1920×1080, frame rate 30fps Baseline length balances measurement accuracy and installation space; high resolution and frame rate meet real-time sensing requirements; wide operating temperature range adapts to complex environments. vehicle end Embedded AI computing platform NVIDIA Orin NX (200 TOPS computing power) High computing power satisfies real-time inference of the Swing Transformer model vehicle end C-V2X OBU module Supports direct PC5 communication with a communication distance of 300-500 meters. It meets the real-time data transmission requirements of vehicle-road anchor points, and ensures the timeliness of online self-calibration with low latency. roadside Roadside Unit (RSU) Coverage radius of 300-500 meters, supports 5G+PC5 dual-mode communication Wide coverage radius reduces blind spots, dual-mode communication supports both vehicle-to-vehicle direct connection and cloud backhaul, adapting to roadside outdoor deployment environments. roadside absolute anchor point on the roadside Select the center / stop line corner of the manhole cover, with a world coordinate accuracy of ≤2cm. High coordinate accuracy ensures calibration results, and reasonable spacing ensures the effectiveness of extrinsic parameter optimization. cloud High-precision map database Supports real-time updates of dynamic elements (road width) Low latency meets the real-time fusion requirements of multi-vehicle data, and large-capacity storage supports massive crowdsourced data. cloud AI learning server 8×GPU (NVIDIA A100), supports DBSCAN clustering High computing power satisfies the global computation requirements of DBSCAN clustering and Kalman fusion, and the distributed architecture supports multi-segment parallel processing.

[0294] An example of calculating the original measured value of road width using the method in this embodiment is as follows:

[0295] The input data is: a left view of a nighttime backlit scene (resolution 1920×1080) captured by a vehicle-mounted binocular camera, with some vegetation obscuring the road surface.

[0296] As shown in Table 2, the calculation process is as follows: First, image preprocessing is performed to eliminate distortion and illumination fluctuations; the image is input into the Swing Transformer network, and after feature extraction and weighting, the semantic segmentation branch accurately identifies the roadside pixels (occluded areas are not misidentified); the depth branch generates a disparity map, which is then converted into a depth map and back-projected to obtain the roadside point cloud; the RANSAC algorithm fits the roadside straight line, and the Euclidean distance in the front-axis normal plane is calculated to obtain... rice.

[0297] Table 2. Semantic-geometric joint perception experiment based on Swing Transformer

[0298] Step number Specific operation content Core parameters / formulas Use of equipment / tools Operating basis Expected results (30 replicates, coefficient of variation CV ≤ 3%) 1-1 The vehicle-mounted binocular camera simultaneously acquires left and right views of the road to ensure consistent exposure parameters. 1920×1080 resolution, 30fps frame rate; 1 / 30s shutter speed, ISO 400, auto white balance. Vehicle-mounted binocular camera (baseline 120mm, lens focal length 8mm) "Image Acquisition and Correction"; Clause 5.1 of GB / T33190-2016 Obtain left and right views with no motion blur and no exposure differences; the left view is used for subsequent processing, and the right view assists in parallax calculation. 1-2 Distortion correction was performed using the Zhang Zhengyou calibration method, followed by grayscale normalization. <![CDATA[Radial distortion coefficients k1 = 0.01, k2 = 0.002, k3 = 0.0001; tangential distortion coefficients p1 = 0.001, p2 = 0.0005; normalization range [0, 255]]]> OpenCV 4.8.0 (cv2.undistort(), cv2.normalize()) Zhang Zhengyou's calibration method principle The corrected image edges show no barrel / pincushion distortion; the normalized pixel value fluctuation is ≤10, and the CV is 2.1%. 1-3 The left view inputs a pre-trained SwinTransformer network, which performs feature extraction and roadside attention module (CAM) weighting. Network structure: number of stages [2,2,18,2], number of feature channels [96,192,384,768]; window size 7×7, stride 3.5. NVIDIA Orin NX (200 TOPS computing power); PyTorch 2.0 (pre-trained dataset Cityscapes road subset) Semantic-Geometric Joint Sensing Algorithm Output enhanced feature map; road edge channel weight ≥ 0.8, horizontal texture channel weight ≤ 0.2; feature extraction time ≤ 30ms / frame. 1-4 A dual-branch decoding network generates semantic segmentation maps and depth maps. <![CDATA[Semantic segmentation: Upsampling rates [2, 4, 8], Softmax activation, 3-class labels (road surface 0 / curb 1 / obstacle 2); Depth map: Semi-global matching network, disparity-depth conversion formula Z = f⋅b / d dis (f = 8mm, b = 120mm, d dis is the disparity value)]]> NVIDIA Orin NX (200 TOPS computing power); PyTorch 2.0 (pre-trained dataset Cityscapes road subset) Binocular vision parallax principle The semantic segmentation map path edge annotation accuracy is ≥98%, with a CV of 1.8%; the depth map error is ≤1cm, with a CV of 2.5%. 1-5 Extract the curb point cloud, fit a spatial straight line using the RANSAC algorithm, and calculate the Euclidean distance in the front-axis normal plane. RANSAC: 1000 iterations, distance threshold 2cm; vehicle coordinate system: origin at front axle center, X = driving direction, front axle normal plane is YZ plane (X=0). Open3D (point cloud processing); Scikit-learn (RANSACRegressor) Road width calculation <![CDATA[Output the original width value W vision = 3.72 m; measurement error 4 cm ± 1.2 cm, CV = 2.8%; night backlight + occlusion scene detection rate 98%]]>

[0299] The method in this embodiment improves the road edge detection rate from 65% of traditional CNNs to 98%, with a measurement error of 4cm, meeting the accuracy requirement of less than or equal to 5cm.

[0300] An example of calibrating and updating extrinsic parameters using the method in this embodiment is as follows:

[0301] The triggering scenario is: a vehicle enters an intersection of a main urban road (RSU coverage area), and the camera's external parameters drift due to long-term vibration, with an initial measurement error of 15cm.

[0302] As shown in Table 3, the processing procedure is as follows: The roadside unit broadcasts the world coordinates of the centers of the three manhole covers: , , The unit is meters; the vehicle-mounted algorithm identified the centers of three manhole covers and extracted their pixel coordinates: , , A reprojection error model was constructed. After three iterations of the LM algorithm, the error J = 8 × 10⁻ 6 The termination condition is met.

[0303] After the new external reference is updated The measurement error was reduced to 4cm. After the equipment ran continuously for 6 months, the measurement error was still less than or equal to 5cm, and no shutdown for calibration was required.

[0304] Table 3 Online self-calibration experiment based on roadside absolute anchor points

[0305] Step number Specific operation content Core parameters / formulas Use of equipment / tools Operating basis Expected results (30 replicates, coefficient of variation CV ≤ 3%) 2-1 The vehicle-mounted C-V2X OBU monitors the RSU wireless signal strength (RSSI) in real time to determine whether it has entered the coverage area. RSSI threshold ≥ -85dBm; RSU coverage radius 300-500 meters, communication frequency 5.9GHz. C-V2X OBU module (Huawei MH5000, supports PC5 direct connection); Roadside RSU (Huawei eLTE8200) First preset condition The false positive rate for coverage area identification is ≤1%; the signal is stable after entering the coverage area, with a packet loss rate of ≤0.5% and a CV of 1.3%. 2-2 Receive the absolute anchor point world coordinate set of RSU broadcasts and record the reception timestamp. <![CDATA[There are 3 anchor points, in the world coordinate system WGS - 84, coordinate accuracy ≤ 2 cm; broadcast frequency 10 Hz; anchor point coordinates: P w1 =(100.23, 200.45, 0.02) m, P w2 =(105.67, 203.12, 0.02) m, P w3 =(102.34, 208.95, 0.02) m]]> Same as above Anchor point data reception Successfully received coordinate set, data transmission delay ≤100ms, CV=1.1%; timestamp synchronization accuracy ≤5ms. 2-3 YOLOv8 detects the physical landmarks corresponding to anchor points in the left view, extracts pixel coordinates, and filters them. YOLOv8 model: Input size 640×640, confidence threshold 0.7; filtering threshold 0.8; Hough circle detection is used for manhole cover center (minimum radius 15px), and Shi-Tomasi algorithm is used for stop line corners (corner response threshold 0.01). NVIDIA OrinNX; YOLOv8 pre-trained model (fine-tuned for road scenes, 100k iterations) Anchor matching Output pixel coordinates: Pimg1=(850,420), Pimg2=(920,430), Pimg3=(880,480); Matching accuracy 100%, CV=0 2-4 A reprojection error model is constructed based on a pinhole camera model, and the LM algorithm is used to iteratively optimize the extrinsic parameters. The formula for mapping world coordinates to pixel coordinates; the nonlinear least squares optimization objective function; the damping coefficient λ = 1e-3, the maximum number of iterations is 50, and the convergence threshold is 1e-5. Python (SciPy.optimize.least_squares); Camera intrinsic parameter matrix K (factory calibration) Extrinsic parameter optimization; LM algorithm principle <![CDATA[Convergence is achieved after 3 iterations, reprojection error = 8×10⁻ 6 m; Optimization time ≤ 0.3 seconds per iteration, CV = 2.2%]]> 2-5 Verify the validity of extrinsic parameters and overwrite the current extrinsic parameter matrix of the vehicle system. Validity verification threshold: Average reprojection error of all anchor points ≤ 3cm; Abnormal log records: timestamp, vehicle GPS location, error value In-vehicle embedded system (Linux kernel, real-time scheduling priority ≥90) External parameter update verification The external parameter update success rate was 99%; after calibration, the measurement error decreased from 15cm to 4cm±0.8cm, CV=2.0%; after 6 months of continuous operation, the error remained ≤4.1cm.

[0306] An example of using the method in this embodiment for multi-source fusion and cloud-based updates based on environmental entropy is as follows:

[0307] The scenario is: a rainstorm at night, with no obvious obstructions on the road surface. Due to construction, the width of a certain section of the road has been reduced from 3.75 meters to 3.2 meters.

[0308] The processing procedure is as follows: Vehicle-side calculation of environmental entropy: In a rainstorm nighttime scenario, H img =0.3, C consistency =0.2, calculate λ=0.25, R k =0.8 (basic variance σ) base =0.1). Kalman Fusion: Cloud W map =3.75 meters (static reference width of this road section), P k =0.05, calculate K k =0.05 / (0.05+0.8)=0.06, W final =3.75 + 0.06 × (3.80 - 3.75) = 3.753 meters (close to W) map (To avoid visual distortion interference). Map update triggered: Multiple vehicles upload W at a construction site. vision =3.2 meters (deviation 55cm, lasting 4 seconds), the cloud performs DBSCAN clustering on the data of 10 vehicles within 5 minutes, the cluster center density = 8 (exceeding the threshold of 5), confirming that the road width has changed to 3.2 meters, immediately updates the high-precision map, and pushes it to subsequent vehicles via OTA.

[0309] Table 4. Experiment on confidence fusion and cloud-based crowdsourcing update based on environmental entropy.

[0310] Step number Specific operation content Core parameters / formulas Use of equipment / tools Operating basis Expected results (30 replicates, coefficient of variation CV ≤ 3%) 3-1 <![CDATA[Calculate the entropy H of the left view brightness histogram img and the binocular disparity consistency passing rate C consistency > <![CDATA[Histogram gray level is 256, probability distribution p(i)=h(i) / (M×N), entropy formula: H img = -∑255 i=0 p(i)㏒2p(i); disparity consistency threshold ≤ 1 pixel, passing rate formula C consistency = N consistent / M×N; environmental confidence coefficient λ = 0.6H img + 0.4C consistency > OpenCV 4.8.0; Python (Matplotlib) Calculation of environmental confidence coefficient <![CDATA[Under the heavy rain at night: H img = 0.3 ± 0.02, C consistency = 0.2 ± 0.03, λ = 0.25 ± 0.01, CV = 2.9%]]> 3-2 Dynamically construct the vehicle-side observation noise variance Rk and calculate the Kalman gain Kk. <![CDATA[Rk=σ base ⋅(1+e −a(λ−0.5 ))(σ base =0.1, adjustment coefficient a=5.0); prediction covariance Pk=0.05; Kalman gain formula Kk= Pk / Pk+Rk]]> Python (NumPy) Fusion weight adjustment In the heavy rain scenario, Rk=0.8±0.04, Kk=0.06±0.003, CV=2.7%. 3-3 <![CDATA[Fuse the original vehicle-end measurement values with the map prior values and output the final road width W final > <![CDATA[Fusion formula W final = W prior + K k ⋅ (W vision − W prior ) (W prior = 3.75 m (map prior value), W vision = 3.80 m (original value at the vehicle end))]]> In-vehicle embedded system Final measurement calculation <![CDATA[W final =3.753m±0.012m, measurement error ≤6cm, CV=3.0%; data availability 96% 3-4 Determine if the abnormal data upload conditions are met to trigger 5G upload. <![CDATA[Second preset condition: ∣W final −W prior ∣ > 5 cm and the duration ≥ 4 seconds (120 frames); Uploaded data: W final , λ, GPS location, timestamp, camera extrinsic parameters]]> Vehicle-mounted C-V2X OBU (supports 5G mode); 5G base station Second preset condition The accuracy rate of uploads triggered by construction sections is 100%, with a false upload rate of ≤2%; upload latency is ≤500ms, and CV = 3.1%. 3-5 DBSCAN clustering analysis is performed in the cloud, triggering high-precision map updates and OTA deployment. Clustering parameters: radius ε = 0.05m, minimum number of samples min_samples = 5, density threshold = 5; time window 5 minutes; OTA delivery range = 3 kilometers. Cloud-based AI server (8×NVIDIA A100); High-precision map dynamic database (supports millisecond-level updates) Cloud update Cluster center = 3.2m ± 0.02m, CV = 2.5%; map update delay 4.2 minutes ± 0.3 minutes; updated map accuracy ± 5cm; vehicle update success rate 99%.

[0311] The method in this embodiment achieves 96% data availability in a rainstorm nighttime scenario, with a map update delay of less than or equal to 5 minutes, and improves the accuracy of vehicle avoidance decisions in construction areas to 97%.

[0312] The second aspect of this embodiment discloses a binocular vision road width measurement system for vehicle-road-cloud cooperation, such as... Figure 2 As shown, the binocular vision road width measurement system includes an image acquisition module, an external parameter update module, a joint perception module, a road width calculation module, a confidence assessment module, a weight adjustment module, a data fusion module, and a cloud update module.

[0313] The image acquisition module is used to acquire and correct road images based on an onboard binocular camera.

[0314] The extrinsic parameter update module is used to calibrate and update the extrinsic parameters of the vehicle-mounted binocular camera based on the acquired roadside absolute coordinate information when the first preset condition is met.

[0315] The joint perception module is used to extract features from the road image based on the extrinsic parameters of the vehicle-mounted binocular camera and the semantic-geometric joint perception algorithm based on the Win Transformer, and generate a semantic segmentation map and a depth map of the road image.

[0316] The road width calculation module is used to obtain the original measured value of the road width based on the semantic segmentation map and depth map through three-dimensional back projection and spatial geometric calculation.

[0317] The confidence assessment module is used to evaluate the reliability of the current visual perception and generate an environmental confidence coefficient.

[0318] The weight adjustment module is used to dynamically adjust the fusion weight of the original width value based on the environmental confidence coefficient.

[0319] The data fusion module is used to generate the final measured value of the road width by fusing the original width value and the prior value of the road width in the vehicle map based on the fusion weight.

[0320] The cloud update module is used to upload abnormal data containing the final measurement value to the cloud when the second preset condition is met, so that the cloud can perform cluster analysis on the abnormal data, and send vehicle map update data to the cloud when the cluster analysis result meets the third preset condition.

[0321] It should be noted that those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system or device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0322] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A binocular vision method for measuring road width in a vehicle-road-cloud cooperative manner, characterized in that, include: Road images are acquired and corrected using vehicle-mounted binocular cameras; When the first preset condition is met, the extrinsic parameters of the vehicle-mounted binocular camera are calibrated and updated based on the acquired roadside absolute coordinate information; Based on the extrinsic parameters of the vehicle-mounted binocular camera, features are extracted from the road image using a semantic-geometric joint perception algorithm based on Swin Transformer, and semantic segmentation map and depth map of the road image are generated. Based on the semantic segmentation map and depth map, the original measurement value of the road width is obtained through three-dimensional back projection and spatial geometric calculation; Assess the reliability of current visual perception and generate an environmental confidence coefficient; Based on the environmental confidence coefficient, the fusion weight of the original width value is dynamically adjusted; Based on the fusion weights, the original width value and the prior road width value in the vehicle map are fused to generate the final measured value of the road width; When the second preset condition is met, the abnormal data containing the final measurement value will be uploaded to the cloud so that the cloud can perform cluster analysis on the abnormal data, and when the cluster analysis result meets the third preset condition, the cloud will send out the vehicle map update data. The first preset condition is that the vehicle enters the coverage area of ​​the roadside unit. When the vehicle enters the coverage area of ​​the roadside unit, the world coordinate set of multiple absolute anchor points broadcast by the roadside unit is received and stored, and the receiving timestamp is recorded. The second preset condition is: the difference between the final measured value and the prior value of the road width is greater than the third threshold, and the duration is greater than the fourth threshold; The cloud performs cluster analysis on abnormal data, and sends out updated vehicle map data when the cluster analysis results meet a third preset condition, including: Grouping abnormal data from multiple vehicles according to road segment ID and timestamp; Invalid and abnormal data are filtered based on preset rules; Cluster analysis is performed on abnormal information of the same road segment within a set time window to determine the true width of the road segment. When the difference between the actual road width and the prior road width is greater than the sixth threshold, the prior road width is updated to the actual road width, and the update timestamp and data source are recorded. The system sends map update commands to vehicles within the preset range of the road section and surrounding areas via OTA, so that the vehicles can update the locally cached map data in real time after receiving the commands. The clustering analysis results and map update status are fed back to the vehicles that participated in uploading abnormal data, so that the vehicles can adjust the weight of the prior prediction values ​​when calculating the final measurement values ​​based on the feedback results. Assess the reliability of current visual perception and generate an environmental confidence coefficient, including: Calculate the brightness histogram entropy of the left view; Calculate the disparity consistency pass rate of the binocular disparity map; The environmental confidence coefficient is obtained by weighted summation of luminance histogram entropy and disparity consistency pass rate. Based on the environmental confidence coefficient, the fusion weight of the original width value is dynamically adjusted, including: The vehicle-end observation noise variance is dynamically constructed based on the environmental confidence coefficient. The formula for calculating the vehicle-end observation noise variance is as follows: ； In the formula, Based on the variance, λ is the adjustment coefficient, and λ is the environmental confidence coefficient. Variance of noise observed at the vehicle end; Based on the observed noise variance at the vehicle end, the fusion weights are obtained using the Kalman gain calculation formula. The formula for calculating the fusion weights is as follows: ； In the formula, To integrate weights, To predict covariance, The variance of the noise observed at the vehicle end.

2. The binocular vision road width measurement method for vehicle-road-cloud collaboration according to claim 1, characterized in that, Road images acquired and corrected using vehicle-mounted binocular cameras include: Road images are acquired using an onboard binocular camera, and the road images include a left view and a right view; The distortion correction of the left and right views was performed using the Zhang Zhengyou calibration method. The corrected left and right views are then subjected to grayscale normalization.

3. The binocular vision road width measurement method for vehicle-road-cloud collaboration according to claim 1, characterized in that, When the first preset condition is met, the extrinsic parameters of the vehicle-mounted binocular camera are calibrated and updated based on the acquired roadside absolute coordinate information, including: When a vehicle enters the coverage area of ​​a roadside unit, it receives and stores the world coordinate set of multiple absolute anchor points broadcast by the roadside unit, and records the receiving timestamp. The pre-trained YOLOv8 detection model identifies the physical landmarks in the left view that correspond to the anchor points broadcast by the roadside unit. Detect the anchor point pixel coordinates corresponding to physical landmarks, and filter out the anchor point pixel coordinates with a detection confidence greater than the first threshold to form a pixel coordinate set corresponding to the world coordinate set; Based on the pinhole camera model, a mapping formula from world coordinates to pixel coordinates is established; The mapping formula is transformed to obtain the projection function; A nonlinear least squares optimization objective function is constructed based on the reprojection error, wherein the reprojection error is the Euclidean distance between the observed pixel coordinates and the projected pixel coordinates; The rotation matrix is ​​mapped to the Lie algebra space and expressed as a perturbation, and an extrinsic parameter update model is established by representing the perturbation. With the goal of minimizing the reprojection error, the Levenberg-Marquardt algorithm is used to iteratively optimize the perturbation until the convergence condition is met, and the optimized perturbation is obtained. Based on the optimized perturbation, the updated extrinsic parameter matrix is ​​calculated using the extrinsic parameter update model, and the validity of the updated extrinsic parameters is verified. If valid, the extrinsic parameter update is performed; otherwise, the original extrinsic parameters are maintained and the anomaly is recorded.

4. The binocular vision road width measurement method for vehicle-road-cloud collaboration according to claim 1, characterized in that, Based on the extrinsic parameters of an onboard binocular camera, features are extracted from the road image using a semantic-geometric joint perception algorithm based on the Swin Transformer, and semantic segmentation map and depth map of the road image are generated, including: Input the left view into the pre-trained Swin Transformer network; The Swin Transformer network divides the input left view into multiple image patches; Each image patch is converted into a feature vector through linear projection, and location information is added to form an initial feature map; The initial feature map is transformed by a feature extraction structure comprising multiple consecutive processing stages to generate and output an image feature map; wherein, at least one processing stage employs a moving window self-attention mechanism to process the feature map input to that processing stage. Global average pooling is performed on the image feature map to obtain channel-level feature vectors; The channel-level feature vectors are nonlinearly transformed through a multilayer perceptron, and the output is an initial weight coefficient with a dimension equal to the number of channels in the image feature map. Calculate the vertical gradient value of each pixel in the image feature map; The initial weight coefficients are corrected based on the vertical gradient graph; The corrected weight coefficients are multiplied element-wise with each channel of the image feature map to obtain the enhanced feature map.

5. The binocular vision road width measurement method for vehicle-road-cloud collaboration according to claim 1, characterized in that, Based on the semantic segmentation map and depth map, the original measurement value of the road width is obtained through 3D back projection and spatial geometric calculation, including: Extract edge pixels from the semantic segmentation map, and filter out the left edge pixel set and the right edge pixel set respectively; Based on the depth map, the road edge pixels are back-projected from the image coordinate system to the camera's 3D coordinate system to obtain the left road edge point cloud set and the right road edge point cloud set; Based on the left-side point cloud set and the right-side point cloud set, the RANSAC algorithm is used to fit the left-side spatial line and the right-side spatial line respectively. Define the vehicle front axle normal plane: with the center of the vehicle front axle as the origin, the vehicle's driving direction as the X-axis, the horizontal direction perpendicular to the driving direction as the Y-axis, and the direction perpendicular to the ground as the Z-axis, construct the vehicle coordinate system, and the vehicle front axle normal plane is the YZ plane; Calculate the projected straight lines on the vehicle's front axle plane along the spatial straight lines of the left and right roads, respectively. Calculate the Euclidean distance between the two projected lines; this Euclidean distance is the original width of the road.

6. The binocular vision road width measurement method for vehicle-road-cloud collaboration according to claim 1, characterized in that, Assess the reliability of current visual perception and generate an environmental confidence coefficient, including: The left view is converted to grayscale to obtain a grayscale image; Statistical histogram distribution of grayscale images; Calculate the probability distribution of the histogram based on the histogram distribution of the grayscale image; The brightness histogram entropy of an image is calculated based on the probability distribution of the histogram. Normalize the brightness histogram entropy of the image to the range [0,1]; Based on the left and right views acquired by the binocular camera, calculate the forward disparity map from the left view to the right view and the reverse disparity map from the right view to the left view, respectively. For the corresponding pixel positions in the forward disparity map and the reverse disparity map, extract their forward disparity value and reverse disparity value, and calculate the absolute value of the difference between the two. If the absolute value is less than the preset consistency threshold, the disparity consistency verification of the corresponding pixel is determined to be passed. Calculate the parallax consistency pass rate; The environmental confidence coefficient is obtained by weighted summation of luminance histogram entropy and disparity consistency pass rate.

7. The binocular vision road width measurement method for vehicle-road-cloud collaboration according to claim 1, characterized in that, The formula for calculating the final measured value is: ； In the formula, For the final measurement value, This is the original width value. The road width is a priori value. For weight fusion.

8. A binocular vision road width measurement system for vehicle-road-cloud collaboration, characterized in that, include: The image acquisition module is used to acquire and correct road images based on an onboard binocular camera; The extrinsic parameter update module is used to calibrate and update the extrinsic parameters of the vehicle-mounted binocular camera based on the acquired roadside absolute coordinate information when the first preset condition is met. The joint perception module is used to extract features from the road image based on the extrinsic parameters of the vehicle-mounted binocular camera and the semantic-geometric joint perception algorithm based on the Win Transformer, and generate a semantic segmentation map and a depth map of the road image. The road width calculation module is used to obtain the original measured value of the road width based on the semantic segmentation map and depth map through three-dimensional back projection and spatial geometric calculation; The confidence assessment module is used to evaluate the reliability of the current visual perception and generate an environmental confidence coefficient. The weight adjustment module is used to dynamically adjust the fusion weight of the original width value based on the environmental confidence coefficient. The data fusion module is used to fuse the original width value and the prior value of road width in the vehicle map based on the fusion weight to generate the final measured value of road width. The cloud update module is used to upload abnormal data containing the final measurement value to the cloud when the second preset condition is met, so that the cloud can perform cluster analysis on the abnormal data, and send the vehicle map update data to the cloud when the cluster analysis result meets the third preset condition. The first preset condition is that the vehicle enters the coverage area of ​​the roadside unit. When the vehicle enters the coverage area of ​​the roadside unit, the world coordinate set of multiple absolute anchor points broadcast by the roadside unit is received and stored, and the receiving timestamp is recorded. The second preset condition is: the difference between the final measured value and the prior value of the road width is greater than the third threshold, and the duration is greater than the fourth threshold; The cloud performs cluster analysis on abnormal data, and sends out updated vehicle map data when the cluster analysis results meet a third preset condition, including: Grouping abnormal data from multiple vehicles according to road segment ID and timestamp; Invalid and abnormal data are filtered based on preset rules; Cluster analysis is performed on abnormal information of the same road segment within a set time window to determine the true width of the road segment. When the difference between the actual road width and the prior road width is greater than the sixth threshold, the prior road width is updated to the actual road width, and the update timestamp and data source are recorded. The system sends map update commands to vehicles within the preset range of the road section and surrounding areas via OTA, so that the vehicles can update the locally cached map data in real time after receiving the commands. The clustering analysis results and map update status are fed back to the vehicles that participated in uploading abnormal data, so that the vehicles can adjust the weight of the prior prediction values ​​when calculating the final measurement values ​​based on the feedback results. Assess the reliability of current visual perception and generate an environmental confidence coefficient, including: Calculate the brightness histogram entropy of the left view; Calculate the disparity consistency pass rate of the binocular disparity map; The environmental confidence coefficient is obtained by weighted summation of luminance histogram entropy and disparity consistency pass rate. Based on the environmental confidence coefficient, the fusion weight of the original width value is dynamically adjusted, including: The vehicle-end observation noise variance is dynamically constructed based on the environmental confidence coefficient. The formula for calculating the vehicle-end observation noise variance is as follows: ； In the formula, Based on the variance, λ is the adjustment coefficient, and λ is the environmental confidence coefficient. Variance of noise observed at the vehicle end; Based on the observed noise variance at the vehicle end, the fusion weights are obtained using the Kalman gain calculation formula. The formula for calculating the fusion weights is as follows: ； In the formula, To integrate weights, To predict covariance, The variance of the noise observed at the vehicle end.

Images