Visual auxiliary positioning method, device and electronic equipment for vehicle laser radar

Abstract

The application discloses a kind of visual auxiliary positioning method, device and electronic equipment of vehicle laser radar.The method comprises the following steps: collecting the laser radar data of target vehicle;According to the laser radar data and the preset map point cloud data are matched, determine initial pose, and obtain a plurality of distribution characteristic data in matching process;According to a plurality of distribution characteristic data, determine the degradation quantitative index of laser radar, according to the degradation quantitative index, determine the state information of laser radar;According to the state information, determine the fusion state of visual auxiliary module, according to the fusion state, the visual data of the visual auxiliary module, the laser radar data and the initial pose determine the positioning information of the target vehicle, real-time discrimination laser radar positioning state, dynamically adjust fusion strategy, effectively improve positioning accuracy and running stability.

Description

Technical Field

[0001] This invention relates to the field of vehicle driving technology, and in particular to a visual-assisted positioning method, device, and electronic device for vehicle lidar. Background Technology

[0002] LiDAR is widely used in autonomous driving positioning due to its high precision and strong geometric perception capabilities.

[0003] LiDAR localization obtains vehicle position and attitude by registering the current point cloud with the prior map point cloud, offering advantages such as high accuracy and low sensitivity to absolute scale. However, in practical applications, the quality of LiDAR point clouds can be affected by factors such as weather, dust, reflective materials, occlusion, and scene geometry. When the point cloud's geometric features are insufficient or the number of noisy points increases, the matching residual will significantly increase, leading to unstable localization results. Summary of the Invention

[0004] This invention provides a visual-assisted positioning method, device, and electronic device for vehicle LiDAR, to solve the problem of unstable positioning of LiDAR in feature degradation scenarios, which leads to a decrease in positioning accuracy of autonomous vehicles.

[0005] According to one aspect of the present invention, a vision-assisted positioning method for a vehicle lidar is provided, comprising: Collect lidar data of the target vehicle; The initial pose is determined by matching the lidar data with the preset map point cloud data, and multiple distribution feature data are obtained during the matching process. The degradation quantization index of the lidar is determined based on multiple distribution feature data, and the state information of the lidar is determined based on the degradation quantization index. The fusion state of the vision assistance module is determined based on the state information, and the positioning information of the target vehicle is determined based on the fusion state, the visual data of the vision assistance module, the lidar data, and the initial pose.

[0006] According to another aspect of the present invention, a visual-assisted positioning device for a vehicle lidar is provided, characterized in that it comprises: The acquisition module is used to acquire LiDAR data from the target vehicle. The feature acquisition module is used to match the lidar data and preset map point cloud data to determine the initial pose and acquire multiple distribution feature data during the matching process, wherein the distribution feature data is used to characterize the state information of lidar positioning. The status information determination module is used to determine the degradation quantization index of the lidar based on multiple distribution feature data, and to determine the status information of the lidar based on the degradation quantization index. The positioning information determination module is used to determine the fusion state of the vision assistance module based on the state information, and to determine the positioning information of the target vehicle based on the fusion state, the visual data of the vision assistance module, the lidar data, and the initial pose.

[0007] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the visual-assisted positioning method of vehicle lidar according to any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the visual-assisted positioning method of vehicle lidar according to any embodiment of the present invention.

[0009] According to another aspect of the present invention, embodiments of the present disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements the visual-assisted positioning method for a vehicle lidar as described in any of the embodiments of the present disclosure.

[0010] The technical solution of this invention collects LiDAR data of the target vehicle to accurately acquire environmental structure data, providing a reliable observation basis for positioning. It matches the LiDAR data with preset map point cloud data to determine the initial pose and acquires multiple distribution feature data during the matching process. This allows for the acquisition of the initial pose and feature extraction, providing a basis for degradation assessment. Based on the multiple distribution feature data, it determines the LiDAR degradation quantification index and the LiDAR state information, quantifying and assessing the degree of positioning degradation and accurately determining the radar's working state. Based on the state information, it determines the fusion state of the visual assistance module. Based on the fusion state, the visual data from the visual assistance module, the LiDAR data, and the initial pose, it determines the positioning information of the target vehicle. This allows for adaptive adjustment of the fusion strategy, achieving high-precision and robust vehicle positioning. It solves the problem of unstable LiDAR positioning in feature degradation scenarios, leading to decreased positioning accuracy in autonomous vehicles. Real-time judgment of the LiDAR positioning state and dynamic adjustment of the fusion strategy effectively improve positioning accuracy and operational stability.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a visual-assisted positioning method for a vehicle lidar according to Embodiment 1 of the present invention; Figure 2 This is a flowchart of a visual-assisted positioning method for a vehicle lidar according to Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the structure of a visual-assisted positioning device for a vehicle lidar according to Embodiment 3 of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device that implements the visual-assisted positioning method of vehicle lidar according to an embodiment of the present invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0017] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0018] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0019] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0020] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0021] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0022] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0023] Example 1 Figure 1 The flowchart of a vision-assisted positioning method for a vehicle LiDAR is provided in Embodiment 1 of the present invention. This embodiment is applicable to the high-precision positioning of autonomous vehicles in feature-sparse or degraded scenarios. The method can be executed by a vision-assisted positioning device of the vehicle LiDAR, which can be implemented in hardware and / or software. Optionally, it can be implemented through an electronic device, such as a mobile terminal, a PC, or a server.

[0024] like Figure 1 As shown, the method may specifically include: S110: Collect lidar data of the target vehicle.

[0025] The target vehicle can be understood as the vehicle that needs to be located and its state estimated. As the carrier of the positioning algorithm, all sensor data (LiDAR, vision, etc.) are collected and calculated around this vehicle. The LiDAR data can be understood as the raw detection data such as point clouds, distance, angle, and intensity generated after the LiDAR emits and receives lasers. It is the core basic data source for vehicle positioning and is used for environmental perception, pose matching, and positioning calculation.

[0026] An optional implementation involves preprocessing the LiDAR data. Preprocessing steps include, but are not limited to: time synchronization, extrinsic parameter calibration, coordinate system unification, point cloud distortion correction, outlier removal, image deblurring, exposure correction, and image distortion correction. The LiDAR point cloud can be matched with high-precision maps, dense point cloud maps, sparse feature maps, or online-constructed local sub-maps, and the visual image can be matched with visual maps, semantic maps, or historical keyframes.

[0027] S120. Match the lidar data and the preset map point cloud data to determine the initial pose and obtain multiple distribution feature data during the matching process.

[0028] The map point cloud data can be understood as pre-constructed and stored 3D point cloud map data of the environment, serving as prior reference data. This data is compared with real-time laser data to calculate the vehicle's initial position and attitude. The initial position can be understood as the preliminary position and attitude result of the vehicle in the global coordinate system obtained through point cloud matching, providing an initial positioning benchmark and initial parameters for subsequent vision-laser fusion positioning. The distribution feature data can be understood as data features extracted during point cloud matching, such as the mean of the matching residuals, the root mean square of the matching residuals, the number of effective corresponding points, the proportion of interior points, the number of matching convergence iterations, and the pose jump variables between adjacent time steps. These features characterize the quality and structural observability of the current laser radar positioning and are important criteria for judging the reliability of the positioning. The laser radar positioning status information can be understood as information describing whether the laser radar positioning is currently normal or degraded. This information is used by the system to determine whether the pure laser radar positioning result should be trusted or whether other sensor assistance is needed.

[0029] In one alternative implementation, the current lidar data is matched with preset map point cloud data to obtain the initial pose at the current moment.

[0030] The point cloud matching method can be ICP matching, NDT matching, GICP matching, point-to-surface registration, point-to-line registration, or other point cloud registration methods that can output matching errors.

[0031] The following distribution feature data are extracted during the matching process and used as input for subsequent degradation determination: The mean of the matching residuals, e_mean, is used to characterize the overall matching error level of the current point cloud; the root mean square of the matching residuals, e_rmse, is used to characterize the impact of large error points on the overall matching quality; the number of effective corresponding points, n_corr, is used to characterize the number of point pairs in the current point cloud that can form a stable matching relationship; the proportion of inliers, r_inlier, is used to characterize the proportion of matching point pairs that meet the preset distance or error threshold; the number of matching convergence iterations, k_iter, and the pose jump variable ΔT between adjacent time steps are used to characterize the stability of matching convergence and the continuity of output.

[0032] S130. Determine the degradation quantization index of the lidar based on multiple distribution feature data, and determine the state information of the lidar based on the degradation quantization index.

[0033] The degradation quantification index can be understood as an evaluation index that uses numerical values ​​to quantify the degree of degradation such as the decrease in lidar positioning accuracy and matching failure, transforming qualitative positioning anomalies into quantitative values ​​to achieve state determination.

[0034] Based on the above scheme, optionally, determining the degradation quantification index of the lidar according to the multiple distribution feature data includes: normalizing the multiple distribution feature data to obtain multiple normalized index information; and determining the degradation quantification index of the lidar according to the weighted sum of the multiple normalized index information.

[0035] The normalization mentioned here can be understood as a numerical processing operation that transforms distribution feature data with different dimensions and scales into a unified numerical range (such as [0,1] or standard normal distribution), eliminating the influence of different physical dimensions or numerical ranges on the distribution features, and ensuring that they participate fairly in the evaluation in subsequent calculations. The normalization index information can be understood as the result obtained after the distribution feature data has been normalized. Each item corresponds to a standardized expression of the original distribution feature data, which is used for subsequent weighted fusion, so that degradation features of different properties can be comprehensively judged on the same scale to determine the degree of degradation of the lidar.

[0036] An alternative implementation method determines the degradation quantification index based on the following formula: ; in, Indicates quantitative indicators of degradation, , , , α represents the normalized index, and β, γ, and δ are weighting coefficients.

[0037] By adopting this technical solution, the differences in characteristic dimensions are eliminated through normalization, and the degree of degradation is accurately quantified by weighted summation, which effectively improves the accuracy and robustness of indicator evaluation and provides a reliable basis for state determination.

[0038] Based on the above scheme, optionally, the status information includes a normal state and a degraded state, and the step of determining the status information of the lidar according to the degradation quantification index includes at least one of the following: if the degradation quantification index is not greater than a preset first threshold, the status information of the lidar is determined to be a normal state; if the degradation quantification index is greater than the first threshold, the degradation state of the lidar is determined according to the degradation quantification index and a preset second threshold.

[0039] The "normal state" can be understood as a working state where the LiDAR's positioning performance and matching effect meet the standard requirements, indicating that the LiDAR positioning is reliable and can be used as the primary positioning data source. The "degraded state" can be understood as an abnormal working state where the LiDAR's positioning accuracy and matching quality decrease, indicating impaired LiDAR positioning capabilities and requiring the activation of auxiliary strategies for compensation. The first threshold can be understood as a pre-set critical value used to distinguish between the normal and degraded states, defining the state judgment boundary and initially identifying whether the LiDAR has experienced positioning degradation. The second threshold can be understood as a pre-set critical value used to further classify different degrees of degradation, further categorizing the degradation state and matching corresponding processing solutions.

[0040] This technical solution uses a two-level threshold to classify radar operating states, accurately distinguishing between normal and degraded conditions. It enables tiered assessment of positioning performance, providing a clear basis for subsequent fusion strategies, and allows for targeted adaptation to different operating conditions. This avoids misjudgments caused by a single judgment standard, effectively ensuring the overall stability and reliability of multi-sensor fusion positioning.

[0041] Based on the above scheme, optionally, the degradation state includes a first degradation state and a second degradation state, wherein the degradation degree of the second degradation state exceeds that of the first degradation state; determining the degradation state of the lidar according to the degradation quantification index and a preset second threshold includes at least one of the following: when the degradation quantification index is not greater than the second threshold, the state information of the lidar is determined to be the first degradation state; when the degradation quantification index is greater than the second threshold, the state information of the lidar is determined to be the second degradation state.

[0042] The first degradation state can be understood as a mild degradation state, i.e., mild degradation (such as some directions being unobservable, and moderate matching uncertainty), indicating that the LiDAR can still provide some usable information, but should be used with caution, possibly requiring some visual assistance or weighting. The second degradation state can be understood as a more severe degradation state (such as multiple directions being unobservable, highly repetitive structures, and severe matching ambiguity), i.e., severe degradation, indicating that the LiDAR positioning reliability is low, requiring stronger visual assistance, constraint enhancement, or even short-term primary vision. The degree of degradation can be understood as an abstract description of the strength of LiDAR positioning unreliability, which increases with the increase of the degradation quantification index, and is used to establish a "severity / level" relationship between different degradation states to support hierarchical processing strategies.

[0043] An alternative implementation method, when When the value is less than the first threshold T1, the lidar is determined to be in normal condition; when... When the value is between T1 and the second threshold T2, it is considered a mild degradation; when... If the value is greater than or equal to T2, it is considered a severe degradation.

[0044] By employing this technical solution, the degradation level of the LiDAR is precisely quantified through subdividing degradation levels, enabling differentiated state recognition from mild to severe degradation. This provides a more refined fusion decision basis for the visual assistance module, ensuring that the optimal compensation strategy can be adopted in different degradation scenarios.

[0045] S140. Determine the fusion state of the vision assistance module based on the state information, and determine the positioning information of the target vehicle based on the fusion state, the visual data of the vision assistance module, the lidar data, and the initial pose.

[0046] The visual assistance module can be understood as a visual perception hardware and software unit composed of an onboard camera and an image acquisition and processing unit. As an auxiliary sensor, it compensates for the positioning shortcomings of LiDAR in degraded scenarios. The fusion state can be understood as a collaborative fusion mode of visual and LiDAR data set according to the LiDAR status, controlling whether visual data is accessed, its weight, and the fusion method to adapt to different positioning conditions. The visual data can be understood as image-based perception data such as images, feature points, and visual poses collected by the visual assistance module, providing visual dimensional environment and pose information to assist in correcting LiDAR positioning results. The visual data includes, but is not limited to: image feature points, line features, semantic boundaries, road edges, lane lines, traffic signs, building outlines, fixed landmarks, and historical keyframe features. Visual-assisted positioning can be achieved using visual odometry, visual relocalization, visual map matching, semantically constrained positioning, or visual position regression methods based on deep networks. The positioning information can be understood as the final output of the target vehicle's precise position, attitude, coordinates, etc., providing high-precision positioning results usable by the entire vehicle to support autonomous driving, navigation, and other services.

[0047] Based on the above scheme, optionally, the fusion state includes confirmed fusion and weak participation; determining the fusion state of the visual assistance module according to the state information includes at least one of the following: when the state information is in a degenerate state, the fusion state of the visual assistance module is confirmed fusion; when the state information is in a normal state, the fusion state of the visual assistance module is weak participation.

[0048] The aforementioned confirmation fusion can be understood as a working mode in which the vision-assisted module fully participates in the positioning calculation and is deeply integrated with the LiDAR. When the LiDAR degrades, the positioning accuracy is enhanced by visual data, ensuring positioning reliability. The aforementioned weak participation can be understood as a working mode in which the vision-assisted module intervenes with low weight and the LiDAR takes the lead. When the LiDAR is functioning normally, the vision only performs auxiliary verification and does not dominate the positioning calculation.

[0049] This technical solution switches the fusion mode according to the operating conditions of the lidar. Under normal conditions, vision is involved to a low degree to ensure that the lidar dominates the positioning. When degradation occurs, deep fusion is activated to play a role in visual compensation. Sensor capabilities are adjusted as needed to balance positioning efficiency and accuracy and improve the overall adaptability of the system.

[0050] The technical solution of this invention collects LiDAR data of the target vehicle to accurately acquire environmental structure data, providing a reliable observation basis for positioning. It matches the LiDAR data with preset map point cloud data to determine the initial pose and acquires multiple distribution feature data during the matching process. This allows for the acquisition of the initial pose and feature extraction, providing a basis for degradation assessment. Based on the multiple distribution feature data, it determines the LiDAR degradation quantification index and the LiDAR state information, quantifying and assessing the degree of positioning degradation and accurately determining the radar's working state. Based on the state information, it determines the fusion state of the visual assistance module. Based on the fusion state, the visual data from the visual assistance module, the LiDAR data, and the initial pose, it determines the positioning information of the target vehicle. This allows for adaptive adjustment of the fusion strategy, achieving high-precision and robust vehicle positioning. It solves the problem of unstable LiDAR positioning in feature degradation scenarios, leading to decreased positioning accuracy in autonomous vehicles. Real-time judgment of the LiDAR positioning state and dynamic adjustment of the fusion strategy effectively improve positioning accuracy and operational stability.

[0051] Example 2 Figure 2This is a flowchart of a vehicle LiDAR vision-assisted localization method according to Embodiment 2 of the present invention. This embodiment is a further refinement of the determination of the target vehicle's localization information based on the fusion state, the visual data of the vision assistance module, the LiDAR data, and the initial pose, based on the above embodiments. Optionally, determining the target vehicle's localization information based on the fusion state, the visual data of the vision assistance module, the LiDAR data, and the initial pose includes at least one of the following: when the fusion state is confirmed fusion, acquiring the visual data and visual confidence of the vision assistance module, determining a first fusion weight of the visual data based on the visual confidence, determining a second fusion weight of the LiDAR data based on the degradation quantization index, determining a third fusion weight of the initial pose based on the first fusion weight and the second fusion weight, and determining a third fusion weight of the initial pose based on the visual data and the first fusion weight of the visual data, the LiDAR data and the second fusion weight of the LiDAR data, and the initial pose. The initial pose and its third fusion weight are weighted and fused together. The positioning information of the target vehicle is determined based on the weighted fusion result. In the case of weak participation in the fusion state, a preset fourth fusion weight of the visual data from the visual assistance module is obtained, as well as a fifth fusion weight of the LiDAR data. A sixth fusion weight of the initial pose is determined based on the fourth and fifth fusion weights. Weighted fusion is then performed based on the visual data and its fourth fusion weight, the LiDAR data and its fifth fusion weight, and the initial pose and its sixth fusion weight. The positioning information of the target vehicle is determined based on the weighted fusion result. For detailed implementation, please refer to the description of this embodiment. Technical features that are the same as or similar to those in the foregoing embodiments will not be repeated here. S260-S270 can be performed selectively, either individually or in all of them; no specific limitation is made here.

[0052] like Figure 2 As shown, the method may specifically include: S210: Collect lidar data of the target vehicle.

[0053] S220. Match the lidar data and the preset map point cloud data to determine the initial pose, and obtain multiple distribution feature data during the matching process, wherein the distribution feature data is used to characterize the state information of lidar positioning.

[0054] S230. Determine the degradation quantization index of the lidar based on multiple distribution feature data, and determine the state information of the lidar based on the degradation quantization index.

[0055] S240. Determine the fusion state of the vision assistance module based on the state information. If the fusion state is confirmed fusion, acquire the visual data and visual confidence of the vision assistance module. Determine the first fusion weight of the visual data based on the visual confidence. Determine the second fusion weight of the lidar data based on the degradation quantization index. Determine the third fusion weight of the initial pose based on the first fusion weight and the second fusion weight. Perform weighted fusion based on the visual data and the first fusion weight of the visual data, the lidar data and the second fusion weight of the lidar data, and the initial pose and the third fusion weight of the initial pose. Determine the positioning information of the target vehicle based on the weighted fusion result.

[0056] The visual confidence level can be understood as an evaluation index representing the reliability of the current visual data, including but not limited to the number of effective feature points, the uniformity of feature point distribution, image clarity, reprojection error, visual matching inlier rate, and dynamic region proportion, used to dynamically adjust the degree of trust in the visual data during fusion. The first fusion weight can be understood as the weight coefficient of the visual data in weighted fusion, reflecting its contribution to the final positioning, and is determined by the visual confidence level when confirming fusion. The second fusion weight can be understood as the weight coefficient of the LiDAR data in weighted fusion, negatively correlated with the degradation quantification index: the more severe the degradation, the lower the weight. The third fusion weight can be understood as the weight coefficient of the initial pose in weighted fusion, which can be determined by the first fusion weight and the second fusion weight. The sum of the first fusion weight, the second fusion weight, and the third fusion weight is one unit. The result of weighted fusion can be understood as the comprehensive pose estimate (position and attitude, etc.) obtained by weighted fusion calculation, which is a numerical result directly used for vehicle positioning output.

[0057] An optional implementation method may employ extended Kalman filtering, error-state Kalman filtering, factor graph optimization, sliding window optimization, or other multi-sensor joint estimation methods. Under normal conditions, the LiDAR localization result is dominant, with visual information serving only as a weak auxiliary or not participating in the fusion. In mildly degraded scenarios, the visual weight is appropriately increased while maintaining the dominant position of the LiDAR to enhance localization robustness. In severely degraded scenarios, the LiDAR weight is significantly reduced, while the constraint effects of vision and inertial navigation are increased to ensure continuous and stable pose estimation. When both LiDAR and vision are simultaneously unreliable, the system enters a degraded mode, performing short-term predictions based on inertial navigation and historical poses, and setting continuity constraints on changes in the output pose.

[0058] Based on the above scheme, optionally, the first fusion weight of the visual data is determined based on the following formula: ; in, Indicates the first fusion weight. Indicates visual confidence level. Indicates the upward slope, This represents the preset threshold for visual confidence.

[0059] The upward slope can be understood as a parameter representing the rate of change of weight with visual confidence, controlling the magnitude and speed of the weight value change. Exceed Subsequently, the first fusion weight increases rapidly and smoothly. The visual confidence threshold can be understood as a pre-set visual confidence critical value, used to determine whether the visual data has reached a level "worth strengthening the fusion". This technical solution dynamically calculates weights based on visual confidence and flexibly adjusts the proportions by combining thresholds and slopes, allowing the contribution of visual data to match its own quality and improving the rationality of fusion positioning.

[0060] Based on the above scheme, optionally, the second fusion weight of the lidar data can be determined based on the following formula: ; in, Indicates the second fusion weight. This indicates the maximum weight of the laser under normal conditions. This represents the preset degradation sensitivity coefficient. This represents a quantitative indicator of degradation.

[0061] The maximum weight of the laser under normal conditions can be understood as the upper limit of the weight that can be set when the lidar is working normally, serving as a benchmark value to construct the basic parameters for weight calculation. The degradation sensitivity coefficient can be understood as a pre-set constant that reflects the sensitivity of the weight to the degree of degradation, controlling the magnitude of weight changes with degradation indicators.

[0062] This technical solution combines the dynamic calculation of weights based on the degree of radar degradation, and adaptively adjusts the baseline weights and sensitivity coefficients to mitigate the impact of abnormal data and ensure stable and reliable fusion positioning.

[0063] S250. When the fusion state is weakly involved, obtain the preset fourth fusion weight of the visual data of the visual assistance module, and obtain the preset fifth fusion weight of the lidar data. Determine the sixth fusion weight of the initial pose based on the fourth fusion weight and the fifth fusion weight. Perform weighted fusion based on the visual data and the fourth fusion weight of the visual data, the lidar data and the fifth fusion weight of the lidar data, and the initial pose and the sixth fusion weight of the initial pose. Determine the positioning information of the target vehicle based on the weighted fusion result.

[0064] The fourth fusion weight can be understood as a pre-set visual data weight (relatively small), which does not change significantly with confidence level, limiting visual influence and mainly serving as a redundancy check or subtle smoothing function. The fifth fusion weight can be understood as the fusion weight of LiDAR data under weak involvement (relatively large), ensuring that LiDAR remains the primary positioning source. The sixth fusion weight can be understood as the fusion weight of the initial pose under weak involvement, determined by the fourth and fifth fusion weights. The sum of the fourth, fifth, and sixth fusion weights is one unit.

[0065] Based on the above scheme, the vehicle's current positioning information (position and attitude), positioning confidence, LiDAR degradation status, visual availability status, and current fusion mode identifier can be output.

[0066] To facilitate interaction with upper-level modules of the autonomous driving system, the output results may also include whether a degraded mode has been entered, the current dominant sensor type, and a positioning result availability flag.

[0067] To suppress sudden jumps in positioning results, constraints can be set on the pose increments of adjacent time steps, and a gradual weight back-cut strategy can be adopted in the recovery phase to enable the system to smoothly recover from the degradation mode to the normal mode.

[0068] The technical solution of this invention distinguishes between two fusion modes. When the lidar degrades, the weights of each data are dynamically calculated based on visual confidence and degradation indicators. Under normal working conditions, a fixed weight configuration is used. The initial pose weight is determined by combining the two types of weights, and the weighted fusion of multi-source data is completed. The data proportion is adaptively allocated according to the actual working state of the sensor, giving full play to the advantages of different data sources, effectively reducing the interference caused by the abnormality of a single sensor, and comprehensively improving the accuracy, stability and environmental adaptability of vehicle positioning in different scenarios.

[0069] Example 3 Figure 3 This is a schematic diagram of the structure of a visual-assisted positioning device for a vehicle lidar according to Embodiment 3 of the present invention. Figure 3 As shown, the device includes: a data acquisition module 310, a feature acquisition module 320, a status information determination module 330, and a positioning information determination module 340.

[0070] The acquisition module 310 is used to acquire LiDAR data of the target vehicle; the feature acquisition module 320 is used to match the LiDAR data with preset map point cloud data to determine the initial pose and acquire multiple distribution feature data during the matching process; the state information determination module 330 is used to determine the degradation quantization index of the LiDAR based on the multiple distribution feature data and determine the state information of the LiDAR based on the degradation quantization index; the positioning information determination module 340 is used to determine the fusion state of the vision assistance module based on the state information and determine the positioning information of the target vehicle based on the fusion state, the visual data of the vision assistance module, the LiDAR data and the initial pose.

[0071] The technical solution of this invention collects LiDAR data of the target vehicle through an acquisition module, accurately acquiring environmental structure data to provide a reliable observation basis for positioning; a feature acquisition module matches the LiDAR data with preset map point cloud data to determine the initial pose and acquires multiple distribution feature data during the matching process, enabling the acquisition of the initial pose and feature extraction, providing a basis for degradation assessment; a state information determination module determines the degradation quantification index of the LiDAR based on the multiple distribution feature data, determines the state information of the LiDAR based on the degradation quantification index, quantifies and assesses the degree of positioning degradation, and accurately determines the working state of the LiDAR; a positioning information determination module determines the fusion state of the visual assistance module based on the state information, and determines the positioning information of the target vehicle based on the fusion state, the visual data of the visual assistance module, the LiDAR data, and the initial pose, adaptively adjusting the fusion strategy to achieve high-precision robust vehicle positioning, solving the problem of unstable LiDAR positioning in feature degradation scenarios, which leads to a decrease in positioning accuracy of autonomous vehicles, and dynamically adjusting the fusion strategy in real time to effectively improve positioning accuracy and operational stability.

[0072] Optionally, the state information determination module includes a degradation quantification index determination submodule. This degradation quantification index determination submodule is used to normalize multiple distribution feature data to obtain multiple normalized index information; and to determine the degradation quantification index of the lidar based on the weighted sum of the multiple normalized index information.

[0073] Optionally, the status information includes a normal state and a degraded state; the status information determination module includes a status information determination submodule. The status information determination submodule is used to determine the status information of the lidar as normal when the degradation quantification index is not greater than a preset first threshold; and to determine the degradation state of the lidar based on the degradation quantification index and a preset second threshold when the degradation quantification index is greater than the first threshold.

[0074] Optionally, the degradation state includes a first degradation state and a second degradation state; the state information determination submodule is specifically used to: determine the state information of the lidar as a first degradation state when the degradation quantification index is not greater than the second threshold; and determine the state information of the lidar as a second degradation state when the degradation quantification index is greater than the second threshold.

[0075] Optionally, the fusion state includes confirmed fusion and weak participation; the positioning information determination module includes a fusion state determination submodule. Specifically, the fusion state determination submodule is used to determine the fusion state of the visual assistance module as confirmed fusion when the state information is in a degraded state, and as weak participation when the state information is in a normal state.

[0076] Optionally, the positioning information determination module includes a positioning information determination submodule. This submodule is used to, when the fusion state is confirmed fusion, acquire visual data and visual confidence from the visual assistance module, determine a first fusion weight for the visual data based on the visual confidence, determine a second fusion weight for the LiDAR data based on the degradation quantization index, determine a third fusion weight for the initial pose based on the first and second fusion weights, and perform weighted fusion based on the visual data and the first fusion weight, the LiDAR data and the second fusion weight, and the initial pose and the third fusion weight. The positioning information of the target vehicle is determined based on the weighted fusion result; when the fusion state is weak participation, a preset fourth fusion weight of the visual data of the visual assistance module is obtained, and a preset fifth fusion weight of the lidar data is obtained. A sixth fusion weight of the initial pose is determined based on the fourth fusion weight and the fifth fusion weight. Weighted fusion is performed based on the visual data and the fourth fusion weight of the visual data, the lidar data and the fifth fusion weight of the lidar data, and the initial pose and the sixth fusion weight of the initial pose. The positioning information of the target vehicle is determined based on the weighted fusion result.

[0077] Optionally, the first fusion weight of the visual data is determined based on the following formula: ; in, Indicates the first fusion weight. Indicates visual confidence level. Indicates the upward slope, This represents the preset threshold for visual confidence.

[0078] Optionally, the second fusion weight of the lidar data is determined based on the following formula: ; in, Indicates the second fusion weight. This indicates the maximum weight of the laser under normal conditions. This represents the preset degradation sensitivity coefficient. This represents a quantitative indicator of degradation.

[0079] The visual aid positioning device for vehicle LiDAR provided in this embodiment of the invention can execute the visual aid positioning method for vehicle LiDAR provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0080] Example 4 Figure 4 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0081] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0082] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0083] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as a vision-assisted localization method for a vehicle's LiDAR.

[0084] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from ROM 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of the embodiments of the present invention.

[0085] In some embodiments, a vision-assisted localization method for a vehicle lidar can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the vision-assisted localization method for a vehicle lidar described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform a vision-assisted localization method for a vehicle lidar by any other suitable means (e.g., by means of firmware).

[0086] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0087] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0088] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0089] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0090] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0091] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0092] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0093] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A visual-assisted positioning method for vehicle lidar, characterized in that, include: Collect lidar data of the target vehicle; The initial pose is determined by matching the lidar data with the preset map point cloud data, and multiple distribution feature data are obtained during the matching process. The degradation quantization index of the lidar is determined based on multiple distribution feature data, and the state information of the lidar is determined based on the degradation quantization index. The fusion state of the vision assistance module is determined based on the state information, and the positioning information of the target vehicle is determined based on the fusion state, the visual data of the vision assistance module, the lidar data, and the initial pose.

2. The method according to claim 1, characterized in that, The step of determining the degradation quantification index of the lidar based on multiple distribution feature data includes: The multiple distribution feature data are normalized to obtain multiple normalized index information; The degradation quantification index of lidar is determined based on the weighted summation of multiple normalized index information.

3. The method according to claim 1, characterized in that, The status information includes normal status and degraded status, and determining the status information of the lidar based on the degradation quantification index includes at least one of the following: If the degradation quantification index is not greater than a preset first threshold, the state information of the lidar is determined to be in a normal state; If the degradation quantification index is greater than the first threshold, the degradation state of the lidar is determined based on the degradation quantification index and the preset second threshold.

4. The method according to claim 3, characterized in that, The degradation state includes a first degradation state and a second degradation state, wherein the degradation degree of the second degradation state exceeds that of the first degradation state; determining the degradation state of the lidar based on the degradation quantification index and a preset second threshold includes at least one of the following: If the degradation quantification index is not greater than the second threshold, the state information of the lidar is determined to be a first degradation state; If the degradation quantification index is greater than the second threshold, the state information of the lidar is determined to be in the second degradation state.

5. The method according to claim 1, characterized in that, The fusion state includes confirmed fusion and weak participation; determining the fusion state of the visual assistance module based on the state information includes at least one of the following: When the status information is in a degraded state, the fusion status of the visual assistance module is confirmed fusion; When the status information is in a normal state, the fusion state of the visual assistance module is weak participation.

6. The method according to claim 1, characterized in that, Determining the positioning information of the target vehicle based on the fusion state, the visual data from the visual assistance module, the lidar data, and the initial pose includes at least one of the following: When the fusion state is confirmed as fusion, the visual data and visual confidence of the visual assistance module are acquired. A first fusion weight of the visual data is determined based on the visual confidence. A second fusion weight of the lidar data is determined based on the degradation quantification index. A third fusion weight of the initial pose is determined based on the first fusion weight and the second fusion weight. Weighted fusion is performed based on the visual data and the first fusion weight of the visual data, the lidar data and the second fusion weight of the lidar data, and the initial pose and the third fusion weight of the initial pose. The positioning information of the target vehicle is determined based on the weighted fusion result. When the fusion state is weakly involved, a preset fourth fusion weight is obtained from the visual data of the visual assistance module, and a preset fifth fusion weight is obtained from the lidar data. A sixth fusion weight for the initial pose is determined based on the fourth and fifth fusion weights. Weighted fusion is performed based on the visual data and the fourth fusion weight, the lidar data and the fifth fusion weight, and the initial pose and the sixth fusion weight. The positioning information of the target vehicle is determined based on the weighted fusion result.

7. The method according to claim 6, characterized in that, The first fusion weight of the visual data is determined based on the following formula: ； in, Indicates the first fusion weight. Indicates visual confidence level. Indicates the upward slope, This represents the preset threshold for visual confidence.

8. The method according to claim 6, characterized in that, The second fusion weight of the lidar data is determined based on the following formula: ； in, Indicates the second fusion weight. This indicates the maximum weight of the laser under normal conditions. This represents the preset degradation sensitivity coefficient. This represents a quantitative indicator of degradation.

9. A visual-assisted positioning device for vehicle lidar, characterized in that, include: The acquisition module is used to acquire LiDAR data from the target vehicle. The feature acquisition module is used to match the lidar data and preset map point cloud data to determine the initial pose and acquire multiple distribution feature data during the matching process, wherein the distribution feature data is used to characterize the state information of lidar positioning. The status information determination module is used to determine the degradation quantization index of the lidar based on multiple distribution feature data, and to determine the status information of the lidar based on the degradation quantization index. The positioning information determination module is used to determine the fusion state of the vision assistance module based on the state information, and to determine the positioning information of the target vehicle based on the fusion state, the visual data of the vision assistance module, the lidar data, and the initial pose.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, which is executed by the at least one processor to enable the at least one processor to perform the visual-assisted positioning method of the vehicle lidar according to any one of claims 1-8.

Images