A method, medium and device for determining the suitability of radar and visual data before fusion

Abstract

This application discloses a method, medium, and device for determining the suitability of radar-visual data before fusion. The method includes the following steps: acquiring point cloud and pixel data of the current frame and quantizing them separately; performing cross-modal matching calculations on the quantization results to obtain quality matching coefficients; simultaneously performing statistical tests on the current frame data to obtain distribution anomaly scores; adaptively updating weight coefficients and judgment thresholds based on the differences in fusion results of historical frames; and, provided that the point cloud and pixel quality meet the basic fusion conditions, calculating a comprehensive suitability score using the updated weight coefficients and comparing it with the updated threshold to determine whether the current frame is suitable for fusion. The medium and device are used to implement the above method. The beneficial effects of this application are: quantizing point cloud and image separately before fusion, combining cross-modal matching and distribution anomaly detection, and updating parameters online based on historical fusion errors to form a closed-loop adaptive screening, significantly improving the accuracy of suitability determination and scene generalization ability.

Description

Technical Field

[0001] This application relates to the field of radar-visual data fusion technology, and in particular to a method, medium and device for determining compatibility before radar-visual data fusion. Background Technology

[0002] With the rapid development of autonomous driving, robotic environmental perception, and 3D reconstruction, single sensors often struggle to provide comprehensive and robust perception information in complex dynamic environments. LiDAR can acquire high-precision 3D spatial geometric information but lacks texture and color; visual cameras provide rich semantic and color information but are susceptible to lighting conditions and weather. Therefore, the fusion of LiDAR point clouds and visual images has become a key technical path to improve the performance of perception systems. The effectiveness of fusing point cloud data and visual pixel data highly depends on the quality and compatibility of the two heterogeneous data types before fusion; however, existing data fusion methods suffer from the following technical shortcomings: (1) Evaluation lag: The multi-focused approach to verifying the effects of fusion lacks a systematic quantitative evaluation system before fusion, making it impossible to identify incompatible data in advance, resulting in a waste of computing power.

[0003] (2) Rigid point cloud evaluation: The use of fixed voxels and single density statistics cannot be adapted to the actual collection scenarios with uneven density, resulting in low evaluation accuracy.

[0004] (3) Pixel noise misjudgment: relying solely on grayscale threshold to determine noise points can easily lead to misjudging edges and textures as noise, resulting in texture semantic distortion.

[0005] (4) Poor robustness of the judgment: The adaptability judgment relies on hard threshold and forced Gaussian test, which is not robust, easy to misjudge, and difficult to match the real collection environment.

[0006] (5) Lack of adaptive capability: There is no parameter self-optimization mechanism, the generalization capability is insufficient, and it cannot adapt to the dynamic changes of different devices and different scenarios. Summary of the Invention

[0007] One objective of this application is to provide a method for determining the compatibility of radar-visual data before fusion, which can solve at least one of the defects in the above-mentioned background art.

[0008] Another object of this application is to provide a computer-readable storage medium capable of implementing a method for determining the compatibility of radar-visual data before fusion, which addresses at least one of the deficiencies in the aforementioned background art.

[0009] Another object of this application is to provide an electronic device capable of implementing a method for determining the compatibility of radar-visual data fusion before addressing at least one of the deficiencies in the aforementioned background art.

[0010] To achieve at least one of the above objectives, one aspect of this application provides a method for determining the compatibility of radar-visual data before fusion, comprising the following steps: S100: Obtain the point cloud data and pixel data to be fused in the current frame, and perform quantization based on the point cloud density distribution and image noise distribution respectively; S200: Perform cross-modal quality matching calculations on the quantized point cloud quality data and pixel quality data to obtain quality matching coefficients; S300: Perform statistical tests on the point cloud data and pixel data to be fused in the current frame to obtain the distribution anomaly degree; S400: Based on the differences in the fusion results of historical frames, update the weight coefficients and judgment thresholds of the current frame; S500: When the point cloud quality data and pixel quality data meet the basic fusion conditions, the point cloud quality data, pixel quality data, quality matching coefficient and distribution anomaly are fused based on the updated weight coefficients to obtain a comprehensive suitability score, which is then compared with the updated judgment threshold to determine suitability.

[0011] Preferably, in step S100, the quantization process for point cloud data based on point cloud density distribution is as follows: the point cloud data is divided into sparse and dense regions, and the local point cloud density corresponding to each sparse and dense region is calculated; the point cloud data is divided into voxels based on the obtained local point cloud density to obtain the effective number of voxels corresponding to the point cloud data; the global average density is calculated based on the ratio of the total number of points in the point cloud data to the effective number of voxels; the density stability coefficient of the point cloud data based on the density standard deviation is calculated based on the global average density and the local point cloud density corresponding to each sparse and dense region; the number of connected points in the point cloud data is counted based on the Euclidean distance between adjacent points in the point cloud data, and the spatial continuity coefficient of the point cloud data is calculated based on the ratio of the number of connected points to the total number of points in the point cloud data; the density stability coefficient and the spatial continuity coefficient are weighted to obtain the comprehensive point cloud quality coefficient.

[0012] Preferably, when dividing the point cloud data of the current frame into voxels for each density region, the voxel scale corresponding to each density region is adapted to be adaptively adjusted according to the distribution anomaly degree of the previous frame; the calculation expression for the voxel scale of each density region in the current frame is: ; ; In the formula, v i Let ρ represent the voxel scale of the i-th density region, k(t) represent the adaptive coefficients of the current frame t, and ρ i Let k represent the local point cloud density corresponding to the i-th density region, k0 represent the basic adaptive coefficient, and β represent the feedback strength. This represents the distribution anomaly degree corresponding to frame t-1.

[0013] Preferably, in step S100, the quantization process for pixel data based on image noise distribution is as follows: for any single pixel in the pixel data, calculate the gray-scale mean of its defined neighborhood range; for each pixel in the pixel data, determine noise based on the difference between the gray-scale value of the pixel center point and the gray-scale mean of its neighborhood range, combined with the gradient magnitude of the pixel; calculate the noise percentage based on the ratio of the number of noise points in the pixel data to the total number of pixels; calculate the noise distribution dispersion coefficient based on the standard deviation of the noise points' coordinates and the mean of the noise coordinates in the pixel data; and obtain the comprehensive pixel quality coefficient based on the weighted sum of the noise percentage and the noise distribution dispersion coefficient.

[0014] Preferably, when determining noise in a pixel, if the absolute value of the difference between the gray value of the pixel's center point and the average gray value in its neighborhood is greater than the gray-level difference threshold of the current frame, and the gradient magnitude of the pixel is less than the gradient threshold of the current frame, then the pixel is determined to be noise. The gray-level difference threshold and gradient threshold of the current frame are adaptively adjusted based on the spatial distribution anomaly of noise in the previous frame, as shown in the following expression: ; ; In the formula, This represents the grayscale difference threshold of the current frame t. This represents the basic grayscale difference threshold. and Both represent feedback intensity coefficients. This indicates the spatial distribution anomaly of image noise in frame t-1. This represents the gradient threshold for the current frame t. This represents the basic gradient threshold.

[0015] Preferably, in step S200, the quality matching coefficient includes a linearly weighted quality matching coefficient and an exponential continuous matching coefficient, and the specific calculation expression is as follows: ; ; In the formula, S represents the linear weighted quality matching coefficient, and ω1, ω2, ω3, and ω4 all represent weighting coefficients. ρ represents the global average density. max Q represents the maximum local point cloud density. d R represents the density stability coefficient. g C represents the noise percentage. g Q represents the noise distribution dispersion coefficient. matchQ represents the exponential continuous matching coefficient, α represents the matching adjustment coefficient, and Q represents the matching coefficient. p Q represents the overall point cloud quality coefficient. i This represents the overall image quality coefficient.

[0016] Preferably, in step S200, the quality matching coefficient further includes a structural consistency index. The acquisition of the structural consistency index includes the following process: projecting point cloud data onto the image plane corresponding to the pixel data to obtain the pixel coordinates and depth value corresponding to each projection point; interpolating the projection points in the pixel coordinate system to generate a sparse depth map and calculating the depth gradient magnitude; obtaining a depth edge binary map and extracting the first edge point set by performing non-maximum suppression and thresholding on the obtained depth gradient magnitude; calculating the gradient magnitude of the pixel data and obtaining an image edge binary map through non-maximum suppression and thresholding, and extracting the second edge point set from the image edge binary map; using direction-weighted Chamfer distance to measure the matching degree between the first and second edge point sets to obtain the structural consistency index.

[0017] Preferably, in step S400, the update formula for the weight coefficients and judgment threshold of the current frame is as follows: ; ; In the formula, and These represent the weighting coefficients corresponding to the current frame t and the historical frame t-1, respectively. η represents the learning rate, and E represents the historical fusion error. This indicates the impact of historical fusion error E on the weighting coefficients. The partial derivative, and These represent the judgment thresholds for the current frame t and the historical frame t-1, respectively, and λ represents the threshold adjustment coefficient. This represents the preset target fusion error.

[0018] Another aspect of this application provides a computer-readable storage medium storing a computer program; when the computer program is executed by a processor, it implements the above-described method for determining compatibility before radar-visual data fusion.

[0019] Another aspect of this application provides an electronic device, including a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program to implement the above-described adaptation determination method before radar-visual data fusion.

[0020] Compared with the prior art, the beneficial effects of this application are as follows: (1) Before fusion, this application independently quantifies the point cloud density distribution and image noise distribution, and introduces cross-modal quality matching coefficient and statistical distribution anomaly degree to avoid the lag and bias caused by traditional methods that rely on post-fusion verification or single modal indicators. At the same time, the weight coefficient and judgment threshold are updated online based on the differences in historical frame fusion results, so that the evaluation criteria are automatically adjusted with environmental changes, which solves the problem of poor robustness of fixed parameters under different scenarios and different devices.

[0021] (2) This application first uses the basic fusion conditions of point cloud and pixel to quickly remove obviously unqualified data, and then uses updated weight coefficients to fuse point cloud quality, pixel quality, matching coefficient and anomaly into a comprehensive suitability score, and compares it with dynamic threshold to make a final judgment. The two-layer structure not only avoids the waste of computing power caused by invalid data entering the subsequent fusion process, but also ensures the precision and reliability of the judgment result through weighted synthesis and adaptive threshold. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the overall working steps of this application. Detailed Implementation

[0023] The present application will now be further described in conjunction with specific embodiments. It should be noted that, in the description of this specification, the use of terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicates that the specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0024] In the description of this application, it should be noted that the terms "center", "lateral", "longitudinal", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc., which indicate the orientation and positional relationship based on the orientation or positional relationship shown in the accompanying drawings, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and should not be construed as limiting the specific protection scope of this application.

[0025] It should be noted that the terms "first," "second," etc., in the specification and claims of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0026] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "joining," and "fixing," etc., should be interpreted broadly. For example, they can refer to a connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0027] In this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature being directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature being directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0028] The terms “comprising” and “having”, and any variations thereof, in the specification and claims of this application are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0029] One preferred embodiment of this application, such as Figure 1 As shown, a method for determining the compatibility of radar-visual data before fusion includes the following steps: S100: Obtain the point cloud data and pixel data to be fused in the current frame, and perform quantization based on the point cloud density distribution and image noise distribution respectively.

[0030] It should be understood that the density distribution of point clouds includes both density and uniformity, which directly affects the reliability of spatial sampling; the noise distribution of images includes the number of noise points and their clustering, which directly affects the realism of texture features. Traditional fusion methods often use global or fixed parameters for coarse estimation, which cannot provide accurate data input for subsequent cross-modal matching.

[0031] This step involves independently quantizing the point cloud density distribution and image noise distribution to obtain quantifiable metrics that reflect single-modal data, namely point cloud quality data and pixel quality data. These quantization results can be used for basic qualification criteria judgment (rapidly removing obviously abnormal frames) and also serve as important factors for subsequent cross-modal matching and comprehensive scoring, achieving effective characterization of data quality before fusion.

[0032] S200: Perform cross-modal quality matching calculations on the quantized point cloud quality data and pixel quality data to obtain quality matching coefficients.

[0033] It should be understood that point cloud quality and pixel quality represent only the independent levels of their respective modalities, and there may be differences in quality or a complementary relationship between them. For example, the point cloud density may be extremely high, but the image may be severely overexposed, or both may have moderate quality but be complementary. Current technology lacks a mechanism for unified comparison and matching degree calculation of two types of heterogeneous quality indicators, making it impossible to determine whether the two modalities are compatible.

[0034] This step uses cross-modal quality matching calculations to fuse point cloud quality data and pixel quality data into one or a set of matching coefficients, intuitively quantifying the degree of synergy between the two modalities. These coefficients can reflect whether the "point cloud-image" quality is comparable or complementary on the same benchmark, providing crucial matching dimension information for subsequent comprehensive suitability scoring.

[0035] S300: Perform statistical tests on the point cloud data and pixel data to be fused in the current frame to obtain the distribution anomaly degree.

[0036] It should be understood that even point cloud or pixel images of acceptable quality may have anomalies in their internal density or noise distribution (e.g., local holes in the point cloud, or clusters of noisy areas in the image). These anomalies are difficult to capture using simple global statistics (such as average density or noise ratio). Existing technologies often ignore or only remove outliers, causing abnormal distribution areas to be treated as normal data and included in the fusion process, introducing local distortion or holes.

[0037] This step uses statistical testing to quantify the degree to which the point cloud density distribution and image noise distribution deviate from the ideal or normal distribution, obtaining a distribution anomaly index. This index can sensitively identify non-random distribution patterns such as local clustering, sparseness, or shape distortion, and outputs the degree of anomaly in scalar form. This is used to penalize anomalous data in subsequent comprehensive scoring, thereby improving the robustness of the fusion system to local anomalous regions.

[0038] S400: Based on the differences in the fusion results of historical frames, update the weight coefficients and judgment thresholds of the current frame.

[0039] It should be noted that existing quality assessment and adaptability determination methods mostly use fixed weights and fixed thresholds, which cannot adapt to different scenarios, different sensors, or acquisition conditions that change over time. When the environment changes (such as entering a tunnel or encountering rain or fog) or the sensor degrades, fixed determination parameters will lead to a large number of misjudgments (judging qualified frames as unqualified, or vice versa), resulting in poor system generalization ability.

[0040] This step introduces the differences in historical frame fusion results as a feedback signal to adaptively update the weight coefficients (for subsequent comprehensive scoring) and judgment thresholds (for final determination) of the current frame online. This mechanism enables the evaluation and decision criteria to be dynamically adjusted automatically according to scene changes and historical system performance, eliminating the need for repeated manual calibration and significantly improving the method's adaptability and stability under different devices and dynamic environments.

[0041] S500: When the point cloud quality data and pixel quality data meet the basic fusion conditions, the point cloud quality data, pixel quality data, quality matching coefficient and distribution anomaly are fused based on the updated weight coefficients to obtain a comprehensive suitability score, which is then compared with the updated judgment threshold to determine suitability.

[0042] It should be understood that using only one type of indicator (such as point cloud quality or matching coefficient alone) cannot comprehensively assess the overall adaptability before fusion, and lacks a rapid filtering mechanism for obviously unqualified data. Existing technologies often use single-layer scoring or a single threshold, which can easily lead to problems such as "inferior data passing because of a good indicator" or "qualified data being misjudged due to local anomalies".

[0043] This step employs a two-tiered decision-making architecture: "basic conditions + comprehensive score." First, clearly unqualified frames are quickly eliminated using basic fusion conditions, avoiding unnecessary subsequent computation. Then, adaptively updated weight coefficients fuse multi-source information such as point cloud quality, pixel quality, matching coefficients, and distribution anomalies into a single comprehensive suitability score, which is compared with an equally adaptively updated judgment threshold to output the final decision. This scheme balances computational efficiency and decision accuracy, and because the weights and thresholds can dynamically change based on historical performance, the decision boundary is more reasonable and robust.

[0044] As can be seen from the above technical solution of this application, this application independently quantifies the point cloud density distribution and image noise distribution before fusion, and introduces cross-modal quality matching coefficients and statistical distribution anomalies, avoiding the lag and bias caused by traditional methods that rely on post-fusion verification or single-modal indicators. At the same time, the weight coefficients and judgment thresholds are updated online based on the differences in historical frame fusion results, so that the evaluation criteria are automatically adjusted with environmental changes, solving the problem of poor robustness of fixed parameters under different scenarios and devices.

[0045] Meanwhile, this application first uses the basic fusion conditions of point cloud and pixels to quickly remove obviously unqualified data, and then uses updated weight coefficients to fuse point cloud quality, pixel quality, matching coefficient, and anomaly degree into a comprehensive suitability score, which is then compared with a dynamic threshold to make a final judgment. The two-layer structure not only avoids the waste of computing power caused by invalid data entering the subsequent fusion process, but also ensures the precision and reliability of the judgment results through weighted synthesis and adaptive threshold.

[0046] For ease of understanding, each step of this application will be described in detail below.

[0047] In a specific embodiment, when performing step S100, the obtained point cloud data P includes n three-dimensional point clouds, and a single three-dimensional point cloud p i Includes three-dimensional coordinates (x) i y i , z i For the obtained pixel data I, the resolution is W×H (i.e., the total number of pixels corresponding to the pixel data), and the grayscale value of a single pixel is g. ij After obtaining the point cloud data P and pixel data I, preprocessing is required. For the point cloud data P, preprocessing includes: removing outlier points, filtering out isolated points, and eliminating sparse points, resulting in the preprocessed point cloud data P. pre For the preprocessing of pixel data I: Perform mean filtering on pixel data I within a defined neighborhood range to remove minor noise and preserve edge structure, resulting in preprocessed pixel data I. pre Simultaneously, timestamp alignment and extrinsic parameter calibration are performed on the point cloud data and pixel data to ensure spatiotemporal consistency. It should be noted that the specific implementation methods of the above preprocessing process are common knowledge to those skilled in the art, and therefore will not be described in detail here. After completing the preprocessing of the point cloud data and pixel data, a quantization process is performed.

[0048] In a specific example, the quantization process for point cloud data based on point cloud density distribution is as follows: S101: Divide the point cloud data into dense and sparse regions, and calculate the local point cloud density corresponding to each dense and sparse region.

[0049] Specifically, the preprocessed point cloud data P pre In three-dimensional space, a coarse rasterization is performed, with each raster corresponding to a density region. The number of points within each raster is counted to obtain the local point cloud density of each density region, which can be expressed by the expression: ρ i =n i / V i ; where ρ i Let n represent the local point cloud density corresponding to the i-th sparse-density region. iV represents the number of point clouds contained in the i-th density region. i Let represent the volume of the i-th sparse-density region. Region partitioning based on local density provides a refined density prior for subsequent adaptive voxel partitioning, ensuring that the evaluation metrics align with the true physical distribution.

[0050] S102: Based on the obtained local point cloud density, the point cloud data is divided into voxels to obtain the effective number of voxels corresponding to the point cloud data.

[0051] Specifically, during voxel segmentation, the voxel scale is dynamically adjusted based on the local point cloud density, automatically subdividing dense regions and coarsely segmenting sparse regions; the specific calculation expression for the voxel scale is as follows: .

[0052] In the formula, v i Let ρ represent the voxel scale corresponding to the i-th density region, k represent the adaptive coefficient, and ρ represent the voxel scale. i This represents the local point cloud density corresponding to the i-th sparse-density region.

[0053] S103: Calculate the global average density based on the ratio of the total number of points to the effective number of voxels in the point cloud data.

[0054] Specifically, global average density The specific calculation expression is as follows: .

[0055] In the formula, n total Represents point cloud data P pre The total number of points included, M represents the number of effective voxels.

[0056] S104: Calculate the density stability coefficient of point cloud data based on the density standard deviation, based on the global average density and the local point cloud density corresponding to each dense and sparse region.

[0057] Specifically, using only the global average density cannot reflect the degree of fluctuation in point cloud density, and high fluctuations often indicate occlusion or sensor malfunction; this step quantifies the relative stability of density using the density standard deviation. Specifically, the density standard deviation σ... ρ The specific calculation expression is as follows: .

[0058] In the formula, ρ j Let M represent the local point cloud density corresponding to the j-th effective voxel, and M represent the number of effective voxels. This represents the global average density.

[0059] Based on the calculated density standard deviation, the density stability coefficient Q can be calculated.d The specific calculation expression is as follows: .

[0060] In the formula, σ ρ Indicates the standard deviation of density. ε represents the global average density, and ε represents the minimum value, used to prevent division by zero.

[0061] It is important to know the density stability coefficient Q. d The closer the value is to 1, the more uniform the density distribution; therefore, the density stability coefficient Q... d It can be used as a measure of the overall quality of point cloud data.

[0062] S105: Based on the Euclidean distance between adjacent points in the point cloud data, count the number of connected points in the point cloud data, and calculate the spatial continuity coefficient of the point cloud data according to the ratio of the number of connected points to the total number of points in the point cloud data.

[0063] Specifically, the spatial continuity coefficient Q c The calculation expression is: .

[0064] In the formula, N connected N represents the number of connected points in the point cloud data. total This indicates the total number of point clouds in the point cloud data.

[0065] It's important to understand that Euclidean distance-based clustering can be used for adjacent points. The specific distance threshold can be set according to the actual needs of those skilled in the art; for example, the distance threshold could be set to 0.2m. In the point cloud data, any two adjacent points with an Euclidean distance less than 0.2m are grouped into the same connected component. When calculating the spatial continuity coefficient, the total number of points corresponding to the largest connected component can be taken as the number of connected points required for the calculation. That is, the structural integrity of the point cloud is quantified by the proportion of the largest connected component, which can directly reflect the continuity of the main target (such as vehicles or roads).

[0066] S106: Weight the density stability coefficient and the spatial continuity coefficient to obtain the comprehensive point cloud quality coefficient.

[0067] Specifically, the overall point cloud quality coefficient Q p The calculation expression is: .

[0068] In the formula, ω Qd and ω Qc Both represent weighting coefficients, ω Qd +ω Qc =1;Qd Q represents the density stability coefficient. c This represents the spatial continuity coefficient. For the weighting coefficient ω... Qd and ω Qc The specific value can be selected according to the actual needs of those skilled in the art; for example, ω can be chosen. Qd =0.6, ω Qc =0.4.

[0069] It is understandable that when performing voxel partitioning on the density regions of the point cloud data in the current frame in step S102, voxel adaptation needs to consider not only the local point cloud density but also the impact of density distribution anomalies on voxel partitioning. Using only local point cloud density for adaptive voxel partitioning would lead to voxel scales being insensitive to abnormal distributions. Therefore, in this embodiment, when executing step S300, the voxel scales corresponding to each density region can be adaptively adjusted based on the point cloud density distribution anomaly degree obtained from the previous frame. That is, when the point cloud density distribution is abnormal (such as extremely uneven density), the voxel scale range is automatically increased to improve evaluation stability; when the point cloud density distribution is normal, fine partitioning is maintained, achieving closed-loop adaptation of the evaluation criteria.

[0070] In a specific example, the expression for calculating the voxel scale of each density region in the current frame is as follows: ; .

[0071] In the formula, v i Let ρ represent the voxel scale of the i-th density region, k(t) represent the adaptive coefficients of the current frame t, and ρ i β represents the local point cloud density corresponding to the i-th sparse and dense region, k0 represents the basic adaptive coefficient, and the specific value can be set according to the actual needs of those skilled in the art, for example, 0.2; β represents the feedback intensity, and the specific value can be set according to the actual needs of those skilled in the art, for example, 0.5; This indicates the anomaly degree of the point cloud density distribution corresponding to frame t-1.

[0072] In a specific example, the process of quantizing pixel data based on the image noise distribution is as follows: S111: For any single pixel in the pixel data, calculate the average gray value within a defined neighborhood. .

[0073] It should be understood that the specific value of the neighborhood range can be set according to the actual needs of those skilled in the art. In this embodiment, a 3×3 neighborhood range is preferred. The specific calculation process of the grayscale mean within the neighborhood range is a well-known technique to those skilled in the art, and therefore will not be described in detail here.

[0074] S112: For each pixel in the pixel data, noise is judged based on the difference between the gray value of the pixel center point and the average gray value in its neighborhood, combined with the gradient magnitude of the pixel.

[0075] Specifically, for each pixel, if it simultaneously satisfies two noise detection conditions, then the pixel is determined to be noise. That is, when determining noise in a pixel, if the absolute value of the difference between the grayscale value of the pixel's center point and the average grayscale value within its neighborhood is greater than the grayscale difference threshold of the current frame, and simultaneously the gradient magnitude of the pixel is less than the gradient threshold of the current frame, then the pixel is determined to be noise. The calculation process for the grayscale value of the pixel's center point and the pixel's gradient magnitude are well-known techniques to those skilled in the art, and therefore will not be described in detail here. This dual-condition joint determination effectively distinguishes isolated noise from structural edges: edge pixels, although exhibiting abrupt grayscale changes, have high gradient magnitudes and will not be misjudged; only isolated outliers that simultaneously satisfy low gradient conditions are marked as noise.

[0076] S113: Calculate the noise percentage based on the ratio of the number of noise points in the pixel data to the total number of pixels.

[0077] It should be understood that after performing noise assessment on all pixels in the pixel data, the number of noise points corresponding to the pixel data can be obtained; the total number of pixels in the pixel data is the aforementioned resolution W×H; the noise ratio R can be obtained by the ratio of the two. g .

[0078] S114: Calculate the noise distribution dispersion coefficient based on the standard deviation of the noise point coordinates and the mean of the noise coordinates in the pixel data.

[0079] Noise distribution dispersion coefficient C g The specific calculation expression is as follows: .

[0080] In the formula, σ x and σ y Both represent the standard deviation of the noise coordinates, μ x and μ y All represent the mean of the noise point coordinates; the specific methods for calculating the standard deviation and mean of the coordinates are well known to those skilled in the art, and therefore will not be described in detail here.

[0081] It should be noted that a smaller noise distribution dispersion coefficient indicates a more dispersed noise distribution, and uniformly distributed noise has a relatively smaller impact on subsequent fusion; if the noise is concentrated in a certain area (C... g If the severity is significant, then severe punishment is necessary.

[0082] S115: The overall pixel quality coefficient is obtained by weighting the noise ratio and the noise distribution dispersion coefficient.

[0083] Overall pixel quality coefficient Q i The calculation expression is: .

[0084] In the formula, ω Rg and ω Cg Both represent weighting coefficients, ω Rg +ω Cg =1; R g C represents the noise percentage. g This represents the noise distribution dispersion coefficient. For the weighting coefficient ω... Rg and ω Cg The specific value can be selected according to the actual needs of those skilled in the art; for example, ω can be chosen. Rg =0.5, ω Cg =0.5.

[0085] It is understandable that when judging noise for each pixel of the current frame's pixel data in step S112, noise in a real image may exhibit different spatial distributions: sometimes it is randomly and uniformly distributed (such as thermal noise), and sometimes it is locally clustered due to environmental factors (raindrops, lens smudges, dark current caused by low illumination). Using a fixed threshold may work well for uniformly distributed noise, but it is prone to failure for clustered noise; that is, it may either misjudge the clustered area as texture (because clustered noise may also generate a certain gradient), or miss a large number of continuous noise. Therefore, when performing step S300 in this embodiment, the gray-level difference threshold and gradient threshold of the current frame can be adaptively adjusted according to the spatial distribution anomaly of the noise in the previous frame, so that when noise clustering is obvious, the gray-level difference threshold is appropriately relaxed and the gradient threshold is tightened.

[0086] In a specific example, the specific expressions for adaptively adjusting the grayscale difference threshold and gradient threshold of the current frame are as follows: ; .

[0087] In the formula, This represents the grayscale difference threshold of the current frame t. This represents the basic grayscale difference threshold. and Both represent feedback intensity coefficients. This indicates the spatial distribution anomaly of image noise in frame t-1. This represents the gradient threshold for the current frame t. This represents the basic gradient threshold.

[0088] In one specific embodiment, when performing step S200, the cross-modal quality matching calculation includes three indicators: linearly weighted quality matching coefficient, exponential continuous matching coefficient, and structural consistency index. For the linearly weighted quality matching coefficient, the relative density level and density stability of the point cloud are linearly combined with the noise ratio and dispersion coefficient of the image to intuitively reflect the quality coordination degree between the two modalities from a global statistical perspective. The closer the linearly weighted quality matching coefficient is to 1, the better the quality level of the point cloud data and pixel data matches. Since the linearly weighted quality matching coefficient can only reflect the overall quality matching level, it cannot sensitively capture the degree of difference between the point cloud and image quality. The exponential continuous matching coefficient can amplify the differences between the point cloud and the image, providing a smooth penalty for quality inconsistencies in subsequent comprehensive scoring. Because both the linearly weighted quality matching coefficient and the exponential continuous matching coefficient are global statistics, this results in a lack of judgment on the consistency of local geometric texture structure between the point cloud and the image. The structural consistency index can quantify the degree of matching between the depth edges of the point cloud projection and the gradient edges of the image.

[0089] In a specific example, the specific expression for calculating the linear weighted quality matching coefficient S is as follows: .

[0090] In the formula, ω1, ω2, ω3, and ω4 all represent weighting coefficients. The specific values ​​can be selected according to the actual needs of those skilled in the art. For example, ω1=0.3, ω2=0.2, ω3=0.3, and ω4=0.2. ρ represents the global average density. max Q represents the maximum local point cloud density. d R represents the density stability coefficient. g C represents the noise percentage. g This represents the dispersion coefficient of the noise distribution.

[0091] In a specific example, the exponential continuous matching coefficient Q match The specific calculation expression is as follows: .

[0092] In the formula, α represents the matching adjustment coefficient, and Q p Q represents the overall point cloud quality coefficient. i This represents the overall image quality coefficient.

[0093] In a specific example, obtaining the structural consistency metric involves the following process: S201: Project the point cloud data onto the image plane corresponding to the pixel data through the calibrated eccentricity and the intrinsic parameters of the vision camera, and obtain the pixel coordinates and depth values ​​corresponding to each projection point to form an initial depth map; in the pixel coordinate system, interpolate the projection points in the initial depth map to generate a sparse depth map.

[0094] S202: Calculate the depth gradient magnitude corresponding to the sparse depth map, and use the Canny edge detection method to perform non-maximum suppression and thresholding on the obtained depth gradient magnitude to obtain a depth edge binary map. In the depth edge binary map, the value corresponding to the edge point is 1, and the value corresponding to the non-edge point is 0; therefore, all points with a value of 1 can be extracted and summarized to obtain the first edge point set.

[0095] S203: Calculate the gradient magnitude for each pixel in the pixel data, and obtain the image edge binary map through non-maximum suppression and thresholding. Similarly, in the image edge binary map, the value corresponding to the edge point is 1, and the value corresponding to the non-edge point is 0. Therefore, all points with a value of 1 can be extracted and summarized to obtain the second edge point set.

[0096] S204: The direction-weighted Chamfer distance is used to measure the matching degree between the first edge point set and the second edge point set, resulting in the structural consistency index Q. struct The specific calculation expression is as follows: .

[0097] .

[0098] In the formula, and Represent the first edge point set E respectively pc Second edge point set E img The corresponding number of edge points in the set, p and q represent the first edge point set E respectively. pc Second edge point set E img The coordinates of any edge point in the middle. θ represents the Euclidean distance, and θ represents the normalization coefficient. The specific value can be determined by those skilled in the art based on their actual needs.

[0099] It should be understood that by comparing the geometric alignment of point cloud depth edges with image gradient edges, it is possible to detect structural inconsistencies caused by local extrinsic errors or dynamic objects. Orientation weighting further eliminates mismatches caused by different edge directions, giving the structural consistency index stronger discriminative power.

[0100] In a specific embodiment, when executing step S300, as described above, the distribution anomaly includes point cloud density distribution anomaly and image noise spatial distribution anomaly. This can be achieved by using the density values ​​of each effective voxel as samples and performing a KS test against a uniform distribution, taking the test statistic (maximum deviation) as the point cloud density distribution anomaly D. pc The location coordinates of each noise point can be regarded as a two-dimensional point set, and a chi-square test can be performed with a uniform distribution, or the coefficient of variation of the nearest neighbor distance can be calculated as the spatial distribution anomaly degree D of the image noise. img By employing nonparametric statistical tests (KS, chi-square) or nearest neighbor distance-based metrics, without pre-setting a distribution model, the degree to which the distribution deviates from the normal state can be objectively quantified. The calculated point cloud density distribution anomaly degree D... pc and noise spatial distribution anomaly degree D img It can be used for subsequent penalties (reducing the fit score) or fed back to step S100 for parameter adjustment. Point cloud density distribution anomaly degree D pc and noise spatial distribution anomaly degree D img The calculation expression is as follows: ; .

[0101] In the formula, σ ρ Indicates the standard deviation of density. ε represents the global average density, ε represents the minimum value used to prevent division by zero, and CV represents the coefficient of variation of the nearest neighbor distance of the noise point, which can be the standard deviation or mean of the nearest neighbor distance.

[0102] In a specific embodiment, when performing step S400, the update formulas for the weight coefficients and judgment thresholds of the current frame are as follows: ; .

[0103] In the formula, and These represent the weighting coefficients corresponding to the current frame t and the historical frame t-1, respectively. η represents the learning rate, which can be set according to the actual needs of those skilled in the art, for example, η=0.05; E represents the historical fusion error, which is the average error between the comprehensive suitability score calculated in the historical time window and the set ideal fusion score threshold. This indicates the impact of historical fusion error E on the weighting coefficients. The partial derivative, and These represent the judgment thresholds for the current frame t and the historical frame t-1, respectively, and λ represents the threshold adjustment coefficient. This represents the preset target fusion error, which can be set according to the actual needs of those skilled in the art, for example, taking... .

[0104] It should be understood that by backpropagating the errors from historical fusion results, the weight coefficients and judgment thresholds can be optimized online, enabling the system to automatically adapt to sensor degradation and environmental changes without the need for repeated manual calibration. Weight updates make the evaluation model closer to the actual fusion performance; threshold updates enable automatic adjustment of the decision criteria (increasing the threshold when the fusion error is higher than the target, making the admission more stringent, and vice versa).

[0105] In a specific embodiment, when executing step S500, fusion is only performed if the point cloud quality data and pixel quality data meet the basic fusion conditions. The basic fusion conditions for point cloud quality data and pixel quality data are as follows: for point cloud quality data, the global average density must be greater than or equal to the minimum local point cloud density, and the density stability coefficient must be greater than or equal to the set stability threshold; for pixel quality data, the noise ratio must be less than or equal to the set ratio threshold, and the noise distribution dispersion coefficient must be less than or equal to the set dispersion threshold.

[0106] It should be understood that fusion is only performed when both point cloud quality data and pixel quality data simultaneously meet the above conditions. The specific values ​​of the stability threshold, proportion threshold, and discrete threshold can be determined by those skilled in the art based on their actual needs; for example, the stability threshold could be 0.7, the proportion threshold 0.1%, and the discrete threshold 0.2.

[0107] In a specific example, the overall fit score Q final The calculation expression is: .

[0108] In the formula, w1, w2, w3, w4, w5, and w6 represent the updated weight coefficients, and Q... p Q represents the overall point cloud quality coefficient. i Q represents the overall image quality coefficient, S represents the linearly weighted quality matching coefficient, and Q represents the overall image quality coefficient. match Q represents the exponential continuous matching coefficient. struct The structure consistency index is represented by D, which represents the distribution anomaly degree, and its value is the point cloud density distribution anomaly degree D. pc and noise spatial distribution anomaly degree D img Weighted fusion.

[0109] It should be noted that when the calculated overall fit score Q... finalIf the value is greater than or equal to the updated judgment threshold T, the point cloud data and pixel data of the current frame are determined to be suitable for fusion; otherwise, they are determined to be unsuitable, and data augmentation or re-acquisition can be triggered.

[0110] To facilitate understanding of the technical solution of this application, the following will take the autonomous driving scenario of a vehicle on Chengdu roads as an example to describe the technical solution of this application in detail.

[0111] (1) Data acquisition and preprocessing.

[0112] The original LiDAR point cloud contained approximately 203,500 points. After removing outliers, filtering out isolated points, and eliminating sparse points, the preprocessed point cloud P was obtained. pre Total points n total =203497.

[0113] Vehicle-mounted camera image: 1920×1080 resolution, after mean filtering over a 3×3 neighborhood to remove minor noise while preserving edge structure, resulting in preprocessed image I. pre .

[0114] Spatiotemporal synchronization: Complete the timestamp alignment and extrinsic parameter calibration of point cloud and image to ensure spatiotemporal consistency of subsequent projection and matching.

[0115] (2) Quality evaluation of point cloud data.

[0116] Adaptive coefficient k=0.2, effective voxel count M=10200, global average density ≈19.95 units / cubic meter.

[0117] Density standard deviation ≈3.8, density stability coefficient Q d ≈0.81, stability threshold E th =0.7, indicating good balance.

[0118] Spatial continuity coefficient Q c ≈0.88, weight ω Qd =0.6、ω Qc =0.4, overall point cloud quality coefficient Q p ≈0.838.

[0119] (3) Quality evaluation of pixel data.

[0120] Threshold T g =30、T gra =15, total number of noise points N g =1245.

[0121] Noise R g ≈0.059%, the percentage threshold R th=0.1%, which meets the requirements.

[0122] Noise distribution dispersion coefficient C g ≈0.166, discrete threshold C th =0.2, the distribution is scattered.

[0123] weight ω Rg =0.5、ω Cg =0.5, overall pixel quality coefficient Q i ≈0.917.

[0124] (4) Cross-modal quality matching.

[0125] Calculation of linear weighted quality matching coefficient: local point cloud density maximum value ρ max =30, weights ω1=0.3, ω2=0.2, ω3=0.3, ω4=0.2, linear weighted quality matching coefficient S≈0.83.

[0126] Calculation of exponential continuous matching coefficients: Matching adjustment coefficient α=2, exponential continuous matching coefficient Q match ≈0.85.

[0127] Structural consistency index calculation: First edge point set E pc Corresponding to approximately 15,200 edge points, the second edge point set E img The middle corresponds to approximately 12,800 edge points, with a normalization coefficient θ = 10, and a structural consistency index Q. struct =0.815.

[0128] (5) Calculation of distribution anomaly.

[0129] Point cloud density distribution anomaly degree D pc =0.08, noise spatial distribution anomaly degree D img =0.1, and the distribution outlier based on average weighting is D=0.09.

[0130] (6) Adaptive parameter update.

[0131] Historical fusion error E=0.08, preset target fusion error E target =0.07; learning rate η=0.05, threshold adjustment coefficient λ=0.1; perform one parameter iteration update to obtain weights w1=0.18, w2=0.18, w3=0.14, w4=0.14, w5=0.12, w6=0.24; judgment threshold T≈0.7.

[0132] (7) Calculation of overall fitness score: Overall fitness score Q final≈0.18×0.838+0.18×0.917+0.14×0.83+0.14×0.85+0.12×0.815-0.24×0.09=0.6273<0.7, therefore it is judged as an unsuitable match.

[0133] Another aspect of this application provides a computer-readable storage medium, in a preferred embodiment of which a computer program is stored on the storage medium; when the computer program is executed by a processor, the above-described method for determining compatibility before radar-visual data fusion is implemented.

[0134] Another aspect of this application provides an electronic device, in a preferred embodiment of which includes a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program to implement the above-described adaptation determination method before radar-visual data fusion.

[0135] The basic principles, main features, and advantages of this application have been described above. Those skilled in the art should understand that this application is not limited to the above embodiments. The embodiments and descriptions in the specification are merely the principles of this application. Various changes and modifications can be made to this application without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claims. The scope of protection claimed by this application is defined by the appended claims and their equivalents.

Claims

1. A method for determining the compatibility of radar-visual data before fusion, characterized in that, Includes the following steps: S100: Obtain the point cloud data and pixel data to be fused in the current frame, and perform quantization based on the point cloud density distribution and image noise distribution respectively; S200: Perform cross-modal quality matching calculations on the quantized point cloud quality data and pixel quality data to obtain quality matching coefficients; S300: Perform statistical tests on the point cloud data and pixel data to be fused in the current frame to obtain the distribution anomaly degree; S400: Based on the differences in the fusion results of historical frames, update the weight coefficients and judgment thresholds of the current frame; S500: When the point cloud quality data and pixel quality data meet the basic fusion conditions, the point cloud quality data, pixel quality data, quality matching coefficient and distribution anomaly are fused based on the updated weight coefficients to obtain a comprehensive suitability score, which is then compared with the updated judgment threshold to determine suitability.

2. The compatibility determination method before radar-visual data fusion as described in claim 1, characterized in that, In step S100, the quantization process for the point cloud data based on the point cloud density distribution is as follows: Divide the point cloud data into sparse and dense regions, and calculate the local point cloud density corresponding to each sparse and dense region; Based on the obtained local point cloud density, the point cloud data is divided into voxels to obtain the effective number of voxels corresponding to the point cloud data. The global average density is calculated based on the ratio of the total number of points to the number of effective voxels in the point cloud data. Based on the global average density and the local point cloud density corresponding to each dense and sparse region, the density stability coefficient of the point cloud data based on the density standard deviation is calculated. Based on the Euclidean distance between adjacent points in the point cloud data, the number of connected points in the point cloud data is counted, and the spatial continuity coefficient of the point cloud data is calculated according to the ratio of the number of connected points to the total number of points in the point cloud data. The density stability coefficient and spatial continuity coefficient are weighted to obtain the comprehensive point cloud quality coefficient.

3. The compatibility determination method before radar-visual data fusion as described in claim 2, characterized in that, When dividing the point cloud data of the current frame into voxels for each dense and sparse region, the voxel scale corresponding to each dense and sparse region is adaptively adjusted according to the anomaly degree of the point cloud density distribution in the previous frame; the calculation expression for the voxel scale of each dense and sparse region in the current frame is as follows: ； ； In the formula, v i Let ρ represent the voxel scale of the i-th density region, k(t) represent the adaptive coefficients of the current frame t, and ρ i Let k represent the local point cloud density corresponding to the i-th density region, k0 represent the basic adaptive coefficient, and β represent the feedback strength. This indicates the anomaly degree of the point cloud density distribution corresponding to frame t-1.

4. The compatibility determination method before radar-visual data fusion as described in claim 2, characterized in that, In step S100, the quantization process for pixel data based on image noise distribution is as follows: For any single pixel in the pixel data, calculate the average gray value within a defined neighborhood range; For each pixel in the pixel data, noise is determined by the difference between the gray value of the pixel center point and the average gray value of its neighborhood, combined with the gradient magnitude of the pixel. The noise percentage is calculated based on the ratio of the number of noise points in the pixel data to the total number of pixels. The noise distribution dispersion coefficient is calculated based on the standard deviation of the noise points' coordinates and the mean of the noise coordinates in the pixel data. The overall pixel quality coefficient is obtained by weighting the noise percentage and the noise distribution dispersion coefficient.

5. The compatibility determination method before radar-visual data fusion as described in claim 4, characterized in that, When judging noise in a pixel, if the absolute value of the difference between the gray value of the pixel center point and the average gray value in its neighborhood is greater than the gray value difference threshold of the current frame, and the gradient magnitude of the pixel is less than the gradient threshold of the current frame, the pixel is judged as noise. The grayscale difference threshold and gradient threshold for the current frame are suitable for adaptive adjustment based on the spatial distribution anomaly of noise in the previous frame. The specific expressions are as follows: ； ； In the formula, This represents the grayscale difference threshold of the current frame t. This represents the basic grayscale difference threshold. and Both represent feedback intensity coefficients. This indicates the spatial distribution anomaly of image noise in frame t-1. This represents the gradient threshold for the current frame t. This represents the basic gradient threshold.

6. The compatibility determination method before radar-visual data fusion as described in claim 4, characterized in that, In step S200, the quality matching coefficients include linearly weighted quality matching coefficients and exponential continuous matching coefficients, and the specific calculation expressions are as follows: ； ； In the formula, S represents the linear weighted quality matching coefficient, and ω1, ω2, ω3, and ω4 all represent weighting coefficients. ρ represents the global average density. max Q represents the maximum local point cloud density. d R represents the density stability coefficient. g C represents the noise percentage. g Q represents the noise distribution dispersion coefficient. match Q represents the exponential continuous matching coefficient, α represents the matching adjustment coefficient, and Q represents the matching coefficient. p Q represents the overall point cloud quality coefficient. i This represents the overall image quality coefficient.

7. The compatibility determination method before radar-visual data fusion as described in claim 6, characterized in that, In step S200, the quality matching coefficient also includes a structural consistency index, and the acquisition of the structural consistency index includes the following process: The point cloud data is projected onto the image plane corresponding to the pixel data to obtain the pixel coordinates and depth value of each projection point. In the pixel coordinate system, the projected points are interpolated to generate a sparse depth map, and the depth gradient magnitude is calculated. By performing non-maximum suppression and thresholding on the obtained depth gradient magnitude, a binary image of the depth edge is obtained and the first edge point set is extracted. The gradient magnitude of the pixel data is calculated, and non-maximum suppression and thresholding are applied to obtain the binary image edge map. The second edge point set is then extracted from the binary image edge map. The direction-weighted Chamfer distance is used to measure the matching degree of the first edge point set and the second edge point set, thus obtaining the structural consistency index.

8. The method for determining compatibility before radar-visual data fusion as described in claim 1, characterized in that, In step S400, the update formulas for the weight coefficients and judgment thresholds of the current frame are as follows: ； ； In the formula, and These represent the weighting coefficients corresponding to the current frame t and the historical frame t-1, respectively. η represents the learning rate, and E represents the historical fusion error. This indicates the impact of historical fusion error E on the weighting coefficients. The partial derivative, and These represent the judgment thresholds for the current frame t and the historical frame t-1, respectively, and λ represents the threshold adjustment coefficient. This represents the preset target fusion error.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program; when the computer program is executed by a processor, it implements the compatibility determination method before radar-visual data fusion as described in any one of claims 1-8.

10. An electronic device, characterized in that, It includes a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program to implement the compatibility determination method before radar-visual data fusion as described in any one of claims 1-8.

Images