Intelligent data acquisition and processing system based on fusion of vision and laser radar

By fusing visual and lidar data and employing multi-source data alignment and optimization formulas, the data alignment problem caused by differences in sensor installation position and angle was solved, achieving high-precision and robust environmental perception.

CN118941905AInactive Publication Date: 2024-11-12SHANGHAI DIANJI UNIV
View PDF 0 Cites 6 Cited by

Patent Information

Application Number
CN202410985782.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-12
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In existing technologies, the different installation positions and angles of visual sensors and lidar sensors result in spatial and temporal differences in the collected data, making it difficult to achieve precise alignment and affecting the accuracy and robustness of environmental perception.

Method used

By fusing visual and lidar data, employing multi-source data alignment and optimization formulas, and combining environmental perception modules and weight parameter adjustments, precise data alignment is achieved.

Benefits of technology

It significantly improves the accuracy and robustness of environmental perception. The camera provides texture and color information, while the LiDAR provides high-precision 3D point cloud data, solving the data discrepancy problem and ensuring efficient operation in various environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941905A_ABST
    Figure CN118941905A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of intelligent system management, and particularly discloses an intelligent data acquisition and processing system based on vision and laser radar fusion, which is used for solving the problem of how to perceive environment change in different environments and realize accurate alignment of two kinds of data. Comprising a camera, a laser radar, a data preprocessing module, an environment sensing module, an environment sensing analysis module, a weight parameter adjustment module, a data alignment optimization module and a data processing analysis module. Visual and laser radar data are fused to improve environmental perception precision and robustness, a camera provides texture and color information, a laser radar provides high-precision three-dimensional point cloud data, a multi-source data alignment and optimization formula is adopted, data difference is solved, accurate alignment is realized, and the environmental perception precision and robustness are improved through an environmental perception module and weight parameter adjustment. And environment changes are dynamically adapted, and efficient operation under various conditions is ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of intelligent system management technology, and more specifically, to an intelligent data acquisition and processing system based on the fusion of vision and laser radar. Background Art

[0002] In the development of modern intelligent systems, accurate data collection and efficient processing are the key to many applications, especially in the fields of autonomous driving, robot navigation, intelligent monitoring, etc., where accurate and real-time environmental perception is essential. Visual sensors can provide rich texture and color information by capturing images formed by light reflection, which is very useful for object recognition and classification. However, the performance of visual sensors is seriously affected under poor lighting conditions. LiDAR generates three-dimensional point cloud data of the environment by emitting laser pulses and measuring the time it takes for them to reflect back. LiDAR can work stably under different lighting conditions and can provide high-precision distance measurement. However, the point cloud data generated by LiDAR is relatively poor in texture and color information. By fusing visual and LiDAR data, the advantages of these two sensors can be comprehensively utilized. The texture information and color information provided by visual sensors can make up for the shortcomings of LiDAR in object recognition, while the precise distance information of LiDAR can improve the accuracy of visual sensors in depth information, thereby improving the accuracy and robustness of environmental perception. However, visual sensors and LiDAR sensors are usually installed at different positions and angles, resulting in differences in the collected data in space and time. How to perceive environmental changes in different environments and achieve accurate alignment of the two data is a technical problem that needs to be solved urgently. Summary of the invention

[0003] In order to overcome the above-mentioned defects of the prior art, the present invention provides an intelligent data acquisition and processing system based on the fusion of vision and lidar, which improves the accuracy and robustness of environmental perception by fusing vision and lidar data. The camera provides texture and color information, and the lidar provides high-precision three-dimensional point cloud data. Multi-source data alignment and optimization formulas are used to solve data differences and achieve precise alignment. Through environmental perception modules and weight parameter adjustment, it can dynamically adapt to environmental changes to ensure efficient operation under various conditions.

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] The invention discloses an intelligent data acquisition and processing system based on the fusion of vision and laser radar, comprising a camera, a laser radar, a data preprocessing module, an environment perception module, an environment perception analysis module, a weight parameter adjustment module, a data alignment optimization module and a data processing analysis module. The environment perception module perceives the average light intensity and the comprehensive weather index in the image, and analyzes the obstacle complexity and the sensor noise level in the point cloud data. The environment perception analysis module uses the data obtained by the environment perception module as input, and inputs the data into a preset comprehensive environment perception coefficient formula to obtain the comprehensive environment perception coefficient. The weight parameter adjustment module obtains the adjustment strategy of the weight parameter under the category to which the environment perception coefficient belongs according to the preset environment perception coefficient classification standard, and obtains the weight coefficient. The data alignment optimization module uses the weighted multi-source data alignment formula to realize the preliminary alignment of the data, and uses the precision optimization formula to further optimize the preliminary alignment result. Combining the advantages of the two methods, the joint multi-source data optimization formula is used based on the weight parameter to realize the data fusion of the two data sources. The data processing analysis module extracts useful feature information according to the fused data of the data alignment optimization module, performs object recognition and scene understanding, and provides detailed environment analysis results.

[0006] As a further solution of the present invention, the camera and lidar are connected to the data preprocessing module, the data preprocessing module is connected to the environment module, the environment perception module is connected to the environment perception analysis module, the environment perception analysis module is connected to the weight parameter adjustment module, the weight parameter adjustment module is connected to the data alignment optimization module, and the data alignment optimization module is connected to the data processing and analysis module.

[0007] As a further solution of the present invention, in the environmental perception module, the light intensity of each pixel in the camera image is counted and its average value is obtained as the average light intensity. The comprehensive weather index is obtained by the average light intensity and rainfall of multiple camera images, wherein the rainfall of each image is identified by edge detection and morphological processing, and the area of ​​the raindrops is extracted and calculated using an empirical formula.

[0008] As a further solution of the present invention, in the environment perception module, the empirical formula for rainfall is:

[0009]

[0010] Where: R is the rainfall in the image, k is the empirical coefficient, which is obtained by calibration based on experimental data and is used to adjust the scale of rainfall calculation, N is the number of raindrops detected in the image, i is the raindrop index, and A is the rainfall index. i is the image area of ​​the i-th raindrop;

[0011] The number and area of ​​raindrops in the image reflect the intensity of rainfall. More and larger raindrops mean greater rainfall. By counting the area of ​​raindrops in the image, the intensity of rainfall can be directly quantified. Since there is a nonlinear relationship between the area of ​​raindrops in the image and the actual rainfall, the empirical coefficient is used to calibrate and adjust this relationship. Through experimental data, a suitable empirical coefficient value can be determined so that the formula can accurately reflect the rainfall.

[0012] The formula for the comprehensive weather index is:

[0013]

[0014] Where: W is the comprehensive weather index, j is the image number index, N j is the total number of pixels in the jth image, k is the pixel index, I j,k is the illumination intensity of the kth pixel in the jth image, R j is the rainfall in the jth image, and M is the number of images in the observation period.

[0015] The formula combines the two key environmental factors, sunlight intensity and rainfall, which together affect weather conditions. It provides a more comprehensive weather assessment by comprehensively calculating the sunlight intensity and rainfall. The average light intensity of all pixels in the jth image is calculated, which reflects the overall brightness of the current image. The higher the light intensity, the better the weather conditions and the higher the quality of visual data. j Used to reflect the negative impact of rainfall on weather conditions. The greater the rainfall, the greater the 1-R j The smaller the value, the worse the weather conditions. In this way, the formula can dynamically adjust the impact of light intensity so that the comprehensive weather index can more accurately reflect the actual weather conditions. What is calculated is the average value of all images, which avoids the accidental error caused by a single image and provides a more stable weather assessment result. Considering images at multiple time points can more comprehensively reflect the trend of weather changes.

[0016] As a further solution of the present invention, in the environment perception module, the obstacle complexity in the point cloud data acquired by the laser radar is the average point density of different segmented areas in the point cloud data. The obstacle complexity formula in the point cloud data is:

[0017]

[0018] Where: C is the complexity of obstacles in the point cloud data, V is the number of segmented regions in the point cloud data, obtained by the point cloud segmentation algorithm, h is the index of the point cloud segmentation region, and D his the point cloud density within the hth segmented region, which is equal to the number of points in the region divided by the volume of the region.

[0019] The number and distribution of obstacles have a direct impact on the complexity of point cloud data. The higher the point cloud density, the more obstacles there are in the area and the more complex the environment. By calculating the point cloud density of different segmented areas, the overall obstacle complexity can be quantified and detailed environmental information can be provided. Dividing the point cloud data into multiple areas can analyze the complexity of the environment in a more fine-grained manner. The point cloud density of each area reflects the distribution of obstacles in the area. The point cloud segmentation algorithm is used to divide the entire point cloud data into several smaller areas, each of which can more accurately analyze the distribution of obstacles. As an important indicator of environmental perception, obstacle complexity can be used to dynamically adjust the weight parameters of the data alignment algorithm to adapt to different environmental conditions and improve the accuracy and robustness of alignment.

[0020] The sensor noise level is the square root of the mean of the camera data point and the square of the distance to the camera data point at each moment in the observation period, and the mean of the lidar data point and the square of the distance to the lidar data point. The sensor noise level formula is:

[0021]

[0022] Where: Q is the sensor noise level, T is the number of data points in the observation period, t is the time index, P camera,t is the camera data point at time t, is the average value of the camera data points, P LiDAR,t is the lidar data point at time t, is the average value of the camera data points.

[0023] The sensor noise level reflects the fluctuations in the data collected by the camera and lidar. The greater the fluctuation, the higher the noise level and the lower the reliability of the data.

[0024] By calculating the deviation between the data point at each moment and its mean, this fluctuation can be quantified. In order to comprehensively evaluate the noise level of the sensor, the formula takes into account the data of the camera and the lidar at the same time, which can reflect the noise level of the overall sensor system. By combining the data of the two sensors, the noise situation in the environment can be more accurately evaluated. By calculating the sum of the squares of the deviations between the data point and its mean, the formula is actually calculating the variance of the data. Variance is an important indicator for measuring data volatility. Taking the arithmetic square root of the variance (i.e., standard deviation) can obtain a noise level indicator consistent with the data unit. By considering data at multiple time points, the changes in the sensor noise level can be more comprehensively reflected.

[0025] As a further solution of the present invention, in the environment perception analysis module, the environment perception coefficient formula uses the ratio of the product of the average light intensity of the image, the comprehensive weather index and the obstacle complexity to the sensor noise level multiplied by a dimensionless conversion coefficient as the environment perception coefficient. The formula of the environment perception coefficient is:

[0026]

[0027] Where: θ is the environmental perception coefficient, μ is the dimensionless conversion coefficient, which is set according to the unit conversion requirements of each variable, and L is the average light intensity of the image.

[0028] Light intensity, obstacle complexity and comprehensive weather index are positive indicators. The higher these factors are, the more complex the environmental conditions are and the more difficult it is to align data. The product of these factors can comprehensively evaluate the complexity of the environment. The sensor noise level is a negative indicator. The higher the noise level is, the lower the reliability of the sensor data is and the greater the impact on data alignment is. Dividing the product of these positive indicators by the noise level can reasonably reflect the comprehensive impact of the environment on data quality. The product of light intensity, obstacle complexity and comprehensive weather index reflects the comprehensive complexity of the environment. Each factor has a multiplication and amplification effect on the environmental perception coefficient, making the evaluation of the comprehensive complexity more sensitive and accurate. By dividing the comprehensive complexity by the sensor noise level, the impact of noise on environmental perception can be eliminated, ensuring that the environmental perception coefficient is more stable and reliable. Different environmental factors have different units and magnitudes. The dimensionless conversion coefficient can unify the impact of these factors within a reasonable magnitude range. Through dimensionless tempering, the formula can be ensured to have a wide range of applicability in different application scenarios, avoiding unreasonable calculations due to unit differences.

[0029] As a further solution of the present invention, the environmental perception coefficient categories preset in the weight parameter adjustment module include low-level environmental perception, intermediate environmental perception, and high-level environmental perception:

[0030] 0<θ≤1, which is low-level environmental perception, and the adjustment strategy formula of the weight parameter is:

[0031] In low-level environmental perception, light intensity L is the main influencing factor, because the environmental conditions are good at this time and the light has a greater impact. The weight parameter formula mainly considers the reverse impact of light intensity. It indicates the ratio of light intensity to all environmental factors, through Make sure to reduce the weight parameter λ when the light intensity is high, so that the sensitivity to the light effect is reduced.

[0032] 1<θ≤10, which is intermediate environmental perception, and the adjustment strategy formula of the weight parameter is:

[0033] In the intermediate environmental perception, weather condition W is the main influencing factor, because the environmental conditions are moderate at this time and the weather factor has a greater impact. Ensure that in the case of bad weather conditions, the weight parameter λ is increased to enhance the sensitivity to weather factors.

[0034] θ>10, which is intermediate environmental perception, the adjustment strategy formula of the weight parameter is:

[0035] In advanced environment perception, obstacle complexity C and sensor noise level Q are the main influencing factors, because the environmental conditions are poor at this time and the impact of obstacles and noise is greater. It represents the ratio of obstacle complexity to all environmental factors. This item is used to increase the sensitivity to obstacle complexity. Represents the inverse effect of the noise level, by which the sensitivity to the noise level is reduced. Ensure that in complex environments, the impact of obstacles and noise is comprehensively considered and the weight parameter λ is adjusted to achieve the best balance.

[0036] As a further solution of the present invention, in the data alignment optimization module, the weighted multi-source alignment formula is:

[0037]

[0038] Where: G is the image obtained by the initial weighted multi-source alignment, R is the rotation matrix, which is the rotation relationship matrix from the camera coordinate system to the lidar coordinate system, obtained through spatial calibration, and P camera,j is the coordinate of the 3D point in the j-th camera image, obtained by camera image analysis, T is the translation vector, is the translation relationship matrix from the camera coordinate system to the lidar coordinate system, obtained by spatial calibration, and P LiDAR,j is the 3D point coordinate in the jth lidar image, B is the number of matching point pairs in the camera image and lidar point cloud, ω j is the weight of the j-th data point, which is set according to the requirements, and ‖·‖ is the symbol for Euclidean distance calculation.

[0039] As a further solution of the present invention, in the data alignment optimization module, the accuracy optimization formula is:

[0040]

[0041] Where: f(R,T) is the image obtained after the initial weighted optimization image is optimized for accuracy, and σ is the scale parameter determined by the training process.

[0042] As a further solution of the present invention, in the data alignment optimization module, the joint multi-source data optimization formula is:

[0043]

[0044] Compared with the prior art, the technical effect of the system proposed in the present invention is as follows: the present invention significantly improves the accuracy and robustness of environmental perception by fusing vision and lidar data. The camera provides rich texture and color information, while the lidar provides high-precision three-dimensional point cloud data. The combination of the two makes up for their respective shortcomings. In terms of data alignment, weighted multi-source data alignment, precision optimization and joint multi-source data optimization formulas are adopted to solve the differences in space and time between camera and lidar data, and achieve precise alignment. The environmental conditions are monitored in real time through the environmental perception module, and the weight parameters can be dynamically adjusted in different environments by using the comprehensive environmental perception coefficient formula and weight parameter adjustment strategy, ensuring the accuracy and robustness of data alignment, significantly improving the environmental perception capability, and ensuring efficient operation under various environmental conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a system block diagram of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the present invention to clearly and completely describe the technical solutions in the present invention. Obviously, the invention content described is only a part of the present invention rather than the whole. Based on the invention content in the present invention, all other technical solutions obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention.

[0047] As Figure 1As shown, the present invention proposes an intelligent data acquisition and processing system based on the fusion of vision and laser radar, including a camera, a laser radar, a data preprocessing module, an environment perception module, an environment perception analysis module, a weight parameter adjustment module, a data alignment optimization module and a data processing analysis module. The environment perception module perceives the average light intensity and the comprehensive weather index in the image, and analyzes the obstacle complexity and the sensor noise level in the point cloud data. The environment perception analysis module uses the data obtained by the environment perception module analysis as input, and inputs it into a preset comprehensive environment perception coefficient formula to obtain the comprehensive environment perception coefficient. The weight parameter adjustment module obtains the adjustment strategy of the weight parameter under the category to which the environment perception coefficient belongs according to the preset environment perception coefficient classification standard, and obtains the weight coefficient. The data alignment optimization module uses the weighted multi-source data alignment formula to realize the preliminary alignment of the data, and uses the precision optimization formula to further optimize the preliminary alignment result. Combining the advantages of the two methods, the joint multi-source data optimization formula is used based on the weight parameter to realize the data fusion of the two data sources. The data processing analysis module extracts useful feature information according to the fused data of the data alignment optimization module, performs object recognition and scene understanding, and provides detailed environment analysis results.

[0048] The present invention significantly improves the accuracy and robustness of environmental perception by fusing vision and lidar data. The camera provides rich texture and color information, while the lidar provides high-precision three-dimensional point cloud data. The combination of the two makes up for their respective shortcomings. In terms of data alignment, weighted multi-source data alignment, accuracy optimization and joint multi-source data optimization formulas are adopted to solve the spatial and temporal differences between camera and lidar data and achieve precise alignment. The environmental conditions are monitored in real time through the environmental perception module, and the weight parameters can be dynamically adjusted in different environments by using the comprehensive environmental perception coefficient formula and weight parameter adjustment strategy, ensuring the accuracy and robustness of data alignment, significantly improving the environmental perception capability, and ensuring efficient operation under various environmental conditions.

[0049] It should be noted that the camera and lidar are connected to the data preprocessing module, the data preprocessing module is connected to the environment module, the environment perception module is connected to the environment perception analysis module, the environment perception analysis module is connected to the weight parameter adjustment module, the weight parameter adjustment module is connected to the data alignment optimization module, and the data alignment optimization module is connected to the data processing and analysis module.

[0050] The system proposed by the present invention works as a process for solving the technical problem, including:

[0051] Step 1, data acquisition and preprocessing: The camera captures the two-dimensional image of the environment, providing texture and color information, and the lidar generates three-dimensional point cloud data of the environment, providing high-precision distance information. The data preprocessing module performs preprocessing such as denoising and contrast enhancement on the camera image, and performs preprocessing such as filtering and noise reduction on the lidar point cloud data;

[0052] Step 2, Environmental Perception: The environmental perception module perceives environmental conditions in real time, including the average light intensity and comprehensive weather index in the image, and analyzes the obstacle complexity and sensor noise level in the point cloud data;

[0053] Step 3, environmental perception analysis: the environmental perception analysis module uses the data obtained by the environmental perception module analysis as input, inputs it into a preset comprehensive environmental perception coefficient formula, and obtains the comprehensive environmental perception coefficient;

[0054] Step 4, weight parameter adjustment: The weight parameter adjustment module determines the category to which the environmental perception coefficient belongs according to the preset environmental perception coefficient classification standard, obtains the corresponding weight parameter adjustment strategy, and dynamically adjusts the weight coefficient;

[0055] Step 5, data alignment optimization: The data alignment optimization module uses the weighted multi-source data alignment formula to achieve preliminary alignment of camera and lidar data, and uses the accuracy optimization formula to further optimize the preliminary alignment results. Combining the advantages of the two methods, based on the weight parameters, the joint multi-source data optimization formula is used to achieve data fusion of the two data sources;

[0056] Step 6, data processing and analysis: The data processing and analysis module extracts useful feature information based on the fused data of the data alignment optimization module, performs object recognition and scene understanding, and provides detailed environmental analysis results.

[0057] It should be noted that in the environmental perception module, the light intensity of each pixel in the camera image is counted, and its average value is obtained as the average light intensity. The comprehensive weather index is obtained by the average light intensity and rainfall of multiple camera images. The rainfall in each image is identified by edge detection and morphological processing, and the area of ​​the raindrops is extracted and calculated using an empirical formula.

[0058] As a further solution of the present invention, in the environment perception module, the empirical formula for rainfall is:

[0059]

[0060] Where: R is the rainfall in the image, k is the empirical coefficient, which is obtained by calibration based on experimental data and is used to adjust the scale of rainfall calculation, N is the number of raindrops detected in the image, i is the raindrop index, and A is the rainfall index. i is the image area of ​​the i-th raindrop;

[0061] Specifically, the number and area of ​​raindrops in the image reflect the intensity of rainfall. More and larger raindrops mean greater rainfall. By counting the area of ​​raindrops in the image, the intensity of rainfall can be directly quantified. Since there is a nonlinear relationship between the area of ​​raindrops in the image and the actual rainfall, the empirical coefficient is used to calibrate and adjust this relationship. Through experimental data, a suitable empirical coefficient value can be determined so that the formula can accurately reflect the rainfall.

[0062] The formula for the comprehensive weather index is:

[0063]

[0064] Where: W is the comprehensive weather index, j is the image number index, N j is the total number of pixels in the jth image, k is the pixel index, I j,k is the illumination intensity of the kth pixel in the jth image, R j is the rainfall in the jth image, and M is the number of images in the observation period.

[0065] Specifically, sunlight intensity and rainfall are two key environmental factors that jointly affect weather conditions. The formula combines the two and provides a more comprehensive weather assessment through the comprehensive calculation of sunlight intensity and rainfall. The average light intensity of all pixels in the jth image is calculated, which reflects the overall brightness of the current image. The higher the light intensity, the better the weather conditions and the higher the quality of visual data. j Used to reflect the negative impact of rainfall on weather conditions. The greater the rainfall, the greater the 1-R j The smaller the value, the worse the weather conditions. In this way, the formula can dynamically adjust the impact of light intensity so that the comprehensive weather index can more accurately reflect the actual weather conditions. What is calculated is the average value of all images, which avoids the accidental error caused by a single image and provides a more stable weather assessment result. Considering images at multiple time points can more comprehensively reflect the weather change trend.

[0066] It should be noted that in the environmental perception module, the obstacle complexity in the point cloud data obtained by the lidar is the average point density of different segmented areas in the point cloud data. The obstacle complexity formula in the point cloud data is:

[0067]

[0068] Where: C is the complexity of obstacles in the point cloud data, V is the number of segmented regions in the point cloud data, obtained by the point cloud segmentation algorithm, h is the index of the point cloud segmentation region, and D his the point cloud density within the hth segmented region, which is equal to the number of points in the region divided by the volume of the region.

[0069] It should be further explained that the number and distribution of obstacles have a direct impact on the complexity of point cloud data. The higher the point cloud density, the more obstacles there are in the area and the more complex the environment. By calculating the point cloud density of different segmented areas, the complexity of the overall obstacles can be quantified and detailed environmental information can be provided. Dividing the point cloud data into multiple areas can analyze the complexity of the environment in a more fine-grained manner. The point cloud density of each area reflects the distribution of obstacles in the area. The point cloud segmentation algorithm is used to divide the entire point cloud data into several smaller areas, each of which can more accurately analyze the distribution of obstacles. As an important indicator of environmental perception, obstacle complexity can be used to dynamically adjust the weight parameters of the data alignment algorithm to adapt to different environmental conditions and improve the accuracy and robustness of alignment.

[0070] The sensor noise level is the square root of the mean of the camera data point and the square of the distance to the camera data point at each moment in the observation period, and the mean of the lidar data point and the square of the distance to the lidar data point. The sensor noise level formula is:

[0071]

[0072] Where: Q is the sensor noise level, T is the number of data points in the observation period, t is the time index, P camera,t is the camera data point at time t, is the average value of the camera data points, P LiDAR,t is the lidar data point at time t, is the average value of the camera data points.

[0073] It should be further explained that the sensor noise level reflects the fluctuation of the camera and lidar when collecting data. The greater the fluctuation, the higher the noise level and the lower the reliability of the data. This fluctuation can be quantified by calculating the deviation between the data point and its mean at each moment. In order to comprehensively evaluate the noise level of the sensor, the formula takes into account the data of the camera and lidar at the same time, which can reflect the noise level of the overall sensor system. By combining the data of the two sensors, the noise in the environment can be more accurately evaluated. By calculating the sum of the squares of the deviations between the data points and their mean, the formula is actually calculating the variance of the data. Variance is an important indicator of data volatility. Taking the arithmetic square root of the variance (i.e., standard deviation) can obtain a noise level indicator consistent with the data unit. By considering data at multiple time points, the changes in the sensor noise level can be more comprehensively reflected.

[0074] It should be noted that in the environmental perception analysis module, the environmental perception coefficient formula uses the ratio of the product of the average light intensity of the image, the comprehensive weather index and the obstacle complexity to the sensor noise level multiplied by a dimensionless conversion coefficient as the environmental perception coefficient. The formula for the environmental perception coefficient is:

[0075]

[0076] Where: θ is the environmental perception coefficient, μ is the dimensionless conversion coefficient, which is set according to the unit conversion requirements of each variable, and L is the average light intensity of the image.

[0077] Specifically, light intensity, obstacle complexity and comprehensive weather index are positive indicators. The higher these factors are, the more complex the environmental conditions are and the more difficult it is to align data. The product of these factors can comprehensively evaluate the complexity of the environment. The sensor noise level is a negative indicator. The higher the noise level is, the lower the reliability of the sensor data is and the greater the impact on data alignment is. Dividing the product of these positive indicators by the noise level can reasonably reflect the comprehensive impact of the environment on data quality. The product of light intensity, obstacle complexity and comprehensive weather index reflects the comprehensive complexity of the environment. Each factor has a multiplication and amplification effect on the environmental perception coefficient, making the evaluation of the comprehensive complexity more sensitive and accurate. By dividing the comprehensive complexity by the sensor noise level, the impact of noise on environmental perception can be eliminated, ensuring that the environmental perception coefficient is more stable and reliable. Different environmental factors have different units and magnitudes. The dimensionless conversion coefficient can unify the impact of these factors within a reasonable magnitude range. Through dimensionless tempering, the formula can be ensured to have a wide range of applicability in different application scenarios, avoiding unreasonable calculations due to unit differences.

[0078] It should be noted that the preset environmental perception coefficient categories in the weight parameter adjustment module include low-level environmental perception, intermediate environmental perception, and high-level environmental perception:

[0079] 0<θ≤1, which is low-level environmental perception, and the adjustment strategy formula of the weight parameter is:

[0080] It should be further explained that, in low-level environmental perception, light intensity L is the main influencing factor, because the environmental conditions are good at this time and the light has a greater impact. The weight parameter formula mainly considers the reverse impact of light intensity. It indicates the ratio of light intensity to all environmental factors, through Make sure to reduce the weight parameter λ when the light intensity is high, so that the sensitivity to the light effect is reduced.

[0081] 1<θ≤10, which is intermediate environmental perception, and the adjustment strategy formula of the weight parameter is:

[0082] It should be further explained that under the intermediate environmental perception, weather condition W is the main influencing factor, because the environmental conditions are moderate at this time and the weather factor has a greater impact. Ensure that in the case of bad weather conditions, the weight parameter λ is increased to enhance the sensitivity to weather factors.

[0083] θ>10, which is intermediate environmental perception, the adjustment strategy formula of the weight parameter is:

[0084] It should be further explained that in advanced environmental perception, obstacle complexity C and sensor noise level Q are the main influencing factors, because the environmental conditions are poor at this time and the impact of obstacles and noise is greater. It represents the ratio of obstacle complexity to all environmental factors. This item is used to increase the sensitivity to obstacle complexity. Represents the inverse effect of the noise level, by which the sensitivity to the noise level is reduced. Ensure that in complex environments, the impact of obstacles and noise is comprehensively considered and the weight parameter λ is adjusted to achieve the best balance.

[0085] It should be noted that in the data alignment optimization module, the weighted multi-source alignment formula is:

[0086]

[0087] Where: G is the image obtained by the initial weighted multi-source alignment, R is the rotation matrix, which is the rotation relationship matrix from the camera coordinate system to the lidar coordinate system, obtained through spatial calibration, and P camera,j is the coordinate of the 3D point in the j-th camera image, obtained by camera image analysis, T is the translation vector, is the translation relationship matrix from the camera coordinate system to the lidar coordinate system, obtained by spatial calibration, and P LiDAR,j is the 3D point coordinate in the jth lidar image, B is the number of matching point pairs in the camera image and lidar point cloud, ω j is the weight of the j-th data point, which is set according to the requirements, and ‖·‖ is the symbol for Euclidean distance calculation.

[0088] This weighted multi-source data alignment formula ensures accurate alignment of visual and lidar data by minimizing the weighted Euclidean distance squared between camera points and lidar points under the action of rotation matrices and translation vectors. The rotation matrix and translation vector are obtained through spatial calibration and accurately map the camera coordinate system to the lidar coordinate system. The weights in the formula allow the contribution of each data point to be adjusted, so that the alignment process can be optimized according to specific needs and the importance of the data point. While ensuring high-precision alignment, it enhances the robustness and flexibility of the system under different environmental and data quality conditions, effectively solving the spatial and temporal differences between visual and lidar data, thereby significantly improving the overall performance and reliability of environmental perception.

[0089] It should be noted that in the data alignment optimization module, the accuracy optimization formula is:

[0090]

[0091] Where: f(R,T) is the image obtained after the initial weighted optimization image is optimized for accuracy, and σ is the scale parameter determined by the training process.

[0092] Using the precision optimization formula, the alignment effect of the camera and lidar data is further refined by applying the exponentially weighted Euclidean distance square to the image after the initial weighted optimization. In the formula, the rotation matrix and translation vector are adjusted so that the distance error between each point pair is emphasized in the form of an exponential function, where the scale parameter controls the sensitivity of the error, which is determined through the training process and can more effectively reduce the negative impact of large error points on the overall alignment results, while strengthening the contribution of small error point pairs, thereby improving the accuracy and robustness of data alignment, ensuring that in the multi-source data fusion process, the visual data and lidar data can achieve higher alignment accuracy, adapt to various complex environments, and improve the overall performance and reliability of the environmental perception system.

[0093] It should be noted that in the data alignment optimization module, the optimization formula for combining multi-source data is:

[0094]

[0095] The contributions of both methods are combined to achieve the best effect of data alignment. The first term in the formula uses the weighted Euclidean distance squared to ensure the initial accurate spatial alignment of the camera and lidar data; the second term uses the exponentially weighted Euclidean distance squared to further optimize the alignment accuracy by reducing the impact of large error point pairs. By adjusting the weight parameters, the relative importance of the two methods can be flexibly controlled under different environmental conditions and data quality, thereby improving the robustness and adaptability of the system, and being able to achieve higher-precision visual and lidar data fusion in various complex environments, significantly improving the overall performance and reliability of the environmental perception system.

[0096] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0097] Finally: The above description is only the technical solution of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An intelligent data acquisition and processing system based on the fusion of vision and laser radar, characterized in that: It includes a camera, a lidar, a data preprocessing module, an environmental perception module, an environmental perception analysis module, a weight parameter adjustment module, a data alignment optimization module and a data processing and analysis module. The environmental perception module perceives the average light intensity and the comprehensive weather index in the image, and analyzes the obstacle complexity and sensor noise level in the point cloud data. The environmental perception analysis module takes the data obtained by the environmental perception module as input, and inputs it into a preset comprehensive environmental perception coefficient formula to obtain the comprehensive environmental perception coefficient. The weight parameter adjustment module obtains the adjustment strategy of the weight parameter under the category to which the environmental perception coefficient belongs according to the preset environmental perception coefficient classification standard, and obtains the weight coefficient. The data alignment optimization module uses the weighted multi-source data alignment formula to achieve preliminary alignment of the data, and uses the precision optimization formula to further optimize the preliminary alignment results. Combining the advantages of the two methods, the joint multi-source data optimization formula is used based on the weight parameters to achieve data fusion of the two data sources. The data processing and analysis module extracts useful feature information based on the fused data of the data alignment optimization module, performs object recognition and scene understanding, and provides detailed environmental analysis results.

2. The intelligent data acquisition and processing system based on vision and laser radar fusion according to claim 1 is characterized in that: The camera and lidar are connected to the data preprocessing module, the data preprocessing module is connected to the environment module, the environment perception module is connected to the environment perception analysis module, the environment perception analysis module is connected to the weight parameter adjustment module, the weight parameter adjustment module is connected to the data alignment optimization module, and the data alignment optimization module is connected to the data processing and analysis module.

3. The intelligent data acquisition and processing system based on vision and laser radar fusion according to claim 2 is characterized in that: In the environmental perception module, the light intensity of each pixel in the camera image is counted and its average value is obtained as the average light intensity. The comprehensive weather index is obtained by the average light intensity and rainfall of multiple camera images. The rainfall in each image is identified by edge detection and morphological processing, and the area of ​​the raindrops is extracted and calculated using an empirical formula.

4. The intelligent data acquisition and processing system based on vision and laser radar fusion according to claim 3 is characterized in that: In the environmental perception module, the empirical formula for rainfall is: Where: R is the rainfall in the image, k is the empirical coefficient obtained by calibration based on experimental data, N is the number of raindrops detected in the image, i is the raindrop index, and A i is the image area of ​​the i-th raindrop; The formula for the comprehensive weather index is: Where: W is the comprehensive weather index, j is the image number index, N j is the total number of pixels in the jth image, k is the pixel index, I j,k is the illumination intensity of the kth pixel in the jth image, R j is the rainfall in the jth image, and M is the number of images in the observation period.

5. The intelligent data acquisition and processing system based on vision and laser radar fusion according to claim 4 is characterized in that: In the environment perception module, the obstacle complexity in the point cloud data obtained by the lidar is the average point density of different segmented areas in the point cloud data. The obstacle complexity formula in the point cloud data is: Where: C is the complexity of obstacles in the point cloud data, V is the number of segmented regions in the point cloud data, obtained by the point cloud segmentation algorithm, h is the index of the point cloud segmentation region, and D h is the point cloud density within the hth segmented region, which is equal to the number of points in the region divided by the volume of the region. The sensor noise level is the square root of the mean of the camera data point and the square of the distance to the camera data point at each moment in the observation period, and the mean of the lidar data point and the square of the distance to the lidar data point. The sensor noise level formula is: Where: Q is the sensor noise level, T is the number of data points in the observation period, t is the time index, P camera,t is the camera data point at time t, is the average value of the camera data points, P LiDAR,t is the lidar data point at time t, is the average value of the camera data points.

6. The intelligent data acquisition and processing system based on vision and laser radar fusion according to claim 5 is characterized in that: In the environmental perception analysis module, the environmental perception coefficient formula uses the ratio of the product of the average light intensity of the image, the comprehensive weather index and the obstacle complexity to the sensor noise level multiplied by a dimensionless conversion coefficient as the environmental perception coefficient. The formula for the environmental perception coefficient is: Where: θ is the environmental perception coefficient, μ is the dimensionless conversion coefficient, which is set according to the unit conversion requirements of each variable, and L is the average light intensity of the image.

7. The intelligent data acquisition and processing system based on vision and laser radar fusion according to claim 6 is characterized in that: The preset environmental perception coefficient categories in the weight parameter adjustment module include low-level environmental perception, intermediate environmental perception, and high-level environmental perception: 0 < θ ≤ 1 is for low-level environmental perception, and the adjustment strategy formula for the weight parameter is 1 < θ ≤ 10 represents intermediate environmental perception, and the adjustment strategy formula for the weight parameter is θ>10, which is intermediate environmental perception, the adjustment strategy formula of the weight parameter is:

8. The intelligent data acquisition and processing system based on vision and laser radar fusion according to claim 7 is characterized in that: In the data alignment optimization module, the weighted multi-source alignment formula is: Where: G is the image obtained by the initial weighted multi-source alignment, R is the rotation matrix, which is the rotation relationship matrix from the camera coordinate system to the lidar coordinate system, obtained through spatial calibration, and P camera,j is the coordinate of the 3D point in the j-th camera image, obtained by camera image analysis, T is the translation vector, is the translation relationship matrix from the camera coordinate system to the lidar coordinate system, obtained by spatial calibration, and P LiDAR,j is the 3D point coordinate in the jth lidar image, B is the number of matching point pairs in the camera image and lidar point cloud, ω j is the weight of the j-th data point, which is set according to the requirements, and ‖·‖ is the symbol for Euclidean distance calculation.

9. The intelligent data acquisition and processing system based on vision and laser radar fusion according to claim 8 is characterized in that: In the data alignment optimization module, the accuracy optimization formula is: Where: f(R,T) is the image obtained after the initial weighted optimization image is optimized for accuracy, and σ is the scale parameter determined by the training process.

10. The intelligent data acquisition and processing system based on vision and laser radar fusion according to claim 9 is characterized in that: In the data alignment optimization module, the optimization formula for joint multi-source data is:

Citation Information

Cited By

  • Multi-source heterogeneous data fusion sky-ground intelligent monitoring system

    CN119691657A

  • Obstacle recognition device based on laser point cloud data

    CN120352877A

  • An obstacle recognition device based on laser point cloud data

    CN120352877B

  • Composite radar photoelectric detection system

    CN120522690A

  • Unmanned aerial vehicle autonomous obstacle avoidance system and method based on millimeter wave radar and multi-mode vision fusion

    CN120653016A