High-precision multi-mode sensor space alignment method in road monitoring

Abstract

The invention provides a high-precision multi-modal sensor space alignment method in road monitoring, which comprises the following steps of: dynamically correcting and optimizing a synchronization relationship among different sensors by establishing a behavior model of the sensors, and making up the limitation caused by the dependence on static hypothesis in a traditional method. The influence of different environmental factors on the sensor performance is considered in the establishment of the sensor behavior model, and dynamic correction is realized through a simulation optimization technology, so that high-precision synchronization and alignment of sensor data are ensured. Besides, by introducing the super-resolution point cloud reconstruction technology, the resolution of radar point cloud data is effectively improved, the registration precision between the point cloud and the visual image is enhanced, the point cloud and the visual image can be better fused, and the space alignment precision is further improved. According to the method, high-precision multi-modal sensor data space alignment can be realized in a complex environment, the sensing precision and robustness of a road monitoring system are remarkably improved, and the defects in the field in the prior art are overcome.

Description

Technical Field

[0001] The present invention belongs to the field of road monitoring, and particularly relates to a high-precision multi-modal sensor spatial alignment method in road monitoring. Background Art

[0002] In modern road monitoring systems, multi-modal sensors (such as infrared sensors, camera arrays, millimeter-wave radars, etc.) are widely used in fields such as environmental perception, traffic monitoring, and obstacle detection. Since these sensors have different working principles and perception methods, they each have different advantages and disadvantages when capturing surrounding environmental information. Infrared sensors can usually effectively perceive temperature changes under low-light or nighttime conditions and are suitable for monitoring low-temperature objects such as pedestrians or animals; camera arrays provide high-resolution visual images that can obtain rich detail information and are helpful for environmental modeling and recognition; radars (such as millimeter-wave radars) have the ability to penetrate adverse weather conditions such as rain and snow, can maintain stable detection capabilities in harsh environments, and provide distance and speed information of objects.

[0003] However, in practical applications, the data fusion problem of these sensors has become one of the main challenges in road monitoring systems. Due to differences in the working principles, field of view angles, resolutions, sampling frequencies, etc. of multiple sensors, the data they collect often shows inconsistencies in time and space. This inconsistency is manifested as asynchronous data timings, misaligned spatial positions, etc., resulting in the inability to directly fuse these data for unified analysis and processing. Existing multi-modal sensor data alignment methods usually rely on rough timestamp synchronization and simplified geometric methods for spatial alignment. For example, some methods adjust the alignment of different sensor data through geometric transformation by presetting the spatial positions and viewing angles of the sensors in advance. However, these methods have several significant drawbacks: First, traditional geometric alignment methods usually assume that the relative positions and postures of the sensors are fixed, which often does not hold in practical applications, especially in dynamic environments where the positioning of the sensors may change, leading to the accumulation of errors; Second, these methods ignore the changing characteristics of the sensors under different environmental conditions, such as weather changes and sensor aging, which makes the traditional methods less adaptable in complex environments and unable to guarantee the alignment accuracy; Finally, although some methods have tried to improve spatial alignment through the fusion of visual and radar point cloud data, due to the resolution differences and data noise between the two, the effect is still limited. Overall, existing spatial alignment methods lack a systematic and high-precision alignment solution that can adapt to dynamic environmental changes when dealing with multi-modal sensor data. Summary of the Invention

[0004] The objective of the present invention is to propose a high-precision multi-modal sensor spatial alignment method in road monitoring, which combines dynamic modeling of sensor characteristics, optimization algorithms, and data super-resolution technology, aiming to solve problems such as insufficient accuracy and poor environmental adaptability encountered in the process of multi-modal sensor data alignment in the prior art.

[0005] To achieve the above objective, the present invention provides a high-precision multi-modal sensor spatial alignment method in road monitoring, and the method includes:

[0006] S1: Obtain multi-modal sensor data, determine the spatial calibration of the sensors in the world coordinate system according to the multi-modal sensor data, and perform standardized processing of unified coordinate system conversion based on spatial alignment as a benchmark, and output the point cloud data after standardized processing.

[0007] S2: Based on the point cloud data after standardized processing in step S1, construct a sensor dynamic feature model for the error propagation process of multi-modal sensor data, perform error compensation on multi-modal sensor data through regression analysis of historical data of multi-modal sensors, and dynamically correct the error compensation according to the changes in real-time monitoring of multi-modal sensors to improve the accuracy of spatial alignment, and obtain the aligned point cloud data after error correction.

[0008] S3: Obtain the aligned point cloud data generated in step S2, sample the aligned point cloud data to generate low-resolution point clouds, construct a super-resolution point cloud enhancement network to enhance the low-resolution point clouds, and generate enhanced super-resolution point clouds; wherein, the super-resolution point cloud enhancement network includes two modules: a feature extraction module and an adaptive residual weighting module, the feature extraction module uses a convolutional neural network to extract spatial and geometric features from low-resolution points, and the adaptive residual weighting module adaptively adjusts the weighting coefficient of each point when combining point cloud enhancement, so as to more finely restore the details of the point cloud.

[0009] S4: Obtain the enhanced super-resolution point clouds generated in step S3, and then obtain the original point cloud data from the multi-modal sensors, and perform weighted fusion of the super-resolution point clouds and the original point cloud data centered on each point to obtain the weighted fusion result of each point.

[0010] S5: According to the weighted fusion result of each point obtained in S4, adjust the processing strategy of the point cloud data in combination with environmental changes to optimize the fusion result, and obtain the adjusted point cloud data.

[0011] Preferably, the multi-modal sensor data includes radar data and image data; the multi-modal sensors include cameras, radars, and infrared sensors.

[0012] In step S1, it also includes preprocessing the multi-modal sensor data, including:

[0013] For radar data, a denoising algorithm based on local reconstruction is adopted. By reconstructing the local neighborhood of each point, abnormal points are removed;

[0014] For image data, a denoising method based on a deep convolutional autoencoder is adopted. The deep convolutional autoencoder denoises by learning low-frequency features and is achieved by minimizing the noise loss function of the image.

[0015] Preferably, in step S1, based on an external high-precision positioning system, the spatial calibration of the multi-modal sensor in the world coordinate system is determined and standardized, specifically including:

[0016] For the external parameter calibration between sensors: The binocular camera and radar calibration method is adopted. By registering the spatial coordinate systems of the camera and the radar, the rotation matrix and translation vector between the camera and the radar are determined;

[0017] At the same time, combining real-time sensor data and high-precision positioning data, the external parameters of the sensors are dynamically adjusted by minimizing the loss function to correct the spatial alignment relationship between the sensors; The loss function is minimized by calculating the error between the modeled position of the sensor at the current time point and the actual position of the sensor at the current time point;

[0018] Finally, through the calculated rotation matrix and translation vector, the radar, camera, and infrared data are converted into the same coordinate system.

[0019] Preferably, in step S2, the sensor dynamic feature model describes the error propagation process of the sensor based on the kinematic equation of the sensor, external environmental conditions, and the relative motion relationship of the target object;

[0020] In step S2, through an error compensation model based on statistical regression, the error of the sensor is quantified and compensated. The error compensation model based on statistical regression fits an error compensation function by performing regression analysis on historical data and combining the motion characteristics of the sensor and environmental changes to correct the errors of different sensors;

[0021] In step S2, an error correction scheme based on an incremental learning algorithm is used to update the error compensation model based on statistical regression in real time to adapt to the dynamic changes of the sensor; Among them, the incremental learning algorithm is an online learning algorithm.

[0022] Preferably, in step S2, a sensor error model fusion algorithm is used to obtain a global error compensation model by weighted fusion of the error models of different sensors, and then it is applied to the alignment process of all sensor data.

[0023] Preferably, in step S3, for each point, the high-resolution point cloud is generated using a super-resolution point cloud enhancement network and is generated by the following formula:

[0024]

[0025] where, is the enhanced high-resolution point, P LR (i) is the low-resolution point, ω i is the adaptive weighting coefficient of the i-th point, is the residual term, which is used to recover the details of the low-resolution point cloud;

[0026] where, in the adaptive residual weighting module, the weighting coefficient of each point is automatically optimized by training the network, so that the network can focus on important regions and improve the enhancement effect; the residual term captures the details in the point cloud data through convolution operations and residual learning, and calculates the relative enhancement value of each point;

[0027] The loss function of the super-resolution point cloud enhancement network is expressed as:

[0028]

[0029] where, is the i-th high-resolution point predicted by the network, P HR (i) is the true high-resolution point, is the residual term of the i-th point, λ is the residual term regularization coefficient, which is used to control the balance of the loss function, and n is the total number of points.

[0030] Preferably, in step S3, a block inference strategy is adopted to meet the real-time processing requirements of large-scale point cloud data, and the point cloud data is divided into several blocks for parallel enhancement processing.

[0031] Preferably, in step S4, a spatial alignment strategy based on a transformation matrix is adopted to align the differences in spatial coordinates and resolutions of data collected by different sensors, specifically including:

[0032] Given an original point cloud data, a transformation matrix is obtained through geometric calibration and external parameter estimation, and then the point cloud is mapped to a unified target coordinate system using this transformation matrix;

[0033] The super-resolution point cloud and the original point cloud data are weighted and fused with each point as the center to obtain the weighted fusion result of each point, specifically including:

[0034] Introduce an adaptive weighting factor for the point cloud fusion process based on the performance and importance of each point in different modalities. According to the adaptive weighting factor, after spatial alignment, calculate the weighted fusion result of each point;

[0035] In step S4, it also includes,

[0036] Design an adaptive weighting learning mechanism based on a deep neural network. By learning the context information and feature similarity of each point, dynamically adjust the adaptive weighting factor of each point to capture the local and global feature differences of different sensor data;

[0037] At the same time, after the fusion process, use a global optimization strategy to improve the accuracy of the point cloud data and reduce errors; the optimization goal of the global optimization strategy is to minimize the geometric error between the fused point cloud obtained from the weighted fusion result of each point and the real point cloud, and maintain the smoothness of the point cloud;

[0038] The optimization objective function of the global optimization strategy is specifically:

[0039] The first term is the L2 distance between the fused point cloud and the real point cloud, which is used to measure the geometric error;

[0040] The second term is the L2 norm of the point cloud gradient, which ensures the smoothness and consistency of the point cloud through regularization, and avoids excessive noise or unnatural geometric forms introduced due to data fusion.

[0041] Preferably, in step S5, adopt an adaptive adjustment mechanism based on environmental feedback, and use environmental changes to adjust the processing strategy of the point cloud data, thereby optimizing the fusion result; among them, the environmental changes are quantified using an environmental perception function, and the environmental perception function is calculated based on the influence of environmental light intensity or weather conditions and the performance score of the sensor;

[0042] At the same time, the adaptive adjustment mechanism based on environmental feedback will adjust the fusion parameters of the point cloud data according to the environmental state score, specifically including:

[0043] Define an adjustment factor, and the adjustment factor dynamically adjusts the weight value or spatial transformation parameter of each point according to the change of the environmental state. Through the adjustment factor, smoothly adjust the weight of each point during fusion, so that the processing result of the point cloud data is more robust in a harsh environment, and can provide a more refined point cloud in a good environment.

[0044] Preferably, in step S5, after obtaining the environmental feedback, update the weighted fusion strategy of each point according to the adjustment factor; among them, the weight of each point during the fusion process will be updated according to the environmental feedback;

[0045] According to the weighted fusion strategy for each point after update, noise and errors caused by environmental changes are reduced through a global optimization strategy. The optimization objective of the global optimization strategy is to minimize the point cloud error caused by environmental changes, that is, to minimize the error between the weighted fusion combination of each point and the true high-resolution point cloud data, and to maintain the smoothness and consistency of the point cloud.

[0046] The beneficial technical effects of the present invention are at least as follows:

[0047] (1) By establishing dynamic behavior models of different sensors, the response characteristics of sensors and the influence of environmental factors on their performance are considered. According to the working characteristics of different sensors (such as the reflection characteristics of radar, the exposure of cameras, etc.) and external environmental factors (such as temperature, humidity, light, etc.), the present invention constructs a sensor error model and combines simulation optimization techniques (such as particle swarm optimization, genetic algorithm, etc.) to achieve high-precision synchronization and spatial alignment of sensor data. This innovation effectively solves the problems of traditional methods relying on static assumptions and lacking the ability to adapt to environmental dynamics.

[0048] (2) The present invention introduces super-resolution point cloud technology to enhance radar point cloud data based on a deep learning model, significantly improving the spatial resolution of the point cloud data. Combining visual information, a multi-modal deep learning model is used to finely register the point cloud and image data, ensuring high-precision alignment of the two in the same coordinate system. Through this technology, the problem of insufficient alignment accuracy caused by the resolution difference between radar point cloud and image data in traditional methods is compensated.

[0049] (3) The present invention designs a real-time feedback mechanism to automatically adjust the working mode and alignment strategy of sensors according to environmental changes (such as weather, light, etc.). This mechanism can maintain high-precision alignment of sensor data in a dynamic environment, ensuring the stable operation of the system in complex and uncertain environments.

[0050] (4) Through innovative technologies such as sensor behavior modeling and simulation optimization, super-resolution point cloud reconstruction and visual fusion, the present invention proposes a solution for high-precision, multi-modal sensor data spatial alignment, effectively overcoming multiple deficiencies in the prior art and greatly improving the perception ability and robustness of the road monitoring system in a dynamic environment. Description of the Drawings

[0051] The present invention is further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation to the present invention. For those of ordinary skill in the art, other drawings can be obtained according to the following drawings without creative efforts.

[0052] Figure 1This is a flowchart of a high-precision multi-modal sensor spatial alignment method in road monitoring of the present invention. Detailed implementation manners

[0053] The embodiments of the present invention will be described in detail below. The examples of the embodiments are shown in the drawings, in which the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are only used to explain the present invention and should not be construed as a limitation to the present invention.

[0054] In one or more embodiments, as Figure 1 shown, a high-precision multi-modal sensor spatial alignment method in road monitoring is disclosed, including:

[0055] S1: Obtain multi-modal sensor data, determine the spatial calibration of the sensor in the world coordinate system according to the multi-modal sensor data, and perform standardized processing of unified coordinate system conversion based on spatial alignment as a benchmark, and output the point cloud data after standardized processing.

[0056] S2: Based on the point cloud data after standardized processing in step S1, construct a sensor dynamic feature model for the error propagation process of multi-modal sensor data, perform error compensation on multi-modal sensor data through regression analysis of historical data of multi-modal sensors, and dynamically correct the error compensation according to the changes of real-time monitoring of multi-modal sensors to improve the accuracy of spatial alignment, and obtain the aligned point cloud data after error correction.

[0057] S3: Obtain the aligned point cloud data generated in step S2, sample the aligned point cloud data to generate a low-resolution point cloud, construct a super-resolution point cloud enhancement network to enhance the low-resolution point cloud, and generate an enhanced super-resolution point cloud; wherein, the super-resolution point cloud enhancement network includes two modules: a feature extraction module and an adaptive residual weighting module. The feature extraction module uses a convolutional neural network to extract spatial and geometric features from the low-resolution points, and the adaptive residual weighting module adaptively adjusts the weighting coefficient of each point when combining point cloud enhancement, so as to more finely restore the details of the point cloud.

[0058] S4: Obtain the enhanced super-resolution point cloud generated in step S3, and then obtain the original point cloud data from the multi-modal sensor, and perform weighted fusion on the super-resolution point cloud and the original point cloud data centered on each point to obtain the weighted fusion result of each point.

[0059] S5: According to the weighted fusion result of each point obtained in S4, combine the environmental changes to adjust the processing strategy of the point cloud data to optimize the fusion result, and obtain the adjusted point cloud data.

[0060] The present invention proposes a high-precision multi-modal sensor spatial alignment method based on sensor behavior modeling and simulation optimization, as well as super-resolution point cloud and visual information fusion. This method combines the dynamic modeling of sensor characteristics, optimization algorithms, and data super-resolution technology, aiming to solve the problems of insufficient accuracy and poor environmental adaptability encountered in the process of multi-modal sensor data alignment in the prior art. Specifically, the present invention establishes a behavior model of the sensor, dynamically corrects and optimizes the synchronization relationship between different sensors, and makes up for the limitations brought by the traditional method relying on static assumptions. The establishment of the sensor behavior model takes into account the influence of different environmental factors on the sensor performance (such as temperature, humidity, light, weather changes, etc.), and realizes dynamic correction through simulation optimization technology to ensure the high-precision synchronization and alignment of sensor data. In addition, by introducing the super-resolution point cloud reconstruction technology, the present invention effectively improves the resolution of radar point cloud data, enhances the registration accuracy between the point cloud and the visual image, enables the two to be better fused, and further improves the accuracy of spatial alignment. Combining these innovative technologies, the present invention can achieve high-precision spatial alignment of multi-modal sensor data in complex environments, significantly improve the perception accuracy and robustness of the road monitoring system, and make up for the deficiencies of the prior art in this field.

[0061] The following further describes specific embodiments of the present invention.

[0062] Step 1: Preprocessing and spatial calibration of multi-modal sensor data:

[0063] Due to the errors of data from different sensors in space (such as coordinate system differences, geometric distortions, etc.). The calibration and spatial alignment of each sensor need to be standardized to ensure the accuracy of fusion.

[0064] Solution:

[0065] Since the data obtained by multi-modal sensors (such as radar, camera, infrared, etc.) are often affected by environmental noise, the working errors of the sensors themselves, and external interferences. In order to ensure that the data can accurately participate in spatial alignment, it is necessary to preprocess and denoise the original data.

[0066] Preferably, for radar data (point cloud), a denoising algorithm based on local reconstruction (LocalReconstructionDenoising, LRD) is adopted. LRD removes abnormal points by reconstructing the local neighborhood of each point. Its basic process is as follows:

[0067]

[0068] where P denoised : The denoised point cloud data. P raw : The original point cloud data. The noisy part obtained after local reconstruction of the original point cloud data.

[0069] For camera images, a denoising method based on a deep convolutional autoencoder (Denoising Convolutional Autoencoder, DCAE) is used. The autoencoder denoises by learning low-frequency features. Its model can be achieved by minimizing the noise loss function of the image:

[0070]

[0071] where I i : The original image data of the i-th pixel. The denoised image pixel output after passing through the denoising network. n: The number of pixels in the image.

[0072] Preferably, in order to unify the data of different sensors into the same coordinate system, the present invention first needs to determine the spatial relationship between the sensors through spatial calibration. Usually, the calibration process is based on an external high-precision positioning system (such as GNSS or IMU) to determine the position and orientation of the sensors in the world coordinate system.

[0073] Using a binocular camera and radar calibration method, by registering the spatial coordinate systems of the camera and the radar, the rotation matrix R and the translation vector t between them are determined. This process can be achieved through reference points in the known world coordinate system. For example, through the reflection marks on the calibration board or environmental calibration objects at known positions. Calibration equation:

[0074] P camera = R·P lidar + t

[0075] where P camera : The point cloud data of the camera. P lidar : The point cloud data of the radar. R: The rotation matrix, representing the rotation relationship between the camera coordinate system and the radar coordinate system. t: The translation vector, representing the translation relationship between the camera coordinate system and the radar coordinate system.

[0076] Preferably, with changes in the environment (such as temperature, humidity, and small deformations of the mechanical structure, etc.), the position and orientation of the sensors may change slightly. Therefore, a dynamic calibration adjustment algorithm is introduced in this step. This algorithm combines real-time sensor data and high-precision positioning data to dynamically adjust the external parameters of the sensors and correct the spatial alignment relationship between the sensors. The dynamic adjustment of the calibration parameters is achieved by minimizing the following loss function:

[0077]

[0078] where P camera,i : The three-dimensional point cloud of the i-th camera image. The radar point cloud adjusted by the calibration parameters. m: the number of registered points. This optimization process corrects the rotation matrix R and the translation vector t through backpropagation of the gradient descent method to achieve dynamic calibration update.

[0079] Preferably, the unified coordinate system conversion of the data is the key to spatial alignment. By unifying the data of all sensors into the global coordinate system, the misalignment problem of different sensor data in space is solved.

[0080] Through the calculated rotation matrix R and translation vector t, the radar, camera, and infrared data are converted into the same coordinate system. For the data P of any sensor i , its result of conversion to the unified coordinate system can be expressed by the following formula:

[0081] P global = R·P i + t

[0082] where, P global : the data in the unified coordinate system. P i : the data of a certain sensor (radar, camera, etc.) in its original coordinate system.

[0083] Through these steps, the preprocessing and spatial calibration of multi-modal sensor data are completed, and they are unified into a common coordinate system. These processed data will become the basis for subsequent steps (such as spatial alignment and fusion), ensuring that the data of different sensors can be accurately compared and analyzed in the same reference system.

[0084] Step 2: Sensor behavior modeling and error modeling:

[0085] In a multi-modal sensor system, especially in road monitoring applications, due to the large differences in the physical principles, working environments, and error sources of various sensors (such as radar, infrared, camera arrays), if the behavior and errors of the sensors are not accurately modeled, it will inevitably have a negative impact on the final spatial alignment result. Therefore, the sensor behavior modeling and error modeling become the core components in the entire spatial alignment process. The goal of this step is to accurately model the behavior characteristics of each sensor in a dynamic environment, quantify various errors, and further perform compensation. Through the dynamic correction of sensor errors, the present invention can improve the spatial alignment accuracy of the multi-sensor system in a complex road environment.

[0086] Solution:

[0087] Preferably, the behavior of the sensor in road monitoring is affected not only by the characteristics of the device hardware (such as the accuracy and resolution of the sensor), but also by the relative motion between the sensor and the target, changes in the external environment (such as temperature, humidity, light, etc.), and factors such as the aging of the sensor itself. Therefore, the present invention models the dynamic behavior of the sensor to describe its performance in a complex environment.

[0088] To achieve dynamic behavior modeling, the present invention proposes a Sensor Dynamic Feature Model (SDFM), which describes the error propagation process of the sensor based on the kinematic equation of the sensor, external environmental conditions, and the relative motion relationship of the target object. For each sensor i, the present invention establishes its dynamic behavior model through the following formula:

[0089]

[0090] Where, At time t, the position modeling result of sensor i. P i (t): The original measurement position of sensor i at time t. Represents the error function of sensor i, which specifically depends on the velocity V i (t), the attitude change angle θ i of the sensor, the influence factor C i of the external environment (such as temperature, humidity, etc.) and time t.

[0091] Preferably, since each sensor has an inherent measurement error, and these errors are affected not only by the type of sensor, but also by factors such as the installation position of the sensor and the use environment, it is necessary to establish an accurate error model for compensation.

[0092] The present invention quantifies and compensates for the error of the sensor through a Statistical Regression Error Compensation Model (SRECM). This model fits an error compensation function through regression analysis of historical data, combined with the motion characteristics of the sensor and environmental changes, to correct the errors of different sensors.

[0093] The key to the error model is to design a regularization term according to the error sources of the sensor (geometric error, time synchronization error, environmental error, etc.) i to enhance the stability of error compensation. Assuming that the error of sensor i can be represented by E

[0094]

[0095] Among them, The modeled position of sensor i at the j-th time point. The true position of sensor i at the j-th time point. n: The total number of points in the point cloud data. The regularization term, which measures the stability of the error compensation result and can be designed as a weighted difference term, for example:

[0096]

[0097] Here, β is a hyperparameter that controls the error penalty strength, and The difference term of may show a non-linear relationship in practical applications due to the different dynamic behaviors of sensors and environmental factors. Through this regularization term, the overfitting phenomenon in the error compensation process can be effectively avoided, and the adaptability of the error model in a complex environment can be ensured.

[0098] Preferably, since the errors of sensors are not static, especially in a road monitoring system, the accuracy and errors of sensors often fluctuate over time and with changes in environmental conditions. Therefore, the present invention introduces a dynamic error correction mechanism, which dynamically adjusts and corrects errors by real-time monitoring the performance of sensors and combining an online learning algorithm.

[0099] For this reason, the present invention proposes an error correction scheme based on an incremental learning algorithm. This scheme adapts to the dynamic changes of sensors by real-time updating the error model, and the specific update formula is as follows:

[0100]

[0101] Among them, The updated error model. The old error model. α: The learning rate, which determines the step size of error correction. This incremental learning mechanism can not only handle the situation where errors change over time, but also dynamically adjust error compensation when new data arrives, ensuring that the spatial alignment between sensors is always in an optimal state.

[0102] Preferably, in order to further improve the spatial alignment accuracy of the multi-sensor system, the present invention introduces a sensor error model fusion algorithm. This algorithm obtains a global error compensation model by weighted fusion of the error models of different sensors, and then applies it to the alignment process of all sensor data.

[0103] When fusing models, the present invention defines a weighting factor w i , which represents the contribution degree of the error models of different sensors to the final alignment result, and the specific formula is as follows:

[0104]

[0105] Among them, P align : The spatial alignment result of multi-sensor data. w i : The weighting factor of the error model of sensor i, which can be dynamically adjusted according to factors such as the accuracy and reliability of the sensor. N: The number of sensors.

[0106] Through this fusion algorithm, the advantages and disadvantages of different sensors can be comprehensively considered, and the robustness and accuracy of the multi-sensor system can be improved. Through the sensor behavior modeling, error modeling and their dynamic correction mechanisms in this step, the present invention realizes the accurate modeling and compensation of sensor errors, providing reliable data support for the subsequent spatial alignment step. This process effectively solves the error problem of multi-modal sensors in the road monitoring environment, ensures high-precision spatial alignment, and thus improves the overall accuracy and reliability of the road monitoring system.

[0107] Step 3: Super-resolution point cloud enhancement:

[0108] Due to the low resolution of the radar point cloud, errors occur when fusing radar data with other visual data (such as camera images), affecting the spatial alignment accuracy.

[0109] Solution:

[0110] Generate low-resolution point cloud data:

[0111] Output through the point cloud data alignment and error correction in the previous steps The present invention downsamples it to generate low-resolution point cloud for use as the input of super-resolution enhancement. The downsampling ratio γ controls the sparsity of the point cloud data.

[0112]

[0113] Among them, Low-resolution point cloud data. The aligned point cloud data generated in the previous steps. γ: The downsampling ratio, usually set to 0.5. subsampled: Downsampling

[0114] Preferably, design a super-resolution point cloud enhancement network:

[0115] The present invention constructs an SR-CNN-W network, which includes two main modules:

[0116] Feature extraction module: Use a convolutional neural network (CNN) to extract spatial and geometric features from the low-resolution point cloud

[0117] ​Adaptive Residual Weighting Module: When combined with point cloud enhancement, it adaptively adjusts the weighting coefficient of each point, thus more precisely restoring the details of the point cloud.

[0118] For each point i, the super-resolution point cloud is generated by the following formula:

[0119]

[0120] where, The enhanced high-resolution point. P LR (i): The low-resolution point. ω i : The adaptive weighting coefficient of the i-th point. The residual term, which is used to restore the details of the low-resolution point cloud.

[0121] Preferably, in the residual weighting module, the present invention automatically optimizes the weighting coefficient ω of each point through training the network i , so that the network can focus on the important regions and further improve the enhancement effect.

[0122] The residual term captures the details in the point cloud data through convolution operations and residual learning, and calculates the relative enhancement value of each point.

[0123] Preferably, in order to optimize the network parameters, the present invention designs the following loss function:

[0124]

[0125] where, The i-th high-resolution point predicted by the network. P HR (i): The real high-resolution point. The residual term of the i-th point. λ: The residual term regularization coefficient, which is used to control the balance of the loss function. By minimizing this loss function, the network can optimize the spatial details of the point cloud, thereby improving the resolution of the point cloud.

[0126] Preferably, to meet the real-time processing requirements of large-scale point cloud data, a block inference strategy is adopted, and the point cloud data is divided into several blocks for parallel processing. The enhancement process of each data block is represented by the following formula:

[0127]

[0128] where, The low-resolution point cloud data of the b-th data block. The high-resolution point cloud data of the b-th data block after enhancement. Super-resolution enhancement operation. Through parallel computing and block-based inference, it can significantly improve the computational efficiency of super-resolution point cloud enhancement.

[0129] After the above process, the finally generated high-resolution point cloud can recover the important details in the low-resolution point cloud and provide high-quality input data for subsequent tasks such as road detection and target recognition.

[0130] Step 4: Spatial alignment and multi-modal data fusion

[0131] Due to different sensor types, their performances in the spatial coordinate system also vary, resulting in alignment errors during direct fusion.

[0132] Solution:

[0133] The input of this step is the super-resolution point cloud data These data come from the super-resolution enhancement process in Step 3. On this basis, the present invention needs to spatially align the point cloud data from different sensors (such as lidar and RGB images) so as to perform multi-modal fusion in the same coordinate system.

[0134] Preferably, due to the differences in spatial coordinates and resolutions of the data collected by different sensors, the present invention adopts a spatial alignment strategy based on a transformation matrix. Given an original point cloud data P HR , the present invention first obtains the transformation matrix T through geometric calibration and extrinsic parameter estimation, and then uses this transformation matrix to map the point cloud into a unified target coordinate system. The formula is as follows:

[0135]

[0136] where, P HR : The original point of the super-resolution point cloud data. The point cloud data after spatial alignment. T: The spatial transformation matrix, which is used to map the point cloud from the original coordinate system to the target coordinate system. This process ensures that the point cloud data from different sources can be subjected to subsequent fusion processing in the same coordinate system, providing a unified spatial basis for multi-modal data fusion.

[0137] Preferably, in order to enhance the accuracy of the point cloud data, especially in the low-resolution region, the present invention proposes a multi-modal data fusion method based on adaptive weighting. This method introduces an adaptive weighting factor ω i .

[0138] For data from different sensors, such as lidar point cloud P HR and RGB image I, after spatial alignment, the present invention calculates the weighted fusion result of each point The specific fusion formula is as follows:

[0139]

[0140] Wherein, The final fusion result of the i-th point. P HR (i): The i-th point from the lidar. I(i): The i-th point from the RGB image. ω i : The adaptive weight, calculated based on the importance of the point in the lidar and the RGB image.

[0141] The weight ω i is not fixed, but is dynamically adjusted according to the spatial context information and feature similarity of each point. For example, when a point has a high confidence in both the lidar and the image, its weight ω i is larger, while when the signal-to-noise ratio of one sensor is low, it is given a smaller weight.

[0142] Preferably, due to the spatial and quality differences between point cloud data in different modalities, the present invention designs an adaptive weighted learning mechanism based on a deep neural network. This mechanism dynamically adjusts the weighting factor ω of each point by learning the context information and feature similarity of each point i .

[0143] The present invention calculates a feature vector f i and a context vector c i for each point, and then uses this information to calculate the adaptive weight ω through the following formula i :

[0144]

[0145] Where: f i : The feature representation of the i-th point in the current modality (such as lidar or RGB image). c i : The context vector of the i-th point, containing the geometric information of the point and its neighborhood. n: The total number of points in the point cloud.

[0146] This weighted learning method based on context information and feature similarity can better capture the local and global feature differences of different sensor data. The weights obtained through deep learning model training can flexibly adjust the importance of different sensor data, thereby improving the quality of the fused point cloud.

[0147] Preferably, after the fusion process, the present invention uses a global optimization strategy to further improve the accuracy of the point cloud data and reduce errors. The optimization objective is to minimize the fused point cloud and the real point cloud P gtThe geometric error between them is reduced while maintaining the smoothness of the point cloud. To this end, the present invention defines the following optimization objective function:

[0148]

[0149] where The fusion result of the i-th point. P gt (i): The true high-resolution point cloud data. The gradient information of the i-th point, representing the local change of the point cloud. λ: The regularization coefficient, used to balance the weights of the geometric error and the smoothness term.

[0150] The first term in this objective function is the L2 distance between the fused point cloud and the true point cloud, which is used to measure the geometric error. The second term is the L2 norm of the point cloud gradient, which ensures the smoothness and consistency of the point cloud through regularization, avoiding excessive noise or unnatural geometric forms introduced by data fusion.

[0151] Step 5: Environment Adaptive Adjustment and Feedback Mechanism

[0152] Due to the influence of environmental changes (such as weather, light, temperature) on the sensor, the spatial alignment accuracy decreases. Existing methods lack adaptability to environmental changes and it is difficult to maintain high-precision spatial alignment in a dynamic environment.

[0153] Solution:

[0154] The input of this step is the point cloud data after multi-modal fusion from the multi-modal data fusion process in Step 4. Through the previous steps, the present invention obtains the fused point cloud data, but still needs to perform adaptive adjustment in the actual environment to cope with the influence of different environmental conditions (such as light changes, sensor errors, etc.). Therefore, the objective of this step is to dynamically adjust the processing parameters of the point cloud data through environmental feedback.

[0155] Preferably, in practical applications, the performance of the sensor in different environments may vary greatly. To address this issue, the present invention proposes an adaptive adjustment mechanism based on environmental feedback, which uses environmental changes (such as light, weather, terrain, etc.) to adjust the processing strategy of the point cloud data, thereby optimizing the fusion result. To evaluate the environmental state in real time, the present invention designs an environmental perception function E, which combines the influence of sensor data and environmental factors.

[0156] The environmental perception function E is defined as:

[0157] E(t) = α·I ambient (t) + β·R sensor (t)

[0158] Among them, E(t): represents the environmental state score at time t, which is used to evaluate the influence degree of the current environment on sensor data. I ambient (t): the influence of environmental light intensity or weather conditions (such as rainfall, etc.), which reflects the influence of the external environment. R sensor (t): the performance score of the sensor, which is calculated based on historical data and reflects the current error and noise conditions of the sensor. α, β: the parameters of the environmental perception model, which respectively represent the contribution degrees of environmental factors and sensor performance to the adjustment strategy. Through this perception function, the present invention can quantify the influence of different environments on the quality of point cloud data and dynamically adjust the data processing strategy accordingly.

[0159] Preferably, based on the output of the environmental perception function E, the present invention designs a feedback mechanism, which adjusts the fusion parameters of the point cloud data according to the environmental state score. Specifically, the present invention defines an adjustment factor γ(t), which dynamically adjusts the weighting value or spatial transformation parameter of each point according to the change of the environmental state. The definition of the adjustment factor γ(t) is as follows:

[0160]

[0161] Among them, γ(t): the adjustment factor, which controls the degree of adaptive adjustment. E(t): the environmental state score (from the environmental perception function). σ: the parameter that controls the sensitivity of environmental state change, which determines the response speed of the adjustment factor to environmental changes. Through the adjustment factor γ(t), the present invention can smoothly adjust the weight of each point during fusion, making the processing result of the point cloud data more robust in a harsh environment and providing a finer point cloud in a good environment.

[0162] Preferably, after obtaining the environmental feedback, the present invention updates the weighted fusion strategy of each point according to the adjustment factor γ(t). Specifically, the weight of each point in the fusion process will be updated according to the environmental feedback:

[0163]

[0164] Among them, The fusion result of the i-th point after environmental adjustment. The i-th point after original fusion. P gt (i): the real high-resolution point cloud data (used as a reference). This adaptive weighted fusion process dynamically adjusts the fused point cloud data according to the environmental feedback to ensure that the quality and accuracy of the point cloud data are optimized under different environmental conditions.

[0165] Finally, the adjusted point cloud data is obtained through the feedback mechanism The present invention further reduces the noise and errors caused by environmental changes through a global optimization strategy. The optimization objective is to minimize the point cloud error caused by environmental changes and maintain the smoothness and consistency of the point cloud. The optimization objective function is defined as follows:

[0166]

[0167] Wherein, The fusion result of the adjusted i-th point. P gt (i): The true high-resolution point cloud data. The gradient information of the i-th point, indicating the local change of the point cloud. λ: The regularization coefficient, used to balance the error and smoothness. Through global optimization, the present invention can further ensure that the accuracy, smoothness and consistency of the point cloud are guaranteed under environmental changes.

[0168] Through the environmental adaptive adjustment and feedback mechanism in this step, the present invention can dynamically optimize the fusion result of the point cloud data according to different environmental conditions. The environmental state perception, feedback mechanism and adaptive weighted adjustment can not only effectively cope with the sensor noise in harsh environments, but also maintain the high-quality performance of the point cloud data in changing environments. This process will enable the point cloud data processing to perform excellently in complex scenarios such as autonomous driving and intelligent transportation, and solve the problem of unstable quality of point cloud data in the prior art under dynamic environments.

[0169] The present invention can be a method, device, system and / or computer program product. The computer program product can include a computer-readable storage medium having thereon computer-readable program instructions for performing various aspects of the present invention.

[0170] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0171] Although the embodiments of the present invention have been shown and described, those skilled in the art can understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and purposes of the present invention. The scope of the present invention is defined by the claims and their equivalents.

[0172] The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein can be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0173] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read from, and write to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.

[0174] In one or more exemplary embodiments, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functions may be stored on or transmitted via a computer-readable medium as one or more instructions or code. The computer-readable medium includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. The storage media may be any available media that can be accessed by a computer. By way of example and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a web site, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, the disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where the disk typically reproduce data magnetically, while the disc reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0175] The foregoing description of the disclosure has been provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A high-precision multimodal sensor spatial alignment method for road monitoring, characterized in that: The method comprises the following steps: S1: Acquire multimodal sensor data, determine the spatial calibration of the sensor in the world coordinate system according to the multimodal sensor data, perform standardized processing of unified coordinate system conversion based on spatial alignment, and output standardized point cloud data; S2: Based on the point cloud data after the standardization processing in step S1, a sensor dynamic feature model is constructed for the error propagation process of the multimodal sensor data, and the multimodal sensor data is error compensated by regression analysis of the multimodal sensor historical data. The error compensation is dynamically corrected according to the real-time monitoring of the changes of the multimodal sensor to improve the accuracy of the spatial alignment and obtain the aligned point cloud data after error correction; S3: Obtain the aligned point cloud data generated in step S2, sample the aligned point cloud data to generate a low-resolution point cloud, construct a super-resolution point cloud enhancement network to enhance the low-resolution point cloud, and generate an enhanced super-resolution point cloud; wherein the super-resolution point cloud enhancement network includes two modules: a feature extraction module and an adaptive residual weighting module, the feature extraction module uses a convolutional neural network to extract spatial and geometric features from low-resolution points, and the adaptive residual weighting module adaptively adjusts the weighting coefficient of each point in combination with point cloud enhancement, thereby restoring point cloud details more finely; S4: Obtain the enhanced super-resolution point cloud generated in step S3, then obtain the original point cloud data from the multimodal sensor, and perform weighted fusion on the super-resolution point cloud and the original point cloud data with each point as the center to obtain a weighted fusion result for each point; S5: According to the weighted fusion result of each point obtained in S4, the processing strategy of the point cloud data is adjusted in combination with the environmental changes to optimize the fusion result and obtain the adjusted point cloud data.

2. The high-precision multimodal sensor spatial alignment method for road monitoring according to claim 1, characterized in that: The multimodal sensor data includes radar data and image data; the multimodal sensor includes a camera, a radar and an infrared sensor; In step S1, the multimodal sensor data is preprocessed, including: For radar data, a denoising algorithm based on local reconstruction is used to remove outliers by reconstructing the local neighborhood of each point; A denoising method based on a deep convolutional autoencoder is used for image data. The deep convolutional autoencoder performs denoising by learning low-frequency features and achieves this by minimizing the noise loss function of the image.

3. The high-precision multimodal sensor spatial alignment method for road monitoring according to claim 1, characterized in that: In step S1, the spatial calibration of the multimodal sensor in the world coordinate system is determined based on an external high-precision positioning system and standardized, specifically including: For external parameter calibration between sensors: the binocular camera and radar calibration method is used to align the spatial coordinate systems of the camera and radar to determine the rotation matrix and translation vector between the camera and radar; At the same time, the real-time sensor data and high-precision positioning data are combined to dynamically adjust the external parameters of the sensor by minimizing the loss function, and correct the spatial alignment relationship between the sensors; the loss function is calculated by minimizing the error between the modeled position of the sensor at the current time point and the actual position of the sensor at the current time point; Finally, the radar, camera and infrared data are converted to the same coordinate system through the calculated rotation matrix and translation vector.

4. The high-precision multimodal sensor spatial alignment method for road monitoring according to claim 1, characterized in that: In step S2, the sensor dynamic characteristic model describes the error propagation process of the sensor based on the kinematic equation of the sensor, the external environmental conditions and the relative motion relationship of the target object; In step S2, the error of the sensor is quantified and compensated by an error compensation model based on statistical regression. The error compensation model based on statistical regression fits an error compensation function through regression analysis of historical data, combined with the motion characteristics of the sensor and environmental changes, to correct the errors of different sensors; In step S2, the error compensation model based on statistical regression is updated in real time to adapt to the dynamic changes of the sensor through an error correction scheme based on an incremental learning algorithm; wherein the incremental learning algorithm is an online learning algorithm.

5. The high-precision multimodal sensor spatial alignment method for road monitoring according to claim 4, characterized in that: In step S2, a sensor error model fusion algorithm is used to obtain a global error compensation model by weighted fusion of error models of different sensors, which is then applied to the alignment process of all sensor data.

6. The high-precision multimodal sensor spatial alignment method for road monitoring according to claim 1, characterized in that: In step S3, for each point, a super-resolution point cloud enhancement network is used to generate a high-resolution point cloud using the following formula: in, is the enhanced high-resolution point, P LR (i) is the low resolution point, ω i is the adaptive weighting coefficient of the i-th point, is the residual term, which is used to restore details of low-resolution point cloud; In the adaptive residual weighting module, the weighting coefficient of each point is automatically optimized by training the network, so that the network can focus on important areas and improve the enhancement effect; the residual term Capture the details in the point cloud data through convolution operations and residual learning, and calculate the relative enhancement value of each point; The loss function of the super-resolution point cloud enhancement network It is expressed as: in, is the i-th high-resolution point predicted by the network, P HR (i) is the real high-resolution point, is the residual term of the i-th point, λ is the regularization coefficient of the residual term, which is used to control the balance of the loss function, and n is the total number of points.

7. The high-precision multimodal sensor spatial alignment method for road monitoring according to claim 6, characterized in that: In step S3, a block reasoning strategy is adopted to adapt to the real-time processing requirements of large-scale point cloud data, and the point cloud data is divided into several blocks for parallel enhancement processing.

8. The high-precision multimodal sensor spatial alignment method for road monitoring according to claim 6, characterized in that: In step S4, a spatial alignment strategy based on a transformation matrix is ​​used to align the differences in spatial coordinates and resolutions of data collected by different sensors, specifically including: Given a raw point cloud data, the transformation matrix is ​​obtained through geometric calibration and external parameter estimation, and then the point cloud is mapped to a unified target coordinate system using the transformation matrix; The weighted fusion of the super-resolution point cloud and the original point cloud data is performed with each point as the center to obtain the weighted fusion result of each point, specifically including: By considering the performance and importance of each point in different modes, an adaptive weighting factor is introduced into the point cloud fusion process. According to the adaptive weighting factor, the weighted fusion result of each point is calculated after spatial alignment. In step S4, it also includes: Design an adaptive weighted learning mechanism based on deep neural network, which dynamically adjusts the adaptive weighting factor of each point by learning the context information and feature similarity of each point, and captures the local and global feature differences of different sensor data; At the same time, after the fusion process, a global optimization strategy is used to improve the accuracy of the point cloud data and reduce the error; the optimization goal of the global optimization strategy is to minimize the geometric error between the fused point cloud obtained by the weighted fusion result of each point and the real point cloud, and to maintain the smoothness of the point cloud; The optimization objective function of the global optimization strategy is specifically: The first term is the L2 distance between the fused point cloud and the true point cloud, which is used to measure the geometric error; The second term is the L2 norm of the point cloud gradient, which ensures the smoothness and consistency of the point cloud through regularization and avoids excessive noise or unnatural geometry due to data fusion.

9. The high-precision multimodal sensor spatial alignment method for road monitoring according to claim 1, characterized in that: In step S5, an adaptive adjustment mechanism based on environmental feedback is adopted to adjust the processing strategy of point cloud data by utilizing environmental changes, thereby optimizing the fusion result; wherein the environmental changes are quantified by an environmental perception function, and the environmental perception function is calculated according to the influence of environmental light intensity or weather conditions and the performance score of the sensor; At the same time, the adaptive adjustment mechanism based on environmental feedback will adjust the fusion parameters of the point cloud data according to the environmental status score, including: An adjustment factor is defined, and the adjustment factor dynamically adjusts the weight value or spatial transformation parameter of each point according to the change of the environmental state. Through the adjustment factor, the weight of each point during fusion is smoothly adjusted, so that the processing result of the point cloud data is more robust in harsh environments, and a more refined point cloud can be provided in good environments.

10. A high-precision multimodal sensor spatial alignment method for road monitoring according to any one of claims 8 to 9, characterized in that: In step S5, after obtaining environmental feedback, the weighted fusion strategy of each point is updated according to the adjustment factor; wherein the weight of each point in the fusion process will be updated according to the environmental feedback; According to the weighted fusion strategy of each point after update, the noise and error caused by environmental changes are reduced through a global optimization strategy. The optimization goal of the global optimization strategy is to minimize the point cloud error caused by environmental changes, that is, to minimize the error between the weighted fusion combination of each point and the real high-resolution point cloud data, and to maintain the smoothness and consistency of the point cloud.

Images