Intelligent environment perception and calibration optimization system and use method

Abstract

The invention discloses an intelligent environment perception and calibration optimization system and a use method, the system comprises a camera internal reference calibration module, an external reference calibration module, a data acquisition and frequency adjustment module, an annotation and training module, a model conversion and reasoning module and a fusion and release module, outputting a high-precision internal reference and a distortion coefficient; the external parameter calibration module interactively calculates a rotation translation matrix between the camera and the laser radar based on the two-dimensional image and the three-dimensional point cloud data; the data acquisition module adjusts topic frequency and supports cloud synchronous storage; the labeling training module is used for training a detection or segmentation model based on models such as YOLO and RTMPose, and realizing efficient reasoning through ONNX conversion and TensorRT acceleration; and finally, the fusion publishing module maps the two-dimensional result to the three-dimensional point cloud, and outputs a sensing result with coordinates. According to the system, through an optimization algorithm and interactive optimization, the calibration precision and adaptability in a dynamic complex environment are effectively improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent perception and machine vision, and particularly relates to an intelligent environment perception and calibration optimization system. BACKGROUND

[0002] With the rapid development of autonomous driving, robots and intelligent transportation systems, multi-sensor fusion technology is increasingly widely used in environment perception and precise positioning. As core sensors, cameras and laser radars (LiDARs) have become key configurations for realizing high-precision environment understanding in autonomous driving systems due to their respective advantages in image semantic information and three-dimensional spatial structure perception. The effectiveness of sensor fusion is highly dependent on the accuracy of camera intrinsic parameters and the precision of camera and laser radar extrinsic calibration, which directly affects the surrounding environment perception ability of the system and the positioning performance of the vehicle. The demand for high reliability and high adaptability perception systems in the industry is constantly driving sensor calibration technology towards higher precision and stronger real-time performance.

[0003] Currently, camera intrinsic calibration mostly adopts traditional methods based on specific calibration boards, which capture images of the calibration board in different poses, and solve the internal parameters of the camera by using corner detection and geometric constraints. In terms of extrinsic calibration, mainstream methods usually rely on images and laser point cloud data collected in static scenes, match the corresponding structures in the image and point cloud by extracting features such as corners and edges in the image, and calculate the relative pose relationship between the camera and the laser radar by using optimization algorithms. This kind of method can achieve high calibration accuracy in well-controlled environments, and has become a common practice in many experiments and engineering applications.

[0004] However, a significant disadvantage of existing methods is their high dependence on environmental staticity and operation normativity, which makes it difficult to maintain calibration accuracy and stability in complex dynamic scenarios. Traditional calibration methods usually require the scene to remain static and the light to be uniform during calibration, while actual application environments such as urban roads and high-speed driving scenarios often have interference factors such as dynamic objects, light changes and sensor jitter, which lead to decreased consistency of feature extraction and increased matching error, seriously affecting the accuracy and robustness of extrinsic calibration. SUMMARY

[0005] The purpose of the present application is to provide an intelligent environment perception and calibration optimization system that can adapt to dynamic and complex environments and automatically update calibration parameters in real time, to solve the problem of decreased calibration accuracy caused by environmental staticity dependence in the prior art. On the other hand, a use method of the intelligent environment perception and calibration optimization system is provided.

[0006] Technical scheme: The intelligent environment perception and calibration optimization system disclosed by the application comprises a camera internal parameter calibration module, a camera and laser radar external parameter calibration module, a data acquisition and frequency adjustment module, a labeling and training module, a model conversion and reasoning module, and a fusion and release module, wherein:

[0007] The camera internal parameter calibration module runs in a preset Ubuntu and ROS environment, supports two image input modes of USB direct connection and ROS topic, collects images by using a checkerboard calibration board, and the calibration process comprises the following steps: firstly, performing preliminary calibration to obtain initial estimated values of camera internal parameters and distortion parameters, then using an optimization algorithm to perform nonlinear optimization on the initial estimated values, and finally outputting an optimized internal parameter matrix, distortion coefficient and image comparison results before and after calibration.

[0008] The camera and laser radar external parameter calibration module, after the camera internal parameter calibration is completed, based on two-dimensional images and three-dimensional point cloud data collected in a static scene, loads three-dimensional coordinate data obtained by a user in a point cloud processing software and corresponding two-dimensional image pixel point coordinates selected by the user in a graphical interface, calls an external parameter solving algorithm to calculate a rotation matrix and a translation vector, displays a projection error in real time and supports removing error points and re-optimizing the external parameter.

[0009] The data acquisition and frequency adjustment module adjusts the frequency of image and point cloud topics by using a ROS tool, and uses a data recording and playback tool to perform unified storage, management and cloud uploading of data.

[0010] The labeling and training module labels target frames, contours or key points in the image by using an image labeling tool, and trains based on a target detection YOLO or pose estimation RTMPose model to generate a corresponding detection or segmentation model.

[0011] The model conversion and reasoning module converts the detection or segmentation model obtained by training into an open neural network exchange format ONNX, and further generates a high-performance reasoning engine, which is used for real-time reasoning on a GPU or an embedded platform by using a C++ program, and outputs detection results or segmentation results of two-dimensional images.

[0012] The fusion and release module maps the detection or segmentation results in the two-dimensional image to a three-dimensional point cloud coordinate system in combination with the camera internal parameters and external parameters, outputs a perception result with three-dimensional coordinates, and releases in the form of a ROS topic.

[0013] The application realizes high-precision intrinsic parameter estimation and distortion correction through a camera intrinsic parameter calibration module, realizes dynamic error optimization and real-time projection verification in combination with a camera and laser radar extrinsic parameter calibration module, effectively improves the parameter accuracy of multi-sensor in complex environment; the data acquisition and frequency adjustment module guarantees efficient synchronization and flexible management of multi-source data, supports stable operation of the system in dynamic scenes; the labeling and training module generates a robust perception model based on actual scene data, enhances the adaptability to environmental changes; the model conversion and reasoning module ensures efficient real-time reasoning of the algorithm on multiple hardware platforms, and outputs reliable two-dimensional perception results; finally, the fusion and release module accurately maps the two-dimensional perception information to the three-dimensional space, forming a unified and accurate environmental perception output, thereby significantly improving the calibration stability and perception accuracy of the system in dynamic complex scenes, overcoming the calibration failure and performance degradation problems caused by the static dependence of traditional methods.

[0014] Preferably, the camera intrinsic parameter calibration module is further configured to perform calibration through Python-OpenCV, Matlab or Calibration tools, and output a calibration report containing re-projection error distribution and straightness recovery evaluation results.

[0015] The camera intrinsic parameter calibration module realizes flexible and high-precision camera parameter estimation and distortion correction by integrating multiple mainstream calibration tools, and the calibration report output by the camera intrinsic parameter calibration module contains re-projection error distribution and straightness recovery evaluation results, which can comprehensively and intuitively reflect the calibration quality, effectively guarantee the verifiability and reliability of the intrinsic parameter calibration process, and significantly improve the accuracy and stability of subsequent perception tasks in complex dynamic environments.

[0016] Preferably, the camera and laser radar extrinsic parameter calibration module supports SQPnP, EPnP and other extrinsic parameter optimization algorithms, and provides a graphical interface for users to interactively select point pairs, remove abnormal points and re-optimize.

[0017] The camera and laser radar extrinsic parameter calibration module integrates multiple robust extrinsic parameter optimization algorithms and provides an intuitive graphical interface, allowing users to flexibly select corresponding points, remove abnormal data in real time and dynamically optimize parameters, significantly improving the accuracy, stability and human-machine collaboration efficiency of the extrinsic parameter calibration process, and effectively enhancing the calibration adaptability and perception consistency of the multi-sensor system in complex dynamic environments.

[0018] Preferably, the data acquisition and frequency adjustment module is configured to reduce the publishing frequency of the original image topics, and realizes unified management and cloud synchronization through rosbag.

[0019] The data acquisition and frequency adjustment module reduces the publishing frequency of original image data dynamically, and realizes unified packaging, flexible management and cloud cooperation of multi-source data in combination with the rosbag tool, significantly reduces the system communication and storage load, improves the overall efficiency and scalability of data acquisition and processing, and provides reliable support for stable acquisition, transmission and long-term operation of large-scale perception data in a complex dynamic environment.

[0020] Preferably, the labeling and training module introduces data enhancement operations including brightness change, rotation, scaling and noise disturbance in the model training process.

[0021] The labeling and training module effectively improves the generalization and robustness of the generated model to illumination conditions, angle changes and environmental disturbances by introducing various data enhancement operations including brightness change, rotation, scaling and noise disturbance in the model training, significantly enhances the adaptability and recognition accuracy of the perception system in real complex dynamic scenes.

[0022] Preferably, the model conversion and inference module supports dynamic resolution input and batch processing strategy.

[0023] The model conversion and inference module significantly enhances the adaptability and inference efficiency of the model under different scene and hardware conditions, effectively improves the processing capability of the perception system to variable input size and concurrent tasks, and guarantees real-time, stable and efficient high-precision inference performance in complex dynamic environments.

[0024] Preferably, the fusion and publishing module publishes the three-dimensional coordinate perception result with time stamp, confidence and model version information.

[0025] The fusion and publishing module significantly improves the traceability, reliability and system transparency of the perception result by attaching time stamp, confidence and model version to the three-dimensional perception result, effectively supports high-precision fusion of multi-modal data and maintenance of time sequence consistency in dynamic scenes, and provides key data support for reliability, reproducibility and continuous optimization of perception decision in complex environments.

[0026] The use method of the intelligent environment perception and calibration optimization system provided by the application comprises the following steps:

[0027] The camera intrinsic calibration module is used to collect a checkerboard image and perform calibration to obtain the intrinsic matrix and distortion parameters of the camera;

[0028] In the case of fixed installation of the camera and the laser radar, the rosbag data packet of the static scene is recorded, and the image frame and the point cloud frame are parsed;

[0029] The correspondence relationship data between the two-dimensional image feature points and the three-dimensional point cloud feature points is provided to a point pair management unit of the extrinsic calibration module;

[0030] An extrinsic parameter solving algorithm is called by the camera and lidar extrinsic calibration module to calculate a rotation matrix and a translation vector.

[0031] The data acquisition and frequency adjustment module is used to reduce the topic frequency and record data.

[0032] The image is labeled using a Labelme tool, and a YOLO or RTMPose model is trained based on the labeled data using the labeling and training module to generate an ONNX format model file.

[0033] The ONNX model is converted into a TensorRT engine and inference is performed using the model conversion and inference module.

[0034] The two-dimensional inference result and the camera parameters are fused using the fusion and publishing module, mapped to the point cloud coordinate system, and three-dimensional perception results are generated and published as ROS topics.

[0035] Through a systematic multi-module collaborative process, a complete closed loop from sensor calibration, data acquisition and processing, model training and optimization to three-dimensional perception output is achieved: first, a reliable sensor parameter basis is constructed with high-precision intrinsic calibration and robust extrinsic parameter estimation, then the model environmental adaptability is improved through flexible data management and enhanced training, and finally, real-time perception calculation is achieved relying on an efficient inference engine, and the two-dimensional result with rich semantic information is accurately mapped to the three-dimensional space to form a traceable and highly reliable perception output, thereby comprehensively enhancing the perception accuracy, robustness and practical value of the system in a dynamic and complex environment.

[0036] Preferably, in the step of calculating the rotation matrix and the translation vector, if the projection error calculated by the extrinsic calibration module exceeds a set threshold, the correspondence relationship data is modified and recalculated.

[0037] By introducing a projection error threshold judgment and a corresponding data modification and recalculation mechanism in the extrinsic calibration process, the iterative optimization ability and fault tolerance of the calibration process are significantly enhanced, effectively ensuring the accuracy and reliability of the extrinsic parameter estimation between the camera and the lidar, and improving the calibration success rate and parameter robustness of the multi-sensor system in complex actual scenarios.

[0038] Preferably, in the step of publishing the three-dimensional perception result as a ROS topic, a timestamp, a confidence and a model version number are additionally published.

[0039] By attaching time stamps, confidence levels, and model version information to the published three-dimensional perception results, the traceability, system transparency, and cross-module collaboration of the perception results are significantly enhanced, effectively supporting the temporal alignment and credible fusion of multi-source data in dynamic environments, and providing reliable perception outputs with complete context information, quantifiable evaluation, and easy iterative optimization for high-level decision-making.

[0040] Advantages: Compared with the prior art, the present application has the following significant advantages: 1. The system can real-time adjust and optimize the calibration parameters through the optimization algorithm of the camera intrinsic calibration module and the multi-algorithm support and interactive optimization of the extrinsic calibration module, effectively dealing with sensor errors caused by environmental changes, and solving the problem of declining calibration accuracy in traditional methods in dynamic and complex environments; 2. The system integrates multiple calibration tools and algorithms, combined with automatic processes and standardized calibration boards, significantly improving the calibration accuracy and efficiency of camera intrinsic parameters and camera-laser radar extrinsic parameters, ensuring stable and reliable operation in complex working conditions; 3. The system has built-in graphical interface and one-key start function, supporting user interactive operation and abnormal point elimination, greatly reducing the operation complexity, making calibration and data management more intuitive and convenient; 4. Through frequency adjustment, rosbag unified management and cloud synchronization function, efficient data acquisition, storage and sharing are realized, taking into account the flexibility of local and cloud deployment, improving the overall resource utilization efficiency of the system; 5. Deep learning model training, ONNX conversion and TensorRT accelerated inference are used, combined with C++ efficient implementation, significantly improving the real-time processing capability of image detection, segmentation and three-dimensional fusion, meeting the high concurrency and low latency application requirements; 6. The system can be flexibly deployed on local client or cloud platform, adapting to different hardware environments and scale requirements, expanding the practicality and accessibility in multiple scenarios such as industry and scientific research; 7. Through modular design, the whole process of calibration, acquisition, training, inference and fusion is integrated, reducing manual intervention and reducing the error rate, realizing end-to-end automatic intelligent perception and calibration optimization. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 The system structure diagram of the present application;

[0042] Figure 2 The system use flow diagram of the present application;

[0043] Figure 3 The camera intrinsic calibration interface diagram of the present application;

[0044] Figure 4 The camera and laser radar extrinsic calibration interface diagram of the present application. DETAILED DESCRIPTION

[0045] The technical solutions of the present application will be further described below in conjunction with the drawings.

[0046] This invention provides an intelligent environmental perception and calibration optimization system, such as... Figure 1 and Figure 2 As shown, it includes: camera intrinsic parameter calibration module, camera and lidar extrinsic parameter calibration module, data acquisition and frequency adjustment module, annotation and training module, model conversion and inference module, and result fusion and publishing module.

[0047] The camera intrinsic parameter calibration module runs in the system's built-in Ubuntu and ROS environment, supporting both USB direct connection and ROS topic input methods. It uses a checkerboard calibration board to acquire multiple sets of images, first performing coarse calibration via ROS, and then using Python-OpenCV, Matlab, or Calibration tools for fine optimization. Figure 3 As shown, the system outputs the focal length, principal point position, and distortion parameters, and generates image comparisons and error reports before and after calibration to ensure the accuracy and reliability of the intrinsic parameter solution.

[0048] The camera and LiDAR extrinsic parameter calibration module is performed after the intrinsic parameters are determined, based on the ROSBAg parsing images and point cloud data recorded in a static scene. For example... Figure 4 As shown, users select the 3D coordinates of the point cloud using CloudCompare and the corresponding image points in the PyQt5 interface, forming point pair data. After loading the point pairs, the system calls extrinsic parameter solving algorithms (such as SQPnP and EPnP) to automatically calculate the rotation matrix and translation vector, displaying the projection results and pixel errors, and supporting the removal of outlier point pairs for re-optimization. The final generated extrinsic parameter file can be saved in multiple formats for direct use in sensor fusion.

[0049] The data acquisition and frequency adjustment module utilizes topic_tools / throttle to reduce topic frequency, effectively reducing data size. For example, reducing a 10Hz video stream to 1Hz compresses storage from 1.8GB to 170MB, significantly reducing bandwidth and storage overhead. The acquired data is managed uniformly using rosbag and uploaded to cloud storage.

[0050] The annotation and training modules perform keypoint or segmentation annotations according to task requirements, using the Labelme tool to generate standardized annotation files. Data is then fed into the deep learning model for training; YOLO is suitable for object detection and segmentation, while RTMPose is suitable for keypoint detection. After training, the model files are exported in ONNX format to ensure cross-platform compatibility.

[0051] The model conversion and inference module converts the ONNX model into a TensorRT engine, which runs on a GPU or embedded platform, and is implemented by a C++ program for efficient inference, balancing real-time performance and computational efficiency. The inference results include two-dimensional detection boxes, key points, or segmentation regions, which are then mapped to point clouds in combination with the camera intrinsic and extrinsic parameters, achieving alignment between images and three-dimensional space.

[0052] The result fusion and publishing module performs three-dimensional reconstruction on the mapping results to generate three-dimensional data of key points or material regions, and publishes them through ROS topics (such as / fusion / keypoints_3d or / fusion / segmentation_3d), providing input for upper-level environment perception, path planning, and control modules. The system supports publishing results with timestamps and confidence levels to ensure the timeliness and robustness of the information.

[0053] Further, the system supports data augmentation, batch processing, and dynamic resolution inference, improving the robustness and applicability of the model in complex environments. The cloud management platform manages data and models in a versioned manner, recording the dataset, parameters, and performance indicators of each training, ensuring traceability and continuous optimization capabilities of the results.

[0054] The detailed usage principles of the above system are as follows:

[0055] (1) Camera intrinsic calibration implementation

[0056] Implementation principle:

[0057] The core of camera intrinsic calibration is to estimate the focal length, principal point position, and distortion parameters of the camera through multiple checkerboard images, thereby establishing an accurate imaging model. This system preinstalls ROS calibration tools and dependencies in the cloud Ubuntu environment, and users only need to start with one key to enter the calibration interface. Through the combination of coarse calibration and fine calibration, together with evaluation indicators such as re-projection error and straightness recovery, high-precision estimation of camera intrinsic parameters can be achieved, and the results can be saved in a standardized format, providing reliable support for subsequent extrinsic calibration and sensor fusion.

[0058] Content:

[0059] The system supports two image input modes: USB interface start and ROS topic start. The USB mode directly calls Python-OpenCV to obtain the camera screen, which is suitable for quick use; the ROS mode requires users to prepare camera SDK drivers and start topics, which is suitable for integrated application scenarios.

[0060] During the calibration process, a standard black and white checkerboard calibration board is used, and the system guides the user to input the checkerboard parameters (number of corner points, single grid length) to ensure accurate dimensions.

[0061] Data collection is divided into three groups, a total of about 200-300 images, covering different angles, distances and positions, the system will automatically remove blurred, overexposed, underexposed images during the collection process, and display the collection coverage in real time.

[0062] The coarse calibration stage outputs the initial value of the intrinsic parameter and the distortion model suggestion from the ROS calibration program. The user can enter the fine calibration according to the prompt. Fine calibration provides three methods:

[0063] Python-OpenCV: lightweight and fast, suitable for on-site calibration;

[0064] Matlab tool: comprehensive visual report, suitable for high-precision research;

[0065] Calibration tool: based on global bundle optimization, suitable for fisheye and large sample data sets.

[0066] After calibration, the system displays the image comparison before and after distortion correction, and intuitively presents the edge straight line recovery and corner error distribution. The user can judge the calibration effect.

[0067] The final result includes the intrinsic matrix, distortion coefficient and statistical error information, which can be saved as a YAML, JSON or ROScamera_info format file, and synchronized to the cloud parameter library.

[0068] Steps:

[0069] The user selects the camera startup mode in the interface, the USB mode directly calls OpenCV, and the ROS mode fills in the camera topic and confirms that the driver is running normally.

[0070] After inputting the checkerboard parameters, start collecting calibration images, and the system automatically monitors the collection quality and generates coverage prompts to ensure data diversity.

[0071] After completing the collection, the system calls the ROS calibration program for coarse calibration and outputs the initial value parameters.

[0072] The user selects the appropriate fine calibration method, and the system completes the optimization and generates a calibration report.

[0073] View the correction comparison effect in the interface. If the error exceeds the threshold, the system prompts to supplement or recalculate.

[0074] After confirming that the result is correct, the user saves the calibration parameters, and the system writes them to the cloud database at the same time, which is called by the external parameter calibration and fusion module.

[0075] (2) Camera and laser radar external parameter calibration implementation

[0076] Implementation principle:

[0077] The camera and laser radar extrinsic calibration is a key step to establish the geometric mapping relationship between the three-dimensional point cloud coordinate system and the two-dimensional image coordinate system. The traditional method relies on special calibration board or complex automatic algorithm. Although these methods can automatically generate point pairs, they often require expensive calibration equipment, strict scene arrangement and high computational cost. The method of the present application is based on the combination of manual interaction and static data, which directly drives the extrinsic parameter optimization algorithm to solve the problem by selecting feature points in the point cloud and image. This not only avoids the dependence on complex hardware, but also reduces the consumption of computing resources, and at the same time ensures the accuracy of point pair selection and the reliability of calibration results through human-computer combination.

[0078] Content:

[0079] The system takes the completed camera intrinsic parameter and distortion correction result as the premise, ensures that the input image data has removed the influence of distortion, and improves the accuracy of projection matching.

[0080] In actual deployment, the user needs to fix the camera and laser radar installation position, record the rosbag file and parse the image and point cloud data in it. Since the calibration process only requires static data, the operation requirement is low and the external environment constraint is small.

[0081] In the point cloud data, the user selects no less than six points with clear spatial position through the tool-picking function of the CloudCompare tool. The system automatically outputs the three-dimensional coordinates (x, y, z) of these points and saves them as a standard format file.

[0082] In the image data, the user selects the two-dimensional points corresponding to the point cloud coordinates through the PyQt5 graphical interface built-in the system. The interface supports automatic recording and provides add, delete, modify and query functions to ensure the flexibility and accuracy of point pair management.

[0083] After completing the point pair extraction, the user enters the extrinsic calibration interface. The system allows to load the saved point pair data and can select various extrinsic parameter estimation algorithms (such as SQPnP, EPnP, etc.). After the algorithm is executed, the system calculates and displays the projection error of each point pair, and simultaneously superimposes the projection result of the three-dimensional point on the image in real time.

[0084] The user can intuitively judge the calibration quality and delete the point pairs with too large error to re-run the optimization when necessary, gradually improving the accuracy and stability of the extrinsic parameter result.

[0085] Finally, the system generates an extrinsic parameter file containing the rotation matrix and translation vector, and supports saving in YAML, JSON or ROS camera_info format. The file can not only be called by subsequent sensor fusion, but also be synchronized to the cloud database to form traceable data with timestamp and version number.

[0086] Steps:

[0087] Read and load the camera intrinsic distortion correction parameters to ensure that the image data is distortion-free.

[0088] Fix the camera and laser radar installation, record the rosbag and parse the image and point cloud frames, and select a frame of static data as the calibration input.

[0089] Use the CloudCompare tool to select feature points in the point cloud, obtain and save the three-dimensional coordinates.

[0090] Select the corresponding image points in the PyQt5 interface, and the system automatically saves the point pair information.

[0091] Open the extrinsic calibration interface, load the point pair data, and select the appropriate extrinsic optimization algorithm to perform calculations.

[0092] Check the pixel error and projection results. If the error is large, remove the point pair and re-calibrate.

[0093] Save the extrinsic calibration results, generate standardized files and synchronize to the cloud for subsequent calls.

[0094] (3) Key point detection and segmentation detection implementation

[0095] Implementation principle:

[0096] After the camera intrinsic distortion correction and camera and laser radar extrinsic calibration are completed, the image and point cloud can be spatially aligned. The present application reduces the publishing frequency of ROS topics, effectively reducing the storage and computing burden, thereby supporting large-scale data acquisition. After image data is labeled, model training and inference acceleration, key point detection or segmentation detection can be achieved, and point cloud data is fused to generate results with three-dimensional coordinates in the form of ROS topics, providing input for upper-level environment perception and decision planning.

[0097] Content:

[0098] In the data acquisition link, the system uses the topic_tools / throttle tool to adjust the topic frequency, reducing the camera image and point cloud topics from 10Hz to 1Hz. It has been verified that this method can compress the storage requirement from 1.8GB to about 170MB, significantly reducing the data volume. The collected data is recorded and parsed by rosbag and uploaded to the cloud for unified management and storage, ensuring data security and traceability.

[0099] In the data processing stage, the collected image data is manually annotated according to task requirements. The system uses the Labelme tool to support key point annotation and region segmentation annotation, and the annotation results are saved as standardized files. The annotated data enters the model training process, and appropriate deep learning models can be selected according to the scene: the YOLO model is suitable for target detection and material segmentation, and the RTMPose model is suitable for high-precision key point detection. After training, the model file is exported in ONNX format to ensure cross-platform deployment compatibility.

[0100] In the inference stage, the ONNX model is converted to a TensorRT model and executed efficiently on a GPU or embedded platform. The inference is implemented in C++, which can reduce latency while ensuring real-time performance. The inference results include two-dimensional detection boxes, segmentation regions, or key point coordinates. The system uses known camera intrinsic and extrinsic parameters to project the results to the point cloud coordinate system, achieving spatial fusion of images and point clouds.

[0101] The fusion results are published as new topics through the ROS interface, such as / fusion / keypoints_3d or

[0102] / fusion / segmentation_3d, which outputs key points or segmentation regions in three-dimensional space, providing direct input for path planning and control algorithms.

[0103] To improve the robustness and practicality of the system, various data augmentation methods are supported during training, such as brightness changes, rotations, and scaling, to enhance the model's adaptability in complex environments. In the inference stage, the system supports dynamic resolution and batch inference strategies to further improve efficiency. The cloud platform also establishes a data and model version management mechanism, recording the dataset, parameter configurations, and performance indicators used in each training, ensuring that the results are traceable and continuously optimized. In addition, detection and segmentation results can be accompanied by confidence distributions to provide uncertainty information, assisting upper-level modules in making more robust decisions.

[0104] Steps:

[0105] Use the topic_tools / throttle tool to reduce the publication frequency of camera and point cloud topics, record rosbag, and upload it to the cloud.

[0106] Parse the data in rosbag, extract image and point cloud frames, and store them centrally.

[0107] Use the Labelme tool to annotate key points or regions on the image, forming standardized annotation files.

[0108] Input the annotated data into the deep learning model for training, and export the ONNX format model file after completion.

[0109] Convert the ONNX model to a TensorRT model and execute inference through a C++ program.

[0110] Combine the two-dimensional detection or segmentation results obtained by inference with the camera-intrinsic and extrinsic parameters, project them into the point cloud coordinate system to realize spatial fusion.

[0111] Publish the fused three-dimensional results as ROS topics to provide input for environment perception and decision planning.

[0112] This system is not only suitable for specific scenarios, but also can be widely applied to environment perception and sensor calibration in the field of autonomous driving, as well as multi-sensor fusion tasks in service robots, industrial robots and other types of intelligent systems. Through the deployment of this system, high-precision camera intrinsic calibration and camera-lidar extrinsic calibration can be achieved in complex dynamic environments, and real-time processing of key point detection and segmentation detection is supported, thereby providing a reliable perception foundation for path planning and obstacle avoidance control of autonomous vehicles, environmental understanding and autonomous operation of robots, with good universality and expandability.

Claims

1. An intelligent environment perception and calibration optimization system, characterized in that, The camera intrinsic calibration module, the camera and lidar extrinsic calibration module, the data acquisition and frequency adjustment module, the labeling and training module, the model conversion and inference module, and the fusion and publishing module, wherein: The camera intrinsic calibration module runs in a preset Ubuntu and ROS environment, supports two image input modes of USB direct connection and ROS topic, and collects images by using a checkerboard calibration board. The calibration process includes: first performing preliminary calibration to obtain initial estimated values of camera intrinsic parameters and distortion parameters, then using an optimization algorithm to perform nonlinear optimization on the initial estimated values, and finally outputting the optimized intrinsic parameter matrix, distortion coefficient, and image comparison results before and after calibration. The camera and lidar extrinsic calibration module, after the camera intrinsic calibration is completed, based on two-dimensional images and three-dimensional point cloud data collected in a static scene, loads three-dimensional coordinate data obtained by a user in a point cloud processing software and corresponding two-dimensional image pixel point coordinates selected by the user in a graphical interface, calls an extrinsic parameter solving algorithm to calculate a rotation matrix and a translation vector, displays the projection error in real time and supports removing error points and re-optimizing the extrinsic parameters. The data acquisition and frequency adjustment module adjusts the frequency of image and point cloud topics through the ROS tool, and uses data recording and playback tools to perform unified storage, management and cloud uploading of data. The labeling and training module labels the target frame, contour or key point in the image through an image labeling tool, and trains based on a target detection YOLO or pose estimation RTMPose model to generate a corresponding detection or segmentation model. The model conversion and inference module converts the trained detection or segmentation model into an open neural network exchange format ONNX, and further generates a high-performance inference engine. Through C++ program, real-time inference is performed on GPU or embedded platform, and the detection result or segmentation result of two-dimensional image is output. The fusion and publishing module maps the detection or segmentation result in the two-dimensional image to the three-dimensional point cloud coordinate system combined with the camera intrinsic parameters and the extrinsic parameters, outputs the perception result with three-dimensional coordinates, and publishes in the form of ROS topic.

2. The intelligent environment perception and calibration optimization system of claim 1, wherein, The camera intrinsic calibration module is further configured to perform calibration through Python-OpenCV, Matlab or Calibration tool, and output a calibration report containing re-projection error distribution and straightness recovery evaluation results.

3. The intelligent environment perception and calibration optimization system of claim 1, wherein, The camera and lidar extrinsic calibration module supports SQPnP, EPnP and other extrinsic parameter optimization algorithms, and provides a graphical interface for users to interactively select point pairs, remove abnormal points and re-optimize.

4. The intelligent environment perception and calibration optimization system of claim 1, wherein, The data acquisition and frequency adjustment module is configured to reduce the publishing frequency of the original image topic, and realizes unified management and cloud synchronization through rosbag.

5. The intelligent environment perception and calibration optimization system of claim 1, wherein, The labeling and training module introduces data enhancement operations including brightness change, rotation, scaling and noise disturbance in the model training process.

6. The intelligent environment perception and calibration optimization system of claim 1, wherein, The model conversion and inference module supports dynamic resolution input and batch processing strategy.

7. The intelligent environment perception and calibration optimization system of claim 1, wherein, The fusion and publishing module publishes the perception result with three-dimensional coordinates with timestamp, confidence and model version information.

8. A method for using an intelligent environmental perception and calibration optimization system, characterized in that, The method comprises the following steps: The camera intrinsic calibration module is used to collect a checkerboard image and perform calibration to obtain an intrinsic matrix and distortion parameters of the camera; In the case of fixed installation of the camera and the laser radar, a rosbag data packet of a static scene is recorded, and image frames and point cloud frames are parsed; Correspondence data between two-dimensional image feature points and three-dimensional point cloud feature points are provided to a point pair management unit of the camera and laser radar extrinsic calibration module; An extrinsic solution algorithm is called by the camera and laser radar extrinsic calibration module to calculate a rotation matrix and a translation vector; The data collection and frequency adjustment module is used to reduce the topic frequency and record data; Labelme is used to label the image, and the labeling and training module is used to train a YOLO or RTMPose model based on the labeled data to generate an ONNX format model file; The model conversion and inference module is used to convert the ONNX model into a TensorRT engine and perform inference; The fusion and publishing module is used to fuse the two-dimensional inference result and the camera parameters, map them to a point cloud coordinate system, generate a three-dimensional perception result, and publish it in a ROS topic.

9. The method of using the intelligent environment-aware and calibration optimization system of claim 8, wherein, In the step of calculating the rotation matrix and the translation vector, if the projection error calculated by the extrinsic calibration module exceeds a set threshold, the correspondence data is modified and recalculated.

10. The method of using the intelligent environment-aware and calibration optimization system of claim 8, wherein, In the step of publishing the three-dimensional perception result in a ROS topic, time stamp, confidence, and model version number information are additionally published.

Images