HIL simulation method for underground mining whole process

Abstract

The invention discloses an HIL simulation method for an underground mining whole process, and relates to the field of automatic driving simulation testing, and the method comprises the steps: collecting point cloud data of an underground scene, and generating a simulation scene in a CARLA format; compiling and importing the generated simulation scene in the CARLA format through a Unreal engine, building a vehicle model and a laser radar sensor model in a Carla simulator, and collecting a point cloud data packet of the simulation scene; according to the point cloud data packet, mapping is carried out through an SLAM algorithm, and a point cloud map in a PCD format is generated; constructing a map of the simulation scene according to the PCD point cloud map; building a hardware-in-the-loop HIL simulation platform comprising an upper computer, a lower computer and an execution mechanism; and loading the constructed map of the simulation scene to the constructed hardware-in-the-loop HIL simulation platform, simulating a real well working condition scene through the simulation platform, and verifying a well working condition unmanned driving algorithm. In order to solve the problem that unmanned driving tests are difficult to comprehensively evaluate in the prior art, the coverage rate of the tests is increased through a hardware integrated full-process closed loop and the like.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving simulation testing, and in particular to a HIL simulation method for the entire process of underground mining. Background Art

[0002] In the mining industry, underground working environments are complex, with major safety risks and high labor intensity. The introduction of autonomous driving technology and the use of unmanned vehicles to achieve mining, transportation and other operations can effectively improve the harsh underground environment and improve operational safety and production efficiency.

[0003] Although autonomous driving technology has been widely used in urban road environments, there are still many technical challenges in transplanting it to underground conditions. Unlike urban roads, underground tunnels are narrow, the lighting conditions are poor, and positioning is difficult, which puts higher demands on the vehicle's environmental perception and autonomous decision-making capabilities. In addition, underground vehicles need to work in coordination with equipment such as tunnel boring machines and transporters, and the operation process is complex and the safety risks are high.

[0004] In order to verify the feasibility and safety of the underground unmanned driving system, it is necessary to conduct a comprehensive test of the system in a simulation environment before actual deployment. However, there is currently a lack of an unmanned driving simulation test system for underground working conditions. Mainstream commercial simulation software such as Prescan and VTD are mainly oriented to urban road scenes and are difficult to directly apply to underground environments. Although Carla is more flexible in scene construction and sensor configuration, it lacks maps and vehicle models that match the underground environment, and the workload of building a full-process simulation system is large.

[0005] On the other hand, although simulation tests based entirely on software can verify the effectiveness of autonomous driving algorithms, they cannot fully evaluate the actual performance of the system after integration. Hardware-in-the-loop simulation can incorporate controller hardware into the simulation environment and interact with real sensors and actuators to obtain test results that are closer to reality. However, how to build a hardware-in-the-loop system that includes on-board controllers, sensors, actuators and other components, and realize real-time data exchange with the software simulation environment is a complex system integration problem.

[0006] Therefore, developing a hardware-in-the-loop simulation system for unmanned vehicles in underground conditions to achieve a closed loop of the entire process from real scene acquisition, 3D reconstruction to simulation testing and hardware integration is of great significance for promoting the development of unmanned driving technology in mines and ensuring system safety. The technical solution proposed in this paper aims to break through the above bottlenecks and provide a low-cost, efficient and reliable unmanned driving simulation test method for the mining industry. Summary of the invention

[0007] In response to the problem in the existing technology that it is difficult to comprehensively evaluate unmanned driving tests, this application provides a HIL simulation method for the entire underground mining process. By flexibly constructing virtual test scenarios from real underground scene acquisition, three-dimensional reconstruction to simulation testing, and a full-process closed loop of hardware integration, the test coverage is improved.

[0008] The purpose of this application is achieved through the following technical solutions.

[0009] The present application provides a HIL simulation method for the entire process of underground mining, including: collecting point cloud data of underground scenes, using laser radar to scan underground scenes to obtain point cloud data; using the collected point cloud data to generate a simulation scene in CARLA format; preferably, converting the established three-dimensional scene model and road model into a three-dimensional model file in FBX format supported by CARLA and a road file in Open DRIVE format. The simulation scene in CARLA format generated by Unreal engine compilation and import, building a vehicle model and a laser radar sensor model in the Carla simulator, and collecting point cloud data packets of the simulation scene; according to the collected point cloud data packets of the simulation scene, mapping is performed through the SLAM algorithm to generate a point cloud map in PCD format; a map of the simulation scene is constructed based on the PCD point cloud map; preferably, Gao Xiang's LiDAR SLAM method is used to generate a three-dimensional point cloud map in PCD format.

[0010] Build a hardware-in-the-loop (HIL) simulation platform that includes a host computer, a slave computer, and an actuator; load the map of the constructed simulation scene into the constructed hardware-in-the-loop (HIL) simulation platform, simulate the real well working scene through the simulation platform, verify the unmanned driving algorithm for the well working condition, and verify the computing power, stability, and control effect of the controller on the hardware by connecting to the unmanned driving domain controller and vehicle hardware; the unmanned driving domain controller uses the NVIDIA Drive PX2 autonomous driving computing platform, which has the execution capabilities of unmanned driving algorithms such as perception, positioning, planning, and control. Unmanned driving algorithms include: LiDAR perception algorithm, HDMap positioning algorithm, motion planning algorithm, and trajectory tracking control algorithm, which work together to complete unmanned driving. When verifying the unmanned driving algorithm, establish a test scene similar to the real underground environment in the simulation environment, set interference factors similar to the actual working conditions, and evaluate the adaptability and robustness of the unmanned driving system.

[0011] Among them, CARLA (Car Learning to Act) is an open source autonomous driving car simulator, which is specially used for research in the field of urban autonomous driving. CARLA format refers to a virtual scene description format that conforms to the CARLA simulator specification, including geometric and material attribute information of environmental elements such as roads, buildings, and vegetation. Unreal Engine is a game engine developed by Epic Games, which is widely used in games, virtual reality, movies and other fields. CARLA simulator is developed based on Unreal Engine. Through the compilation tool provided by Unreal Engine, the scene file in CARLA format can be converted into a scene model that can be loaded and run in CARLA simulator. CARLA is a simulator dedicated to autonomous driving research. It provides realistic urban road environment, vehicle dynamics model and multiple sensor models, which can easily build an autonomous driving simulation test system. CARLA simulator supports interfaces with other software such as ROS, which is convenient for integrating unmanned driving algorithms. SLAM (Simultaneous Localization and Mapping) is a common algorithm in the field of robotics and autonomous driving. The SLAM algorithm realizes the positioning and mapping of robots in unknown environments by fusing multi-sensor data. Common SLAM algorithms include EKF-SLAM, graph optimization SLAM, laser SLAM, etc. PCD (Point Cloud Data) is a commonly used point cloud data storage format defined by the PCL (Point Cloud Library) library. PCD files store 3D point coordinates, color and other attribute information in ASCII or binary form. The PCD format facilitates the exchange and use of point cloud data between different software. The host computer refers to the supervisory control layer in the automation control system, which is usually composed of industrial computers, human-machine interfaces, etc. The host computer is responsible for communicating with the lower computer, sending high-level control instructions, receiving and processing feedback information from the lower computer, and realizing the monitoring and scheduling of the entire control system. The lower computer refers to the execution control layer in the automation control system, which is directly connected to the sensor and the actuator, and is responsible for real-time data collection and execution of control instructions. The lower computer usually uses embedded controllers such as PLC, DSP, and single-chip microcomputers, which have the characteristics of real-time and high reliability. The actuator is the terminal device in the control system, which completes specific operation tasks according to the control instructions. For example, the actuator of an unmanned vehicle includes controllers such as throttle, brake, and steering, which realize autonomous driving of the vehicle by controlling the speed, acceleration and direction of the vehicle. HIL (Hardware-in-the-Loop) is a real-time simulation testing technology that integrates some physical hardware into the simulation environment to achieve closed-loop interaction between the simulation model and the physical hardware.The hardware-in-the-loop simulation platform has built a host computer, a slave computer and an actuator. By connecting to the unmanned driving control algorithm and hardware, the performance of the control system can be verified in a laboratory environment, shortening the development and testing cycle and improving test coverage.

[0012] Furthermore, generating a simulation scene in the CARLA format includes: importing the collected point cloud data into the software RoadRunner; performing three-dimensional modeling on the simulation scene according to the morphological characteristics of the imported point cloud data to restore the real underground scene; and building a road network and roads corresponding to the real underground scene according to the established three-dimensional model to generate a simulation scene in the CARLA format.

[0013] Specifically, the collected point cloud data is imported into the 3D modeling software RoadRunner; in the RoadRunner software, the underground scene is 3D reconstructed according to the morphological characteristics of the imported point cloud data, and a 3D model matching the actual underground scene is established to restore the spatial structure and topographic features of the underground tunnel; on the basis of the established 3D model, combined with the width, height, turning radius and other parameters of the underground tunnel, the road editing tool provided by RoadRunner is used to construct a road model matching the actual underground road in the 3D scene, and build a complete underground road network; the simulation environment containing the 3D scene model and the road model constructed in the RoadRunner software is converted into a 3D model file in the FBX format supported by the CARLA simulator through the export tool provided by RoadRunner, and the road model is converted into a road file in the Open DRIVE format supported by the CARLA simulator, so as to generate a complete underground simulation scene in the CARLA format.

[0014] Furthermore, a map of the simulation scene is constructed based on the PCD point cloud map, including: importing the collected simulation scene point cloud data packet into the point cloud processing framework PCL to pre-process the point cloud data; based on the pre-processed point cloud data, using the laser SLAM algorithm to perform inter-frame registration and global optimization of the point cloud data to generate a point cloud map, and saving the point cloud map as a point cloud map in PCD format; post-processing the generated point cloud map in PCD format, downsampling the point cloud map by voxels, reducing the number of point clouds, and improving subsequent processing efficiency; applying a plane extraction algorithm to extract plane features in the point cloud map, and identifying flat areas such as the ground and walls; using a clustering segmentation algorithm to segment the point cloud map into different objects and areas, and distinguishing different objects such as roads and obstacles. elements; according to the plane extraction results, select the plane that meets the road characteristics and extract the road point cloud; through the clustering segmentation results, aggregate the point clouds belonging to the same road to form a complete road point cloud fragment; convert the road point cloud in the point cloud map into geometric elements, and extract the road centerline, boundary line and other features; according to parameters such as road width and curvature, generate a mathematical model that matches the actual road; smooth the road model to eliminate noise and abnormal points to obtain a continuous and smooth road curve; align the road model with the three-dimensional scene model through coordinate transformation and scale scaling; attach semantic information such as road material, markings, and traffic signs to the road model; replace the fused road model with the three-dimensional scene to obtain a simulation scene map containing the real road.

[0015] Among them, voxel is a cubic unit in three-dimensional space. Voxel downsampling refers to dividing the point cloud into several voxels, and only retaining one representative point in each voxel, thereby reducing the amount of point cloud data and improving processing efficiency. Common voxel downsampling methods include voxel grid filtering and random downsampling. Plane extraction refers to identifying plane features such as ground, wall, roof, etc. from point cloud data. Common plane extraction algorithms include RANSAC (Random Sample Consensus), Hough transform, region growing, etc. Through plane extraction, the plane area in the point cloud can be segmented to simplify the scene representation. Cluster segmentation refers to dividing the point cloud data into several non-overlapping subsets (clusters), and the points in each cluster have similar attributes or characteristics. Common point cloud clustering algorithms include K-means, DBSCAN (Density-Based Spatial Clustering of Applications with Noise), Euclidean clustering, etc. Cluster segmentation can separate different objects or regions in the point cloud for subsequent identification and processing. Point cloud features refer to quantitative indicators in point cloud data that can characterize the attributes of objects or scenes, and are often used for point cloud analysis, recognition, and matching. Point cloud features can be divided into local features and global features. Local features describe the local geometric properties of point clouds, such as normal vectors, curvature, roughness, etc.; global features describe the overall shape and topological structure of point clouds, such as shape diameter, topological connectivity, etc. Extracting appropriate point cloud features is the basis for tasks such as point cloud semantic segmentation and object detection. Semantic maps refer to environmental maps that contain scene semantic information. They not only represent the geometric structure of the environment, but also contain high-level semantics such as the category and attributes of objects. For example, for road scenes, semantic maps not only contain the geometric shape of the road, but also mark elements such as lane lines, traffic signs, and roadside buildings. Constructing semantic maps requires associating point cloud data with semantic labels, which is achieved through point cloud semantic segmentation, instance segmentation, and other technologies. Semantic maps can provide richer environmental information for unmanned driving and improve the perception and decision-making capabilities of vehicles.

[0016] Furthermore, a point cloud map is generated, including: using the pre-processed continuous frame point cloud data, performing inter-frame registration through the ICP algorithm, estimating the relative motion between adjacent frames, and obtaining an inter-frame transformation matrix; wherein the ICP algorithm iteratively calculates the optimal transformation matrix by minimizing the distance error between corresponding points; on the basis of the ICP inter-frame registration, setting motion constraints, and optimizing and correcting the inter-frame transformation estimated by the ICP algorithm using the odometer and IMU sensor data; on the basis of inter-frame registration and motion constraint optimization, using a loop detection algorithm, determining whether the unmanned driving has repeatedly reached the same position, and if a loop is detected, adding a closed-loop constraint between the corresponding positions, and globally optimizing and adjusting the entire trajectory through a graph optimization algorithm to obtain a globally consistent point cloud map; saving the optimized point cloud map as a point cloud file in PCD format.

[0017] Among them, ICP algorithm: ICP (Iterative Closest Point) is a commonly used point cloud registration algorithm used to estimate the rigid body transformation (translation + rotation) between two sets of point clouds. The ICP algorithm finds the optimal transformation matrix between the two sets of point clouds through iterative optimization, so that the distance error between the registered point clouds is minimized. The basic steps of the ICP algorithm include: establishing the correspondence between point clouds, estimating the transformation matrix, applying the transformation matrix, and recalculating the correspondence until convergence or reaching the maximum number of iterations. Motion constraints refer to the introduction of additional constraints when estimating inter-frame transformations to improve the accuracy and robustness of the estimation. Common motion constraints include: motion model constraints (such as assuming uniform motion), motion range constraints (such as limiting maximum speed and acceleration), sensor data constraints (such as fusing odometer and IMU data), etc. By introducing motion constraints, outliers in the ICP algorithm estimation can be filtered out and the accuracy of registration can be improved. IMU (Inertial Measurement Unit) is an inertial measurement unit, usually including a three-axis accelerometer and a three-axis gyroscope. IMU sensors can measure the acceleration and angular velocity of an object and are used to estimate the motion state of the object. In unmanned driving, the IMU sensor provides the vehicle's posture information (pitch angle, roll angle, heading angle) and acceleration information, which can be fused with the odometer data to improve the accuracy of vehicle motion estimation. Loop closure detection refers to identifying whether the robot has repeatedly reached a previously visited location during the SLAM process. The loop closure detection algorithm determines whether a loop has occurred by comparing the similarity between the current observation data and the historical data. Commonly used loop closure detection algorithms include FAB-MAP based on visual features, Scan Context based on lidar features, etc. The purpose of loop closure detection is to establish long-distance constraint relationships during the SLAM process, reduce cumulative errors, and achieve globally consistent map construction.

[0018] Loop Closure Constraint refers to adding constraint relationships between corresponding positions based on loop detection, indicating that these positions correspond to the same location. Loop closure constraints are usually expressed in the form of relative pose transformation, that is, relative translation and rotation between loop positions. The introduction of loop closure constraints can effectively reduce the cumulative error in the SLAM process, so that the map remains consistent globally. Graph Optimization is an optimization algorithm used to solve SLAM problems. The SLAM problem is represented as a graph structure, in which nodes represent the robot's pose or the position of landmarks, and edges represent the constraint relationship between nodes (such as odometer measurement, loop constraints, etc.). The graph optimization algorithm optimizes the entire graph by minimizing the errors of all constraints to obtain a globally consistent trajectory and map. Commonly used graph optimization algorithms include G20, Ceres, GTSAM, etc. Graph optimization algorithms can effectively deal with nonlinear factors and noise in SLAM problems and improve the accuracy of positioning and mapping.

[0019] Furthermore, the inter-frame transformation estimated by the ICP algorithm is optimized and corrected, including: obtaining the speed and heading angle information of the unmanned vehicle through the odometer, obtaining the attitude angle information of the unmanned vehicle through the IMU, and time-synchronizing the obtained speed, heading angle information and attitude angle information; pre-integrating the odometer and IMU sensor data after time synchronization to obtain the motion increment between the two frames of point cloud data; using the motion increment as a motion constraint, fusing it with the inter-frame transformation estimated by the ICP algorithm, and constructing an optimization objective function, the optimization objective function including the ICP matching error term and the motion constraint error term; using a nonlinear optimization algorithm to solve the optimization objective function and obtain the optimized inter-frame transformation; using the optimized inter-frame transformation to transform the current frame point cloud, obtain point cloud data in the same coordinate system as the previous frame point cloud, and perform point cloud inter-frame registration; repeating the above steps until all point cloud data frames are processed to obtain point cloud inter-frame registration results optimized by motion constraints.

[0020] Among them, the heading angle (YawAngle) refers to the rotation angle of an object in the horizontal plane, indicating the orientation of the object. In unmanned driving, the heading angle information indicates the angle between the vehicle's driving direction and a reference direction (such as true north). The heading angle information can be measured by the gyroscope in the IMU, or it can be obtained by calculating the steering angle of the vehicle through the odometer. Accurate heading angle information is crucial for the positioning and control of unmanned vehicles. The attitude angle (Attitude Angle) refers to the rotation angle of an object relative to the reference coordinate system, which usually includes three components: pitch angle (Pitch), roll angle (Roll) and heading angle (Yaw). The attitude angle information describes the rotation state of an object in three-dimensional space.

[0021] In unmanned driving, attitude angle information reflects the tilt and orientation state of the vehicle, which can be measured by the accelerometer and gyroscope in the IMU. Attitude angle information can be used to compensate for the impact of the vehicle's tilt on sensor data and improve the accuracy of positioning and mapping. Pre-integration is a technology used to process IMU data. The IMU measurement value is integrated between two time points to obtain the increment of position, velocity and attitude. The purpose of pre-integration is to compress the IMU data into a compact representation and reduce the computational complexity in the optimization process. Pre-integration converts the IMU measurement value into an increment relative to the starting time by assuming that the IMU bias remains unchanged for a short time, and constructs a pre-integration factor for graph optimization or filter update. Pre-integration technology can effectively process IMU data and improve the efficiency and accuracy of state estimation. Nonlinear optimization is a mathematical method for solving nonlinear least squares problems. In SLAM, since the robot's motion and sensor measurements are nonlinear, a nonlinear optimization algorithm is required to estimate the robot's trajectory and map. Commonly used nonlinear optimization algorithms include Gauss-Newton and Levenberg-Marquardt. These algorithms construct and solve a linearized least squares problem in each step in an iterative manner, and continuously update the optimization variables until they converge to the optimal solution. Nonlinear optimization algorithms can effectively deal with nonlinear factors in SLAM problems, such as camera projection models, odometer motion models, etc., to improve the accuracy and robustness of SLAM. In the optimization process, it is usually necessary to reasonably select optimization variables, error functions, and optimization strategies to balance computational efficiency and accuracy.

[0022] Furthermore, a globally consistent point cloud map is obtained, including: according to the motion trajectory obtained by inter-frame registration, using the bag-of-words model algorithm, extracting the feature descriptor of each frame of point cloud data, and constructing a visual dictionary; using the constructed visual dictionary, performing feature matching on each frame of point cloud data, and calculating the similarity score between the current frame and the historical frame; when the similarity score exceeds a set threshold, it is determined that the unmanned vehicle has repeatedly reached the same position and a loop is detected; according to the loop detection result, a closed-loop constraint is established between corresponding positions, and the closed-loop constraint is added to the constructed optimization objective function to construct a closed-loop constraint graph; using the G20 graph optimization library, the constructed closed-loop constraint graph is optimized and solved, and the motion trajectory of the unmanned vehicle is adjusted; according to the optimized motion trajectory, multiple frames of point cloud data are spliced ​​together to obtain a globally consistent point cloud map.

[0023] Among them, the Bag-of-Words (BoW) model is a feature representation method for image retrieval and recognition. The Bag-of-Words model regards image or point cloud data as a set of discrete "visual words", each of which represents a local feature. The basic steps of the Bag-of-Words model algorithm include: feature extraction, dictionary construction, and feature encoding. First, extract local feature descriptors for each frame of data; then, cluster all descriptors into several classes through a clustering algorithm (such as K-means), each class corresponds to a visual word; finally, encode the feature descriptors of each frame of data to obtain a word frequency vector representation. The Bag-of-Words model can convert high-dimensional feature data into a compact vector representation, which is convenient for subsequent similarity calculation and matching. Visual Vocabulary is a key concept in the Bag-of-Words model, which represents a set of representative local feature patterns. The visual vocabulary is obtained by clustering the feature descriptors of a large amount of data, and each cluster center corresponds to a visual word. The size and quality of the visual vocabulary have an important impact on the performance of the Bag-of-Words model. A good visual dictionary should have the following characteristics: good distinction between visual words, able to cover various feature patterns in the data; moderate number of visual words, sufficient expressiveness, but not too redundant. K-means clustering algorithm is usually used to build visual dictionaries, and the final cluster center is obtained through iterative optimization. G20 (General Graph Optimization) is a C++ library for solving graph optimization problems, which is widely used in the fields of SLAM and computer vision. Graph optimization is a method of representing optimization problems as graph structures, where nodes represent optimization variables and edges represent error terms or constraints. G20 provides a set of flexible and efficient optimization tools by abstracting graph optimization problems into a unified interface. G20 supports various types of optimization variables (such as camera pose, landmark position, etc.) and error functions (such as reprojection error, IMU pre-integration error, etc.), and provides a rich selection of solvers (such as Gauss-Newton method, Levenberg-Marquardt method, etc.). Users can easily define their own optimization problems, build optimization graphs, and use G20 to solve them. The modular design and excellent performance of G20 make it one of the mainstream optimization libraries in the SLAM field.

[0024] Furthermore, the Levenberg-Marquardt algorithm is used to solve the optimization objective function. Among them, the Levenberg-Marquardt (LM) algorithm is an optimization algorithm for solving nonlinear least squares problems, which combines the advantages of the Gauss-Newton method and the gradient descent method. In each iteration, the LM algorithm adjusts the optimization step size by introducing a damping factor, and adaptively switches between the Gauss-Newton method and the gradient descent method. When the damping factor is small, the LM algorithm is close to the Gauss-Newton method and has the characteristics of fast convergence; when the damping factor is large, the LM algorithm is close to the gradient descent method and can cross the local minimum. The LM algorithm strikes a balance between convergence speed and robustness by dynamically adjusting the damping factor, and is suitable for various nonlinear optimization problems. In SLAM, the LM algorithm is often used to optimize variables such as camera pose and landmark position, and to estimate the optimal solution by minimizing the reprojection error or other error functions. In each iteration, the LM algorithm calculates an increment based on the current estimated parameter value and the gradient information of the error function to update the parameter estimate. This process is repeated until convergence is reached or the maximum number of iterations is reached.

[0025] Specifically, a nonlinear optimization algorithm is used to solve the optimization objective function and obtain the optimized inter-frame transformation, wherein the nonlinear optimization algorithm adopts the Levenberg-Marquardt algorithm: suppose the variable to be optimized is the Lie algebraic form of the posture transformation ξ=(v,ω), wherein in this application ξ=(v,ω) represents the posture transformation of the unmanned vehicle between two adjacent frames of point cloud data; v represents the translation vector, and ω represents the rotation vector. The optimization objective function is constructed as follows: minimizef(ξ)=∑||pr(Ke ξ Tp)-p'|| 2 +∑||e ξ Tg-Tg'|| 2 , where pr represents the function of projecting 3D points to a 2D pixel coordinate system, and in this application, the process of projecting the collected point cloud data to the camera coordinate system; K is the camera internal parameter matrix, which corresponds to the internal parameter matrix of the lidar sensor model in this application; p is the 3D feature point coordinate, which represents the coordinates of the 3D feature point extracted from the first frame of point cloud data; p' is the corresponding 2D feature point coordinate, which represents the coordinates of the 2D feature point matching p in the second frame of point cloud data; Tg and Tg' are the initial pose transformation and optimized pose transformation between adjacent frames, respectively. Specifically, T is the initial pose transformation obtained by pre-integration of odometer and IMU data, and T' is the pose transformation optimized by the Levenberg-Marquardt algorithm. The Levenberg-Marquardt algorithm is used to iteratively solve the least squares problem: (J T J+λI)Δξ=J Tf(ξ), where J is the Jacobian matrix of the objective function f with respect to ξ. In this application, J contains the partial derivatives of the ICP matching error term and the motion constraint error term with respect to ξ. λ is the damping coefficient, and Δξ is the iterative update amount, which represents the update value of the pose transformation ξ in each iteration.

[0026] According to the pose transformation obtained by the solution, the sparse point cloud is aligned to form a globally consistent dense point cloud map; based on the dense point cloud map, a three-dimensional mesh model of the virtual simulation scene is generated through surface reconstruction and texture mapping. Among them, ξ=(v,ω) represents the variable to be optimized, which represents the Lie algebraic form of the pose transformation. v: translation vector, which represents the translation part in the pose transformation. ω: rotation vector, which represents the rotation part in the pose transformation. pr represents the function of projecting 3D points to the 2D pixel coordinate system. Assuming that the homogeneous coordinates of the 3D point are P=(X,Y,Z,1), the calculation formula of pr is: Where fx and fy are the focal lengths of the camera, cx and cy are the coordinates of the principal points. K: camera intrinsic parameter matrix, describing the internal parameters of the camera. The calculation formula of K is:

[0027] e ξ The Lie algebra ξ is converted into the exponential mapping of the corresponding SE(3) transformation matrix, and the calculation formula is: Where SE(3) represents the special Euclidean group in three-dimensional Euclidean space, that is, the rigid body transformation group including rotation and translation. In visual SLAM, SE(3) is used to represent the position of the camera or robot in three-dimensional space, that is, the combination of rotation and translation; R is the rotation matrix, which is a 3x3 orthogonal matrix representing the rotation transformation in three-dimensional space. The rotation matrix R satisfies R T ×R=I and det(R)=1, where I is the unit matrix and det(R) represents the determinant of R. The rotation matrix R can rotate a vector from one coordinate system to another. t is the translation vector, which is a 3x1 vector representing the translation transformation in three-dimensional space. The translation vector t describes the position change of the origin of the coordinate system. The calculation formulas for R and t are: dot(w) represents the dot product of vector ω and itself. ω is a 3x1 vector, which represents the rotation part in Lie algebra SE(3), that is, the rotation vector. The direction of the rotation vector ω represents the rotation axis, and its length represents the rotation angle. skew(ω) is the antisymmetric matrix of ω, which is used to convert the rotation vector ω into an antisymmetric matrix. The antisymmetric matrix skew(ω) satisfies skew(ω) T =-skew(ω) property, in the exponential mapping of Lie algebra SE(3), skew(ω) is used to construct the rotation matrix R.

[0028] When θ≈0:

[0029] R=I+skew(ω)+0.5×skew(ω) 2

[0030] t=v

[0031] Other cases:

[0032]

[0033] Skew(ω) is the antisymmetric matrix of ω:

[0034]

[0035] p represents the coordinates of the 3D feature point, which represents the coordinates of the 3D feature point extracted in the first frame. p' represents the coordinates of the corresponding 2D feature point, which represents the coordinates of the 2D feature point that matches p in the second frame. T and T' represent the pose transformation matrices between adjacent frames. T represents the initial pose transformation, which is usually obtained by estimation or prediction. T' represents the optimized pose transformation, which is solved by a nonlinear optimization algorithm.

[0036] In visual SLAM, the position of the camera or robot needs to be optimized to minimize the reprojection error. The optimization problem is usually solved using a nonlinear least squares method, such as the Levenberg-Marquardt algorithm. The solution process requires calculating the Jacobian matrix of the objective function (i.e., the reprojection error) to the optimization variable (i.e., the Lie algebra ξ). Specifically, in this application, J represents the Jacobian matrix of the objective function f to ξ, and J represents the Jacobian matrix of the optimized objective function f (including the ICP matching error term and the motion constraint error term) to the optimization variable ξ. The calculation formula of J is:

[0037] in, is the partial derivative of the objective function f with respect to the i-th component of ξ. λ: Damping coefficient, used to control the convergence speed and stability of the Levenberg-Marquardt algorithm. Usually the initial value is set to a small positive number, such as 0.001. Among them, the Jacobian matrix J linearizes the nonlinear optimization problem near the current estimation point and approximates the original problem as a linear least squares problem. Through the Jacobian matrix J and the objective function value, the increment Δξ of the optimization variable ξ can be calculated to update the current estimate.

[0038] Furthermore, a hardware-in-the-loop HIL simulation platform including a host computer, a slave computer and an actuator is built, including: data communication between the host computer and the slave computer via Ethernet, using the multi-machine communication principle of ROS; communication between the slave computer and the actuator via the CAN bus, using the CAN protocol to send corresponding CAN messages for communication.

[0039] Among them, ROS (Robot Operating System) is an open source robot software framework that provides a multi-machine communication mechanism that can realize data exchange and collaborative work between different computers. In ROS multi-machine communication, there is a master node (Master) and multiple slave nodes (Slave). The master node is responsible for managing the communication of the entire ROS system, including node registration, topic publishing and subscription, etc. The slave node is the computer that actually runs the ROS program, and realizes data exchange and collaborative work through communication with the master node. The node in ROS is a process that performs specific tasks and can run on different computers. Each node has a unique name to identify its identity and function. Nodes communicate with each other through mechanisms such as topics, services, and parameters. In multi-machine communication, nodes on different computers can communicate with each other to realize distributed data processing and task execution. Topic is an asynchronous communication mechanism in ROS for transmitting data between nodes. A node can publish messages on a specific topic, and other nodes can subscribe to the topic to receive messages. In multi-machine communication, nodes on different computers can transmit data through topics to realize distributed data sharing. In order to achieve multi-machine communication, it is necessary to configure the network of the computers involved in the communication. Usually, Ethernet connection is used, and a unique IP address is assigned to each computer. During the startup process of ROS, the ROS_MASTER_URI and ROS_HOSTNAME environment variables need to be set to specify the IP address of the master node and the IP address of the local machine to ensure that the network connection can be correctly established between the nodes. In multi-machine communication, ROS messages need to be converted into a format that can be transmitted on the network. This process is called data serialization. ROS uses custom data types and serialization mechanisms to convert messages into binary data streams and transmit them over the network. The receiving end restores the binary data stream to ROS messages through deserialization so that the nodes can process them. Through the above mechanism, ROS realizes multi-machine communication, allowing nodes on different computers to communicate with each other, share data and work together. In the communication between the upper computer and the lower computer, the multi-machine communication principle of ROS can be used, with the upper computer as the master node and the lower computer as the slave node, and data exchange and task collaboration can be achieved through mechanisms such as topics and services. Through reasonable network configuration and node design, an efficient and scalable multi-machine communication system can be built to support complex robotic applications and simulation tasks.

[0040] Furthermore, the host computer acts as a server and the slave computer acts as a client. The simulation platform is started for simulation in the following ways: Start the Unreal engine of the host computer and enter the Carla simulation environment; Start Roscore and run the ROS bridge. If the sensor data corresponding to the sensor model in the simulation scene is received in the ROS topic, it is judged that Carla is started successfully; Run the unmanned driving algorithm on the slave computer for simulation testing; Start the RVIZ tool on the host computer to display the vehicle running trajectory corresponding to the vehicle control signal output by the slave computer and the point cloud data collected by the sensor model; Among them, RVIZ (ROSVisualization) is a three-dimensional visualization tool for displaying various sensor data and robot status information in ROS. Through RVIZ, the robot's motion trajectory, sensor data, map, point cloud, etc. can be viewed in real time, which is convenient for debugging and monitoring the robot's operating status. RVIZ supports a variety of data types and display methods, such as lidar point cloud, camera image, robot model, etc. By subscribing to relevant ROS topics, RVIZ can update the display content in real time and provide an intuitive visualization interface. In the unmanned driving simulation platform, RVIZ can be used to display the vehicle's running trajectory and the data collected by the sensors, making it convenient to observe and analyze the performance and behavior of the unmanned driving algorithm.

[0041] Furthermore, the upper computer installs the Ubuntu system, runs the Unreal engine and the Carla simulation software, receives the vehicle control signal from the lower computer, controls the vehicle model in the simulation scene, and generates sensor simulation data according to the sensor model in the simulation scene, and sends the sensor simulation data to the lower computer through the ROS bridge; after the lower computer receives the sensor simulation data sent by the upper computer through the ROS bridge, it inputs the sensor simulation data into the unmanned driving algorithm, generates the vehicle control signal, sends it to the upper computer through the ROS bridge, and sends the corresponding CAN message to the actuator through the CAN bus; the actuator is the vehicle hardware, which includes the drive system, the braking system and the steering system, and performs the corresponding vehicle hardware action according to the control signal output by the lower computer; among them, Ubuntu is a Linux operating system based on Debian, which is famous for its ease of use, stability and rich software ecology. Ubuntu uses GNOME as the default desktop environment, providing a friendly user interface and complete system management tools. In the field of robotics and unmanned driving, Ubuntu is a commonly used operating system choice, thanks to its good support for ROS and wide hardware compatibility. Ubuntu provides the official installation package and configuration tools of ROS, which can easily build the ROS development environment. At the same time, Ubuntu also supports various sensor drivers and hardware interfaces, which is convenient for integrating different sensors and actuators.

[0042] In the unmanned driving simulation platform, the host computer can run the Ubuntu system and use its rich software resources and ROS ecosystem to build the simulation environment and communicate data. The drive system refers to the vehicle's power system, including components such as the engine, gearbox, and drive shaft, which are responsible for providing power and driving force for the vehicle. In unmanned vehicles, the drive system is usually driven by a motor, and the vehicle's acceleration, deceleration, and constant speed are achieved by controlling the motor's speed and torque. The drive system adjusts the motor's working state according to the speed or acceleration command output by the upper-level control algorithm to achieve the expected vehicle motion effect. The performance and response characteristics of the drive system have an important impact on the power, smoothness, and energy consumption of the unmanned vehicle. In the hardware-in-the-loop simulation platform, the drive system in the actuator performs the corresponding drive action according to the received control signal to simulate the motion behavior of the real vehicle. The braking system is a safety-critical system for the vehicle, which is used to achieve vehicle deceleration, stopping, and stationary holding. Common types of braking systems include hydraulic braking, pneumatic braking, and electronic braking. In unmanned vehicles, the braking system usually adopts electronic braking technology to adjust the braking force by controlling the braking actuator (such as electronic vacuum booster, electronic parking brake, etc.). The braking system quickly and accurately controls the braking force of the wheel according to the braking torque or deceleration command output by the upper control algorithm to achieve safe deceleration and stopping of the vehicle. The performance and stability of the braking system are crucial to the safety of unmanned vehicles. In the hardware-in-the-loop simulation platform, the braking system in the actuator performs the corresponding braking action according to the received control signal to simulate the braking behavior of the real vehicle. The steering system is the steering control system of the vehicle, which is used to achieve the steering and direction control of the vehicle. Common types of steering systems include mechanical steering, hydraulic power steering and electric power steering. In unmanned vehicles, the steering system usually adopts electric power steering technology to adjust the steering angle by controlling the steering actuator (such as electric power steering motor). The steering system accurately controls the steering angle of the wheel according to the steering angle command output by the upper control algorithm to achieve smooth steering and path tracking of the vehicle. The response speed, accuracy and stability of the steering system have an important impact on the maneuverability and path control ability of unmanned vehicles. In the hardware-in-the-loop simulation platform, the steering system in the actuator performs corresponding steering actions according to the received control signals to simulate the steering behavior of a real vehicle.

[0043] Compared with the prior art, the advantages of this application are:

[0044] This application constructs a high-fidelity virtual simulation scene by three-dimensionally reconstructing the real underground environment. On this basis, a variety of test scenarios can be flexibly designed to fully simulate the operating conditions of vehicles in harsh conditions such as cramped, dark, and muddy conditions. Compared with traditional methods, this application is not limited by the actual environment and resources, and can cover more potential extreme conditions, thereby comprehensively evaluating the adaptability and robustness of the unmanned driving system.

[0045] By integrating the unmanned vehicle controller hardware into the simulation environment, this application achieves highly automated hardware-in-the-loop testing, avoiding the tedious preparation process for real vehicle road testing and significantly shortening the testing cycle.

[0046] This application collects real underground environment data for scene reconstruction, ensuring the consistency between the simulation environment and the actual working conditions. At the same time, integrating the vehicle controller hardware into the simulation environment can accurately evaluate the execution effect of the control algorithm. Hardware-in-the-loop simulation not only examines the perception and decision-making algorithms themselves, but also tests the algorithm's operating performance on the hardware platform, obtaining more comprehensive and reliable evaluation results.

[0047] Based on the hardware-in-the-loop simulation system, it is easy to repeatedly debug and compare algorithms such as perception, planning, and control, and quickly evaluate the performance differences of different algorithm solutions, thereby guiding algorithm improvement and parameter tuning. At the same time, the simulation environment can easily introduce various interference factors to examine the system's fault tolerance and failure safety mechanism. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The present application will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0049] Figure 1 is a schematic diagram of the overall structure of a hardware-in-the-loop simulation platform according to some embodiments of the present application;

[0050] Figure 2 is an exemplary flow chart of a well operation simulation scenario based on Carla according to some embodiments of the present application;

[0051] Figure 3 It is a schematic diagram of a road network in a simulation scenario according to some embodiments of the present application;

[0052] Figure 4 It is a schematic diagram of a simulation scenario shown in some embodiments of the present application. DETAILED DESCRIPTION

[0053] The method and system provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0054] Figure 1 It is a schematic diagram of the overall structure of the hardware-in-the-loop simulation platform shown in some embodiments of the present application, the connection between the autonomous driving domain controller (lower computer) and the vehicle hardware (actuator), the bench construction and assembly of the vehicle hardware, and the construction of a corresponding hardware bench according to the mechanical structure and layout of the actual vehicle. The bench needs to include the core actuators of the vehicle, such as the drive system, braking system, steering system, etc. The drive system can use a motor or an engine, and be equipped with corresponding controllers and sensors, such as a motor controller, throttle position sensor, etc. The braking system can use hydraulic braking or electronic braking, and be equipped with pressure sensors, brake actuators, etc. The steering system can use electric power steering or hydraulic power steering, and be equipped with angle sensors, steering actuators, etc. The various subsystems are assembled according to the layout of the actual vehicle to form a complete vehicle hardware actuator.

[0055] CAN bus connection and configuration: The lower computer is connected to the actuator through the CAN bus. It is necessary to select appropriate CAN transceivers and cables. The CAN transceiver needs to match the voltage level standard (such as 3.3V, 5V, etc.) and communication rate of the lower computer and the actuator. The connecting cable needs to have sufficient shielding and anti-interference properties, and the length should not be too long to ensure the communication quality. Configure the CAN bus parameters on the lower computer and the actuator, such as baud rate, frame format, frame ID, etc., to ensure that both parties match. Configure the bus termination resistance, usually 120 ohms, to ensure the signal quality of the bus.

[0056] Control command transmission based on CAN protocol. According to the CAN protocol, the vehicle control command is encapsulated into a CAN message for transmission. The CAN message contains fields such as frame ID, data length, and data segment. Different control commands correspond to different frame IDs and data segment formats. Common control commands include throttle control, brake control, and steering control, which correspond to values ​​within a certain range. The lower computer generates the corresponding CAN message based on the calculation results of the unmanned driving algorithm and sends it to the actuator through the CAN bus. After the CAN interface of the actuator receives the message, it parses the control command and executes the corresponding action, such as throttle opening control, brake pressure control, steering angle control, etc., and the connection between the upper computer and the lower computer.

[0057] The hardware configuration of the host computer needs to meet the minimum configuration requirements of the Carla simulator, such as CPU, memory, graphics card, etc. It is recommended to use a high-performance workstation or gaming desktop, such as Inteli7 / i9, 32G memory, NVIDIARTX series graphics card, etc., install Ubuntu system versions compatible with Carla, such as 16.04, 18.04, etc., and install NVIDIA drivers that match the graphics card to ensure that Carla can run normally. Ethernet connection and configuration, the host computer and the slave computer are directly connected via an Ethernet cable, or connected through a switch. The network cable needs to have sufficient transmission bandwidth. It is recommended to use a Gigabit or higher network cable and network port. Configure network parameters such as IP address and subnet mask on the host computer and the slave computer respectively to ensure that both parties are in the same network segment. The ping command can be used to test the connectivity between the upper and lower computers.

[0058] Based on ROS multi-machine communication, install ROS versions matching Ubuntu on the host and slave computers, such as ROSKinetic, Melodic, etc., run ROSCORE on the host computer as the master node of ROS communication, configure ROS environment variables on the slave computer so that it can connect to ROSCORE of the host computer, the host computer as the host of ROS, and the slave computer as the slave computer, respectively create ROS nodes, subscribe and publish corresponding topics, the host computer sends the sensor data generated by Carla to the slave computer through the topic, receives the control instructions sent by the slave computer through the topic, the slave computer receives the sensor data through the topic, and sends the control instructions to the host computer, and sends them to the hardware actuator through the CAN bus. Through the above hardware construction and connection configuration, a complete hardware-in-the-loop simulation platform is formed. The host computer provides simulation environment and data through the Carla simulator, the slave computer runs the autonomous driving algorithm and generates control instructions, the hardware actuator performs corresponding actions according to the control instructions, and finally presents them in real time in the Carla environment of the host computer, realizing the closed-loop simulation test of the entire autonomous driving system.

[0059] This hardware-in-the-loop testing method can fully verify the feasibility and stability of the autonomous driving algorithm and the control effect of the domain controller on the vehicle before the actual vehicle is put on the road for testing, greatly improving the testing efficiency and reducing the testing cost and risk. At the same time, due to the use of standardized interfaces and protocols, such as CAN bus, ROS, etc., the platform has good versatility and scalability, and can be easily connected to different domain controllers and vehicle models to meet various testing needs.

[0060] Software construction: Ubuntu system environment construction, installation of Ubuntu system version compatible with Carla on the host computer, installation of Python compiler, NVIDIA driver and other dependent environments matching the system, installation and deployment of Carla simulator, cloning or downloading the latest Carla source code from GitHub, compiling Carla source code, automatically downloading city map resources and checking the availability of dependent libraries, installing the ROS bridge (Carla_ROS_Bridge) provided by Carla for interaction with unmanned driving algorithms, construction and deployment of simulation test scenarios, and building a 3D simulation environment matching the real scenario in Carla according to test requirements, and configuring sensors and vehicle models consistent with the real vehicle in the simulation environment.

[0061] The workflow of the hardware-in-the-loop simulation platform is as follows: the host computer runs the Unreal engine and the Carla simulator to generate sensor data and vehicle status data for the simulation environment. Through the ROS bridge, the host computer sends the sensor data to the slave computer, receives the control instructions from the slave computer, and displays the simulation animation and data in real time. The slave computer runs the unmanned driving algorithm, receives the sensor data from the host computer, calculates and generates vehicle control instructions, and the slave computer sends the control instructions to the host computer via Ethernet, and sends the control instructions (CAN messages) to the actuator via the CAN bus. The actuator executes the corresponding vehicle actions such as acceleration, braking, steering, etc. according to the received CAN messages. The execution effect of the vehicle hardware will be fed back to the Carla simulator and presented in real time in the simulation environment of the host computer.

[0062] The sensors provided in Carla include lidar, depth camera, IMU inertial measurement unit, GNSS navigation satellite, etc. In the well working scenario, since there is no GNSS signal underground, underground unmanned driving can only be positioned through SLAM, and requires the support of high-precision maps. When setting up the well working test scenario, the necessary sensors are mainly lidar and IMU. The configuration of sensor position parameters and performance parameters is through importing json format configuration files. The configurable parameters of lidar include installation position, number of lidar lines, detection distance, fov angle, rotation frequency, etc. Taking lidar as an example, its json format configuration file is as follows:

[0063] {"type":"sensor.lidar.ray_cast",

[0064] "id": "lidar",

[0065] "spawn_point": {"x": 0.0, "y": 0.0, "z": 2.4, "roll": 0.0, "pitch": 0.0, "yaw": 0.0},

[0066] "range": 50,

[0067] "channels": 32,

[0068] "points_per_second": 640000,

[0069] "upper_fov": 2.0,

[0070] "lower_fov": -26.8,

[0071] "rotation_frequency": 40,

[0072] "noise_stddev": 0.0},

[0073] The vehicle model in Carla provides a vehicle model for passenger cars. The model of the mining truck in the mine working scene needs to be 3D modeled and rendered and imported based on the real vehicle model. The import method needs to refer to the official documentation of Carla. Similarly, its vehicle dynamics model also needs to be modified and configured accordingly.

[0074] Figure 2 is an exemplary flow chart of building a well working condition simulation scenario based on Carla according to some embodiments of the present application. The steps of building a well working condition simulation scenario based on Carla are as follows: Figure 2 As shown. Collect PCD point cloud data of real scenes: Use a data collection vehicle equipped with a laser radar to collect all-round point cloud data in the target underground environment. The collected point cloud data should cover typical scenes such as underground tunnels, tank tunnels, and chambers, and reflect the geometric structure and detailed features of the scene as completely as possible. The collected point cloud data will be used as the basic data for simulation scene modeling and restoration. Use Road Runner software to import the collected point cloud data: Import the collected PCD format point cloud data into Road Runner software. RoadRunner is a tool software specifically used for road scene modeling and editing, which supports the import and processing of point cloud data.

[0075] Model the simulation scene and restore the real scene in RoadRunner software. Based on the imported point cloud data, perform 3D modeling and restoration of the underground scene in RoadRunner: Import the collected underground environment point cloud data into RoadRunner software. Use the point cloud editing tools provided by RoadRunner to pre-process the imported point cloud data, remove noise, and perform downsampling to reduce the density of point cloud data and improve the efficiency of subsequent modeling. Use the terrain modeling function of Road Runner to generate a 3D terrain model that matches the real underground environment based on the point cloud data. By setting appropriate parameters, control the accuracy and detail of the terrain model to balance modeling efficiency and restoration.

[0076] Through the scene geometry reflected by the point cloud data, a 3D model that is highly consistent with the real scene is established, including tunnels, tank tunnels, chambers, etc.: Based on the generated terrain model, the key components of the underground scene are 3D modeled using RoadRunner's scene modeling tool. For tunnels, use RoadRunner's road modeling tool to draw a 3D model that matches the real tunnel according to the direction and width of the tunnel in the point cloud data. For tank tunnels, use the tank tunnel shape and size reflected by the point cloud data, and use RoadRunner's model library or custom modeling tools to create a 3D model that is highly consistent with the real tank tunnel. For chambers, use RoadRunner's building modeling tool to build a 3D model that conforms to the characteristics of the real chamber according to the location, size and shape of the chamber in the point cloud data. During the modeling process, pay attention to the scene geometry details reflected by the reference point cloud data, such as the corners of the tunnel, the connecting parts of the tank tunnel, the entrance of the chamber, etc., to improve the authenticity of the model.

[0077] Figure 3According to the simulation scene road network schematic diagram shown in some embodiments of the present application, the road network and roads in the simulation environment are built according to the road part in the point cloud data to ensure consistency with the direction and connectivity of the actual underground road: In RoadRunner, the road information reflected by the point cloud data is used to draw a road network topology that matches the real underground road network. Use Road Runner's road editing tool to generate a three-dimensional road model consistent with the geometric features of the real road based on the attributes such as the width, curvature and slope of the road in the point cloud data. For complex structures such as intersections and forks of the road, refer to the actual situation in the point cloud data and use Road Runner's intersection editing function to accurately reproduce the connectivity of the road. In the process of road modeling, pay attention to maintaining consistency with the real underground road network to ensure that the direction, connectivity and detailed features of the road match the actual situation. For some special road signs, such as traffic signs, road signs, etc., you can use RoadRunner's sign library to add a three-dimensional model corresponding to the real scene.

[0078] Figure 4 According to the simulation scene schematic diagram shown in some embodiments of the present application, a scene file in CARLA format is generated, and the scene is loaded in the CARLA simulator for data collection. The scene export function of RoadRunner is used to export the completed underground simulation scene to a CARLA compatible format file: After completing the modeling and editing of the underground simulation scene in the RoadRunner software, select the export function of the software. In the export options, select CARLA as the target format and set the corresponding export parameters, such as file path, coordinate system, etc. After performing the export operation, Road Runner will automatically generate a set of scene files compatible with CARLA. The exported files include a three-dimensional model file in FBX format (.fbx), a road network file in Open DRIVE format (.xodr) and a Road Runner scene data file (.rr data.xml): The FBX file contains a three-dimensional geometric model of the underground simulation scene, including terrain, buildings, roads and other elements. The Open DRIVE file describes the road network information in the simulation scene, including the geometry of the road, lane attributes, traffic signs, etc. The Road Runner scene data file contains other properties and settings of the scene, such as materials, lighting, environmental parameters, etc. These files together constitute a complete CARLA format scene description that can be recognized and loaded by the CARLA simulator.

[0079] Compile and import the CARLA format scene file exported by Road Runner into Unreal Engine through the "make import" command provided by Unreal Engine: Copy the scene file exported by Road Runner to the corresponding directory of the Unreal Engine project. In the command line interface or batch file of Unreal Engine, execute the "make import" command and specify the path of the scene file to be imported. Unreal Engine will automatically parse the scene file and convert it into the data format inside the engine. The compilation and import process converts FBX, Open DRIVE and other files into asset formats that Unreal Engine can recognize and use: FBX files will be converted into Unreal Engine's Static Mesh assets, which represent the 3D geometric models in the scene. Open DRIVE files will be converted into Unreal Engine's Road Network assets, which contain the geometry and topology information of the road. The properties and settings in the Road Runner scene data file will be applied to the corresponding Unreal Engine assets, such as materials, lighting, etc. After the conversion is completed, the simulation scene will be presented in the scene format of Unreal Engine and can be viewed and edited in the engine editor.

[0080] Load the compiled underground simulation scene in the CARLA simulator: Start the CARLA simulator and specify to load the compiled Unreal engine scene. The CARLA simulator will read the scene data and reconstruct a 3D virtual scene corresponding to the real underground environment in the simulation environment. After loading, the underground simulation scene will be presented in the CARLA simulator, and interaction and data collection can be performed.

[0081] Use the previously built unmanned vehicle model and sensor models such as LiDAR to collect data in the simulation scene: In the CARLA simulator, load the pre-built unmanned vehicle model and configure the sensor models such as LiDAR. Use the API provided by CARLA to control the driving of the unmanned vehicle in the simulation scene and the working status of the sensor. During the driving of the unmanned vehicle, the LiDAR sensor model will collect point cloud data of the simulation environment in real time and transmit the data to the data recording module.

[0082] Control the vehicle model to drive in the simulation scene, and record the point cloud data of the simulation environment through the sensor model to generate a point cloud data package: Control the driving path and speed of the unmanned vehicle model in the underground simulation scene by writing a control script or using the automatic driving function provided by CARLA. During the driving process, the lidar sensor model continuously collects point cloud data of the simulation scene and transmits the data to the data recording module. The data recording module packages and stores the received point cloud data and generates a point cloud data package file for subsequent SLAM mapping and other analysis tasks. By adjusting the driving path and acquisition parameters of the unmanned vehicle, full coverage and high-quality point cloud data of the simulation scene can be obtained.

[0083] The SLAM algorithm is used to map the point cloud data of the simulation scene and generate a high-precision map suitable for the unmanned driving algorithm. The SLAM algorithm is used to replay the recorded simulation scene point cloud data packet: the collected point cloud data packet is imported into the data processing module of the SLAM algorithm. The SLAM algorithm estimates the motion trajectory and posture changes of the unmanned vehicle in the simulation scene by analyzing continuous point cloud frame data. By matching point cloud data at different times, the SLAM algorithm can identify repeated structures in the scene and establish associations between data frames. Using optimization algorithms, such as graph optimization or filtering algorithms, the estimated trajectory and map are optimized to reduce cumulative errors and improve the accuracy of mapping.

[0084] The SLAM algorithm is used to achieve three-dimensional reconstruction and mapping of the simulation environment and generate a high-precision point cloud map: In the process of playing back the point cloud data, the SLAM algorithm gradually integrates new data frames to update and expand the coverage of the map. Through point cloud registration and fusion technology, the point cloud data collected from different perspectives and positions are aligned and merged to form a complete three-dimensional point cloud map. The point cloud map generated is denoised and simplified using point cloud filtering and downsampling algorithms to improve the quality and efficiency of the map. After being processed by the SLAM algorithm, the generated point cloud map can represent the three-dimensional structure of the simulation scene in a high-precision and high-density form.

[0085] The resulting point cloud map has a high consistency with the real underground environment and can be used as a reference map for the unmanned driving algorithm: Since the simulation scene is modeled based on the point cloud data of the real underground environment, the point cloud map generated by the SLAM algorithm has a high geometric consistency with the real environment. The point cloud map can accurately reflect the spatial structure and positional relationship of key areas such as underground tunnels, tank tunnels, and chambers, providing a reliable reference for the unmanned driving algorithm. The unmanned driving algorithm can use the point cloud map for positioning, navigation, obstacle avoidance and other tasks to ensure that the behavior in the simulation environment is similar to that in the real environment.

[0086] The high-precision point cloud map obtained by SLAM mapping is further used to create a simulation scene map suitable for the unmanned driving algorithm: the point cloud map generated by SLAM is post-processed to extract environmental information that is important for the unmanned driving algorithm. Through point cloud segmentation and clustering algorithms, key elements such as the ground, walls, and obstacles in the point cloud map are identified. Using feature extraction algorithms such as plane fitting and edge detection, important features such as roads, landmarks, and intersections are extracted from the point cloud map. The extracted environmental information is converted into a map format suitable for the unmanned driving algorithm, such as a vector map or a raster map.

[0087] Post-process the point cloud map to extract key information such as roads and landmarks, and generate vector maps or raster maps that are easy for unmanned driving algorithms to use: For vector maps, the extracted roads, landmarks and other information are represented as geometric primitives, such as line segments, polygons, etc., and the corresponding attribute information is attached. For raster maps, the point cloud map is divided into regular grids, and each grid represents an environmental attribute within a certain range, such as accessibility, height, etc. The generated vector map or raster map is structured and has good compressibility, which facilitates the unmanned driving algorithm to perform rapid query and analysis. According to the needs, the generated map is further edited and improved, such as adding semantic information, road network topology, etc., to meet the specific needs of the unmanned driving algorithm.

[0088] After the scene is built, simulation verification is performed. The host computer acts as the server, starts the Unreal engine and enters the CARLA simulation environment: Open the Unreal engine editor on the host computer and load the previously built underground simulation scene. Through the CARLA plug-in of the Unreal engine, set the scene to CARLA simulation mode and configure relevant parameters, such as server port, sensor settings, etc. Click the "Play" button to start the CARLA simulation environment, run it in the background, and wait for the connection of the lower computer.

[0089] Start Ros core and CARLA-ROS bridge on the host computer: Open the terminal and enter the ros core command to start the ROS master node to provide support for subsequent ROS communication. In another terminal, enter the ros launch carla_ros_bridge carla_ros_bridge.launch command to start the CARLA-ROS bridge. The CARLA-ROS bridge is responsible for converting sensor data, vehicle status and other information in the CARLA simulation environment into ROS topics and publishing them to the ROS network. Through the ROS topic, the driverless algorithm of the lower computer can obtain real-time data of the simulation environment and send control instructions to the simulated vehicle.

[0090] Check the ROS topic to confirm that the CARLA simulation environment has been successfully started: In the terminal of the host computer, enter the rostopiclist command to view the current ROS topic list. Confirm that the topic list contains topics related to the simulation sensor, such as / carla / ego_vehicle / lidar / lidar1 / point_cloud, etc., which represent the topic of the lidar point cloud data. At the same time, check the topics of other sensors, such as camera, GPS, etc., to ensure that the data is published normally. If you see the data topic of the relevant sensor, it means that the CARLA simulation environment has been successfully started and connected to the ROS system.

[0091] The lower computer acts as a client and runs the unmanned driving algorithm for simulation testing: On the lower computer, make sure that the ROS environment and the dependent libraries required for the unmanned driving algorithm have been installed and configured. Copy the code package of the unmanned driving algorithm to the ROS workspace of the lower computer and compile it to generate an executable file. In the terminal of the lower computer, enter the rosrun command to start the main program of the unmanned driving algorithm and start the simulation test. The unmanned driving algorithm obtains sensor data from the upper computer by subscribing to the ROS topic, such as lidar point cloud, camera image, etc. The algorithm performs environmental perception, positioning, path planning and control decisions based on the acquired sensor data, and generates control instructions. The control instructions are sent back to the upper computer through the ROS topic to control the movement of the simulated vehicle in the CARLA environment.

[0092] Start RVIZ on the host computer to display the running trajectory and sensor data of the simulated vehicle in real time: In the terminal of the host computer, enter the ros run rviz rviz command to start the RVIZ visualization tool. In RVIZ, add relevant display plug-ins, such as the "Odometry" plug-in to display the vehicle's motion trajectory, the "PointCloud2" plug-in to display the lidar point cloud data, etc. Optimize the visualization effect by configuring the display parameters of RVIZ, such as coordinate system, color, size, etc. During the simulation process, RVIZ will update and display the vehicle's motion trajectory and sensor data in real time, which is convenient for observing and analyzing the performance of the unmanned driving algorithm. In a specific embodiment of the present application, after adding an obstacle vehicle, the lidar of the unmanned vehicle can accurately detect the position of the obstacle vehicle in front, and generate a subsequent obstacle avoidance path according to the obstacle position.

Claims

1. A HIL simulation method for the entire process of underground mining, comprising: Collect point cloud data of underground scenes; Generate simulation scenes in CARLA format using the collected point cloud data; The simulation scene in CARLA format is compiled and imported into the Unreal engine, the vehicle model and lidar sensor model are built in the Carla simulator, and the point cloud data package of the simulation scene is collected; According to the collected point cloud data packets of the simulation scene, the SLAM algorithm is used to build a map to generate a point cloud map in PCD format; the map of the simulation scene is built according to the PCD point cloud map; Build a hardware-in-the-loop HIL simulation platform including a host computer, a slave computer and actuators; The map of the constructed simulation scene is loaded into the constructed hardware-in-the-loop (HIL) simulation platform. The real well operating scenario is simulated through the simulation platform to verify the unmanned driving algorithm for the well operating condition. The computing power, stability and control effect of the controller on the hardware are verified by connecting to the unmanned driving domain controller and vehicle hardware.

2. The HIL simulation method for the entire underground mining process according to claim 1, characterized in that: Generate simulation scenarios in CARLA format, including: Import the collected point cloud data into the software Road Runner; According to the morphological features of the imported point cloud data, the simulation scene is 3D modeled to restore the real underground scene; Based on the established 3D model, a road network and roads corresponding to the real underground scene are built to generate a simulation scene in CARLA format.

3. The HIL simulation method for the entire underground mining process according to claim 2, characterized in that: Construct a map of the simulation scene based on the PCD point cloud map, including: Import the collected simulation scene point cloud data package into the point cloud processing framework PCL to pre-process the point cloud data; According to the pre-processed point cloud data, the laser SLAM algorithm is used to perform inter-frame registration and global optimization of the point cloud data to generate a point cloud map, which is then saved as a point cloud map in PCD format. Post-process the generated point cloud map in PCD format, including voxel downsampling, plane extraction and cluster segmentation; extract point cloud features containing roads in the point cloud map; Based on the extracted point cloud features, a semantic map of the simulation scene is constructed, and the roads in the point cloud map are converted into geometric elements to generate a map that matches the real environment of the simulation scene.

4. The HIL simulation method for the entire underground mining process according to claim 3 is characterized in that: Generate point cloud map, including: Using the pre-processed continuous frame point cloud data, the ICP algorithm is used to perform inter-frame registration, estimate the relative motion between adjacent frames, and obtain the inter-frame transformation matrix; the ICP algorithm iteratively calculates the optimal transformation matrix by minimizing the distance error between corresponding points; Based on ICP inter-frame registration, motion constraints are set, and the inter-frame transformation estimated by the ICP algorithm is optimized and corrected using odometer and IMU sensor data. Based on inter-frame registration and motion constraint optimization, the loop detection algorithm is used to determine whether the driverless vehicle has repeatedly reached the same location. If a loop is detected, a closed-loop constraint is added between the corresponding locations, and the entire trajectory is globally optimized and adjusted through a graph optimization algorithm to obtain a globally consistent point cloud map. Save the optimized point cloud map as a point cloud file in PCD format.

5. The HIL simulation method for the entire underground mining process according to claim 4, characterized in that: Optimize and correct the inter-frame transformation estimated by the ICP algorithm, including: The speed and heading angle information of the unmanned vehicle is obtained through the odometer, the attitude angle information of the unmanned vehicle is obtained through the IMU, and the obtained speed, heading angle information and attitude angle information are synchronized in time; Pre-integrate the time-synchronized odometer and IMU sensor data to obtain the motion increment between two frames of point cloud data; The motion increment is used as a motion constraint and is fused with the inter-frame transform estimated by the ICP algorithm to construct an optimization objective function, which includes the ICP matching error term and the motion constraint error term. A nonlinear optimization algorithm is used to solve the optimization objective function and obtain the optimized inter-frame transformation; The optimized inter-frame transformation is used to transform the point cloud of the current frame to obtain point cloud data in the same coordinate system as the point cloud of the previous frame, and inter-frame registration of the point cloud is performed; Repeat the above steps until all point cloud data frames are processed to obtain the point cloud frame registration results optimized by motion constraints.

6. The HIL simulation method for the entire underground mining process according to claim 5, characterized in that: Obtain a globally consistent point cloud map, including: According to the motion trajectory obtained by inter-frame registration, the bag-of-words model algorithm is used to extract the feature descriptors of each frame of point cloud data and construct a visual dictionary; Using the constructed visual dictionary, feature matching is performed on each frame of point cloud data to calculate the similarity score between the current frame and the historical frames; When the similarity score exceeds the set threshold, it is judged that the driverless vehicle has repeatedly reached the same position and a loop is detected; According to the loop detection results, a closed-loop constraint is established between corresponding positions, and the closed-loop constraint is added to the constructed optimization objective function to construct a closed-loop constraint graph; Use the G20 graph optimization library to optimize and solve the constructed closed-loop constraint graph and adjust the motion trajectory of the unmanned vehicle; According to the optimized motion trajectory, multiple frames of point cloud data are spliced ​​together to obtain a globally consistent point cloud map.

7. The HIL simulation method for the entire underground mining process according to claim 5, characterized in that: A nonlinear optimization algorithm is used to solve the optimization objective function and obtain the optimized inter-frame transformation, wherein the nonlinear optimization algorithm adopts the Levenberg-Marquardt algorithm.

8. The HIL simulation method for the entire underground mining process according to claim 1, characterized in that: Build a hardware-in-the-loop HIL simulation platform including the host computer, slave computer and actuators, including: The upper computer and the lower computer communicate data via Ethernet using the multi-computer communication principle of ROS; The lower computer and the actuator communicate through the CAN bus by sending corresponding CAN messages using the CAN protocol.

9. The HIL simulation method for the entire underground mining process according to claim 8, characterized in that: Also includes: The upper computer acts as the server and the lower computer acts as the client. The simulation platform is started for simulation in the following ways: Start the Unreal engine on the host computer and enter the Carla simulation environment; Start Roscore and run the ROS bridge. If sensor data corresponding to the sensor model in the simulation scenario is received in the ROS topic, it is determined that Carla has been started successfully. Run the unmanned driving algorithm on the lower computer and conduct simulation tests; Start the RVIZ tool on the host computer to display the vehicle trajectory corresponding to the vehicle control signal output by the slave computer and the point cloud data collected by the sensor model.

10. The HIL simulation method for the entire underground mining process according to claim 8, characterized in that: Also includes: The upper computer is installed with Ubuntu system, runs Unreal engine and Carla simulation software, receives vehicle control signals from the lower computer, controls the vehicle model in the simulation scene, generates sensor simulation data according to the sensor model in the simulation scene, and sends the sensor simulation data to the lower computer through the ROS bridge; After receiving the sensor simulation data sent by the upper computer through the ROS bridge, the lower computer inputs the sensor simulation data into the unmanned driving algorithm, generates a vehicle control signal, sends it to the upper computer through the ROS bridge, and sends the corresponding CAN message to the actuator through the CAN bus; The actuator is the vehicle hardware, which includes a drive system, a brake system and a steering system. The actuator executes corresponding vehicle hardware actions according to the control signal output by the lower computer.

Images