An HIL simulation method for a whole process of underground mining

Abstract

The application discloses a kind of HIL simulation methods of underground mining whole process, it is related to the field of autonomous driving simulation test, including: the point cloud data of underground scene is collected, and CARLA format simulation scene is generated;The generated CARLA format simulation scene is compiled and imported by Unreal engine, vehicle model and laser radar sensor model are built in Carla simulator, and the point cloud data packet of simulation scene is collected;According to the point cloud data packet, mapping is carried out by SLAM algorithm, and the point cloud map of PCD format is generated;According to the PCD point cloud map, the map of simulation scene is constructed;Hardware-in-the-loop HIL simulation platform including host computer, lower computer and actuating mechanism is built;The map of simulation scene constructed is loaded to the hardware-in-the-loop HIL simulation platform constructed, and the real mine working condition scene is simulated through simulation platform, and the mine working condition unmanned driving algorithm is verified.The present application improves the coverage of test by whole process closed loop etc., for the difficulty of comprehensive evaluation unmanned driving test in prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automatic driving simulation testing, in particular to a HIL simulation method for underground mining full process. BACKGROUND

[0002] In the mining industry, the underground operation environment is complex, and there are problems such as high safety hidden danger and high personnel operation intensity. By introducing automatic driving technology, unmanned vehicles are used to realize mining, transportation and other operations, which can effectively improve the underground harsh environment and improve the operation safety and production efficiency.

[0003] Although automatic driving technology has been applied in urban road environment, it still faces many technical challenges when transplanted to underground working conditions. Unlike urban roads, underground tunnels are narrow, have poor lighting conditions, and are difficult to locate, which puts higher requirements on the environmental perception and autonomous decision-making ability of the vehicle. In addition, underground vehicles need to work with tunneling machines, transport machines and other equipment, and the operation process is complex and the safety risk is high.

[0004] In order to verify the feasibility and safety of the underground unmanned system, it is necessary to comprehensively test the system in a simulation environment before actual deployment. However, there is currently a lack of an unmanned simulation test system for underground working conditions. The mainstream commercial simulation software such as Prescan, VTD, etc. mainly faces urban road scenes and is difficult to be directly applied to underground environment. Although Carla is more flexible in scene construction and sensor configuration, it lacks a map and vehicle model that matches the underground environment, and the workload of building a full-process simulation system is large.

[0005] On the other hand, completely software-based simulation testing can verify the effectiveness of the unmanned driving algorithm, but cannot comprehensively evaluate the actual performance of the system integration. Hardware-in-the-loop simulation can include controller hardware in the simulation environment, interact with real sensors and actuators, and obtain a more realistic test effect. However, how to build a hardware-in-the-loop system including vehicle-mounted controllers, sensors, actuators and other components, and realize real-time data exchange with the software simulation environment is a complex system integration problem.

[0006] Therefore, developing an unmanned vehicle hardware-in-the-loop simulation system for underground working conditions, realizing the full-process closed loop from real scene collection, three-dimensional reconstruction to simulation testing, hardware integration, is of great significance to promote the development of mine unmanned technology and ensure system safety. The technical solution proposed in this paper aims to break through the above bottlenecks and provide a low-cost, efficient and reliable unmanned simulation testing method for the mining industry. SUMMARY

[0007] In view of the problem in the prior art that it is difficult to comprehensively evaluate unmanned driving test, the application provides a HIL simulation method for a whole process of underground mining, which improves the coverage of the test by flexibly constructing a virtual test scene through collection from a real underground scene, three-dimensional reconstruction to simulation test, and a whole-process closed loop of hardware integration.

[0008] The purpose of the application is achieved by the following technical solutions.

[0009] The application provides a HIL simulation method for a whole process of underground mining, comprising: collecting point cloud data of an underground scene, and using a laser radar to scan the underground scene to obtain the point cloud data; generating a simulation scene in CARLA format using the collected point cloud data; and preferably, converting the established three-dimensional scene model and road model into a three-dimensional model file in FBX format supported by CARLA and a road file in Open DRIVE format. The generated simulation scene in CARLA format is compiled and imported by using an Unreal engine, a vehicle model and a laser radar sensor model are built in a Carla simulator, and point cloud data packets of the simulation scene are collected; a PCD format point cloud map is generated by mapping according to the collected point cloud data packets of the simulation scene through a SLAM algorithm; a map of the simulation scene is constructed according to the PCD point cloud map; and preferably, a LiDAR SLAM method of Gaoke is used to generate a PCD format three-dimensional point cloud map.

[0010] A hardware-in-the-loop (HIL) simulation platform comprising an upper computer, a lower computer and an actuator is built; the map of the constructed simulation scene is loaded to the built HIL simulation platform, a real mine working condition scene is simulated through the simulation platform, the mine working condition unmanned driving algorithm is verified, and the control effect of the controller on the hardware is verified by connecting the unmanned driving domain controller and the vehicle hardware; the unmanned driving domain controller uses an NVIDIA Drive PX2 automatic driving calculation platform and has the execution ability of unmanned driving algorithms such as perception, positioning, planning and control. The unmanned driving algorithm comprises a laser radar perception algorithm, an HDMap positioning algorithm, a motion planning algorithm and a trajectory tracking control algorithm, and cooperates to complete unmanned driving. When the unmanned driving algorithm is verified, a test scene similar to the real underground environment is established in the simulation environment, interference factors similar to the actual operation working condition are set, and the adaptability and robustness of the unmanned driving system are evaluated.

[0011] CARLA (Car Learning to Act) is an open-source autonomous vehicle simulator specifically designed for urban autonomous driving research. CARLA format refers to the virtual scene description format that conforms to the CARLA simulator specification, including geometric and material attribute information of environmental elements such as roads, buildings, and vegetation. Unreal Engine is a game engine developed by Epic Games, widely used in gaming, virtual reality, film, and other fields. CARLA simulator is based on Unreal Engine. Through the compilation tools provided by Unreal Engine, CARLA format scene files can be converted into scene models that can be loaded and run in CARLA simulator. CARLA is a simulator specifically designed for autonomous driving research. It provides realistic urban road environments, vehicle dynamics models, and various sensor models, making it easy to build autonomous driving simulation test systems. CARLA simulator supports interfaces with other software such as ROS, facilitating the integration of autonomous driving algorithms. SLAM (Simultaneous Localization and Mapping) is a common algorithm in the field of robotics and autonomous driving. SLAM algorithm fuses multi-sensor data to simultaneously realize the localization and mapping of robots in unknown environments. Common SLAM algorithms include EKF-SLAM, graph optimization SLAM, and laser SLAM. PCD (Point Cloud Data) is a commonly used point cloud data storage format defined by the PCL (Point Cloud Library) library. PCD files store 3D point coordinates, color attributes, and other attribute information in ASCII or binary format. PCD format facilitates the exchange and use of point cloud data between different software. The host computer refers to the supervisory control layer in the automation control system, usually composed of industrial computers, human-machine interfaces, etc. The host computer is responsible for communication with the lower computer, sending high-level control instructions, receiving and processing lower computer feedback information, and realizing the monitoring and scheduling of the entire control system. The lower computer refers to the execution control layer in the automation control system, directly connected to sensors and actuators, responsible for real-time data acquisition and control instruction execution. The lower computer usually uses embedded controllers such as PLC, DSP, and single-chip microcomputer, with high real-time and reliability. The actuator is the end device in the control system, which completes specific operation tasks according to control instructions. For example, the actuators of an autonomous vehicle include throttle, brake, and steering controllers, which control the speed, acceleration, and direction of the vehicle to achieve autonomous driving. HIL (Hardware-in-the-Loop) is a real-time simulation testing technology that integrates part of the physical hardware into the simulation environment to realize closed-loop interaction between the simulation model and the physical hardware.The hardware-in-the-loop simulation platform builds the upper computer, the lower computer and the actuator, and through connecting the unmanned control algorithm and hardware, the performance of the control system can be verified in the laboratory environment, the development and test cycle is shortened, and the test coverage is improved.

[0012] Further, a CARLA format simulation scene is generated, including: importing the collected point cloud data into software RoadRunner; according to the morphological characteristics of the imported point cloud data, three-dimensional modeling is performed on the simulation scene to restore the real underground scene; according to the established three-dimensional model, a road network and a road corresponding to the real underground scene are built, and a CARLA format simulation scene is generated.

[0013] Specifically, the collected point cloud data is imported into the three-dimensional modeling software RoadRunner; in the RoadRunner software, according to the morphological characteristics of the imported point cloud data, three-dimensional reconstruction is performed on the underground scene, and a three-dimensional model matched with the real underground scene is established to restore the spatial structure and topographic features of the underground roadway; on the basis of the established three-dimensional model, combined with the width, height, turning radius and other parameters of the underground roadway, a road editing tool provided by RoadRunner is used to construct a road model matched with the real underground road in the three-dimensional scene, and a complete underground road network is built; the simulation environment including the three-dimensional scene model and the road model built in the RoadRunner software is converted into a three-dimensional model file in FBX format supported by the CARLA simulator and a road file in Open DRIVE format supported by the CARLA simulator through an export tool provided by RoadRunner, and a complete CARLA format underground simulation scene is generated.

[0014] Further, a map of the simulation scene is constructed according to the PCD point cloud map, including: importing the collected simulation scene point cloud data packet into a point cloud processing framework PCL, and preprocessing the point cloud data; according to the preprocessed point cloud data, inter-frame registration and global optimization of the point cloud data are performed through a laser SLAM algorithm, a point cloud map is generated, and the point cloud map is saved as a PCD format point cloud map; the generated PCD format point cloud map is post-processed, the point cloud map is voxel down-sampled, the number of point clouds is reduced, and the subsequent processing efficiency is improved; a plane extraction algorithm is applied to extract the plane features in the point cloud map, and identify flat areas such as the ground and the wall; a clustering segmentation algorithm is used to segment the point cloud map into different objects and regions, and distinguish different elements such as roads and obstacles; according to the plane extraction result, a plane conforming to the road feature is selected to extract road point clouds; through the clustering segmentation result, the point clouds belonging to the same road are aggregated together to form a complete road point cloud segment; the road point clouds in the point cloud map are converted into geometric elements to extract features such as road center lines and boundary lines; according to parameters such as road width and curvature, a mathematical model matching the actual road is generated; the road model is smoothed to eliminate noise and abnormal points, and a continuous and smooth road curve is obtained; through coordinate transformation and scale scaling, the road model is aligned with the three-dimensional scene model; semantic information such as road material, marking lines, and traffic signs is attached to the road model; the fused road model is replaced into the three-dimensional scene to obtain a simulation scene map containing a real road.

[0015] Voxel is a cube unit in three-dimensional space. Voxel down-sampling refers to dividing the point cloud into several voxels, and only retaining one representative point in each voxel, thereby reducing the amount of point cloud data and improving processing efficiency. Common voxel down-sampling methods include voxel grid filtering, random down-sampling, etc. Plane extraction refers to identifying planar features such as ground, walls, and roofs from point cloud data. Common plane extraction algorithms include RANSAC (Random Sample Consensus), Hough transform, region growing, etc. Through plane extraction, planar regions in the point cloud can be segmented out, simplifying the scene representation. Clustering segmentation refers to dividing point cloud data into several disjoint subsets (clusters), with points in each cluster having similar attributes or features. Common point cloud clustering algorithms include K-means, DBSCAN (Density-Based Spatial Clustering of Applications with Noise), Euclidean clustering, etc. Clustering segmentation can separate different objects or regions in the point cloud, facilitating subsequent identification and processing. Point cloud features refer to quantitative indicators in point cloud data that can represent object or scene attributes, commonly used for point cloud analysis, identification, and matching. Point cloud features can be divided into local features and global features. Local features describe the local geometric properties of point clouds, such as normal vectors, curvatures, roughness, etc.; global features describe the overall shape and topological structure of point clouds, such as shape diameter, topological connectivity, etc. Extracting appropriate point cloud features is the basis for tasks such as point cloud semantic segmentation and object detection. Semantic map refers to an environmental map that contains scene semantic information, not only representing the geometric structure of the environment, but also containing the class and attributes of objects, etc. For example, for a road scene, the semantic map not only contains the geometric shape of the road, but also labels lane lines, traffic signs, roadside buildings, etc. Building a semantic map requires associating point cloud data with semantic labels, achieved through point cloud semantic segmentation, instance segmentation, etc. Semantic maps can provide more rich environmental information for autonomous driving, improving the perception and decision-making capabilities of vehicles.

[0016] Further, the point cloud map is generated, including: using the pre-processed continuous frame point cloud data, inter-frame registration is performed through an ICP algorithm, relative motion between adjacent frames is estimated, and an inter-frame transformation matrix is obtained; wherein the ICP algorithm iteratively calculates an optimal transformation matrix by minimizing the distance error between corresponding points; on the basis of the ICP inter-frame registration, motion constraints are set, and the inter-frame transformation estimated by the ICP algorithm is optimized and corrected by using odometer and IMU sensor data; on the basis of the inter-frame registration and motion constraint optimization, a loop detection algorithm is used to determine whether the unmanned vehicle has repeatedly arrived at the same position, if a loop is detected, a closed loop constraint is added between the corresponding positions, and the entire trajectory is globally optimized and adjusted through a graph optimization algorithm to obtain a globally consistent point cloud map; and the optimized point cloud map is saved as a point cloud file in PCD format.

[0017] ICP algorithm: ICP (Iterative Closest Point) is a commonly used point cloud registration algorithm, which is used to estimate the rigid body transformation (translation + rotation) between two sets of point clouds. The ICP algorithm finds the best transformation matrix between two sets of point clouds through iterative optimization, so that the distance error between the registered point clouds is minimized. The basic steps of the ICP algorithm include: establishing the correspondence between the point clouds, estimating the transformation matrix, applying the transformation matrix, recalculating the correspondence, until convergence or reaching the maximum number of iterations. Motion constraints refer to introducing additional constraints when estimating inter-frame transformation to improve the accuracy and robustness of the estimation. Common motion constraints include: motion model constraints (such as assuming uniform motion), motion range constraints (such as limiting maximum speed and acceleration), sensor data constraints (such as fusing odometer and IMU data), etc. By introducing motion constraints, abnormal values in the ICP algorithm estimation can be filtered out, improving the accuracy of registration. IMU (Inertial Measurement Unit) is an inertial measurement unit, which usually includes a three-axis accelerometer and a three-axis gyroscope. The IMU sensor can measure the acceleration and angular velocity of the object, and is used to estimate the motion state of the object. In unmanned driving, the IMU sensor provides the vehicle's attitude information (pitch angle, roll angle, heading angle) and acceleration information, which can be fused with the odometer data to improve the accuracy of vehicle motion estimation. Loop detection (Loop Closure Detection) refers to identifying whether the robot has repeatedly arrived at a previously visited location during SLAM. The loop detection algorithm compares the similarity of the current observation data and the historical data to determine whether a loop has occurred. Common loop detection algorithms include FAB-MAP based on visual features, Scan Context based on lidar features, etc. The purpose of loop detection is to establish long-distance constraints during SLAM, reduce cumulative errors, and achieve globally consistent map construction.

[0018] Loop Closure Constraint refers to adding constraint relationships between corresponding positions based on loop detection, indicating that these positions correspond to the same location. Loop closure constraint is usually represented in the form of relative pose transformation, i.e., the relative translation and rotation between loop positions. The introduction of loop closure constraint can effectively reduce the cumulative error in the SLAM process, making the map consistent in the global range. Graph Optimization is an optimization algorithm for solving SLAM problems, which represents the SLAM problem as a graph structure, where nodes represent the poses of the robot or the positions of landmarks, and edges represent the constraint relationships between nodes (such as odometry measurements, loop closure constraints, etc.). Graph optimization algorithms optimize the entire graph by minimizing the errors of all constraints to obtain a globally consistent trajectory and map. Commonly used graph optimization algorithms include G20, Ceres, GTSAM, etc. Graph optimization algorithms can effectively handle nonlinear factors and noise in SLAM problems, improving the accuracy of localization and mapping.

[0019] Further, the inter-frame transformation estimated by the ICP algorithm is optimized and corrected, including: obtaining the speed and heading angle information of the unmanned vehicle through the odometer, obtaining the attitude angle information of the unmanned vehicle through the IMU, and time-synchronizing the obtained speed, heading angle information and attitude angle information; pre-integrating the time-synchronized odometer and IMU sensor data to obtain the motion increment between the two frames of point cloud data; fusing the motion increment as a motion constraint with the inter-frame transformation estimated by the ICP algorithm to construct an optimization objective function, the optimization objective function including an ICP matching error term and a motion constraint error term; using a nonlinear optimization algorithm to solve the optimization objective function to obtain an optimized inter-frame transformation; using the optimized inter-frame transformation to transform the current frame of point cloud to obtain point cloud data in the same coordinate system as the previous frame of point cloud for point cloud inter-frame registration; repeating the above steps until all frames of point cloud data are processed to obtain the motion-constrained optimized point cloud inter-frame registration result.

[0020] Yaw Angle refers to the rotation angle of an object in the horizontal plane, indicating the orientation of the object. In unmanned driving, the yaw angle information represents the included angle between the driving direction of the vehicle and a certain reference direction (such as due north). Yaw angle information can be measured by the gyroscope in the IMU, or the steering angle of the vehicle can be calculated by the odometer. Accurate yaw angle information is crucial for the positioning and control of unmanned vehicles. Attitude Angle refers to the rotation angle of an object relative to a reference coordinate system, usually including three components: Pitch, Roll, and Yaw. Attitude angle information describes the rotation state of the object in three-dimensional space.

[0021] In autonomous driving, the attitude angle information reflects the tilt and orientation state of the vehicle, which can be measured by the accelerometer and gyroscope in the IMU. The attitude angle information can be used to compensate for the influence of the tilt of the vehicle on the sensor data, improving the accuracy of positioning and mapping. Pre-integration is a technique for processing IMU data, which integrates the IMU measurements between two time points to obtain the increments of position, velocity and attitude. The purpose of pre-integration is to compress the IMU data into a compact representation, reducing the computational complexity in the optimization process. Pre-integration converts the IMU measurements into increments relative to the starting time by assuming that the bias of the IMU remains constant in a short time, and constructs a pre-integration factor for the update of the graph optimization or filter. The pre-integration technique can effectively process the IMU data, improving the efficiency and accuracy of state estimation. Nonlinear optimization is a mathematical method for solving nonlinear least squares problems. In SLAM, since the motion of the robot and the sensor measurements are nonlinear, nonlinear optimization algorithms are needed to estimate the trajectory and map of the robot. Common nonlinear optimization algorithms include Gauss-Newton, Levenberg-Marquardt, etc. These algorithms construct and solve a linearized least squares problem at each step through iteration, constantly updating the optimization variables until convergence to the optimal solution. Nonlinear optimization algorithms can effectively handle nonlinear factors in SLAM problems, such as camera projection models, odometry motion models, etc., improving the accuracy and robustness of SLAM. In the optimization process, it is necessary to reasonably select the optimization variables, error functions and optimization strategies to balance the computational efficiency and accuracy.

[0022] Further, obtaining a globally consistent point cloud map includes: according to the motion trajectory obtained by inter-frame registration, using a bag-of-words model algorithm to extract feature descriptors of each frame of point cloud data, and constructing a visual dictionary; using the constructed visual dictionary to perform feature matching on each frame of point cloud data, and calculating the similarity score between the current frame and the historical frame; when the similarity score exceeds a set threshold, it is judged that the autonomous vehicle has repeatedly reached the same position, and a loop is detected; according to the loop detection result, a loop constraint is established between the corresponding positions, the loop constraint is added to the constructed optimization objective function, and a loop constraint graph is constructed; using the G20 graph optimization library to optimize and solve the constructed loop constraint graph, and adjusting the motion trajectory of the autonomous vehicle; according to the optimized motion trajectory, multiple frames of point cloud data are spliced together to obtain a globally consistent point cloud map.

[0023] Among them, the bag-of-words (Bag-of-Words, BoW) is a feature representation method for image retrieval and recognition. The bag-of-words model regards an image or point cloud data as a set of discrete "visual words", each of which represents a local feature. The basic steps of the bag-of-words algorithm include feature extraction, dictionary construction and feature coding. First, extract the local feature descriptor of each frame data; then, cluster all descriptors into several classes by clustering algorithm (such as K-means), each class corresponds to a visual word; finally, encode the feature descriptor of each frame data to get a word frequency vector representation. The bag-of-words model can convert high-dimensional feature data into compact vector representation, which is convenient for subsequent similarity calculation and matching. Visual vocabulary is a key concept in bag-of-words, which represents a set of representative local feature patterns. Visual vocabulary is obtained by clustering a large number of feature descriptors of data, and each cluster center corresponds to a visual word. The size and quality of visual vocabulary have important influence on the performance of bag-of-words. A good visual vocabulary should have the following characteristics: good distinguishability between visual words, which can cover various feature patterns in the data; moderate number of visual words, which should have enough expression ability and cannot be too redundant. K-means clustering algorithm is usually used to construct visual vocabulary, and the final cluster center is obtained by iterative optimization. G20 (General Graph Optimization) is a C++ library for solving graph optimization problems, widely used in SLAM and computer vision fields. Graph optimization is a method of representing optimization problems as graph structures, where nodes represent optimization variables and edges represent error terms or constraints. G20 abstracts graph optimization problems into a unified interface, providing a flexible and efficient optimization tool. G20 supports various types of optimization variables (such as camera pose, landmark position, etc.) and error functions (such as re-projection error, IMU pre-integration error, etc.), and provides rich solver selection (such as Gauss-Newton method, Levenberg-Marquardt method, etc.). Users can easily define their own optimization problems, construct optimization graphs, and use G20 for solving. The modular design and excellent performance of G20 make it one of the mainstream optimization libraries in the field of SLAM.

[0024] Further, the Levenberg-Marquardt algorithm is used to solve the optimization objective function. The Levenberg-Marquardt (LM) algorithm is an optimization algorithm used to solve nonlinear least squares problems, combining the advantages of the Gauss-Newton method and the gradient descent method. In each iteration, the LM algorithm adjusts the optimization step size by introducing a damping factor, and adaptively switches between the Gauss-Newton method and the gradient descent method. When the damping factor is small, the LM algorithm is close to the Gauss-Newton method, with fast convergence characteristics; when the damping factor is large, the LM algorithm is close to the gradient descent method, which can jump over local minima. The LM algorithm balances the convergence speed and robustness by dynamically adjusting the damping factor, and is suitable for various nonlinear optimization problems. In SLAM, the LM algorithm is often used to optimize camera pose, landmark position and other variables, by minimizing the re-projection error or other error functions to estimate the optimal solution. In each iteration, the LM algorithm calculates an increment based on the current estimated parameter value and the gradient information of the error function, which is used to update the parameter estimate. Repeat this process until the convergence condition is met or the maximum number of iterations is reached.

[0025] Specifically, a nonlinear optimization algorithm is used to solve the optimization objective function to obtain the optimized inter-frame transformation, wherein the nonlinear optimization algorithm uses the Levenberg-Marquardt algorithm: Let the variables to be optimized be the Lie algebra form of the pose transformation ξ = (v, ω), wherein in this application, ξ = (v, ω) represents the pose transformation of the unmanned vehicle between adjacent two frames of point cloud data; v represents the translation vector, and ω represents the rotation vector. The optimization objective function is constructed as follows: minimize f(ξ) = ∑||pr(K ξ Tp)-p'|| 2 +∑||e ξ Tg-Tg'|| 2 , wherein pr represents a function of projecting 3D points to a 2D pixel coordinate system, in this application, the process of projecting the collected point cloud data to the camera coordinate system; K is a camera intrinsic matrix, in this application, it corresponds to the internal parameter matrix of the laser radar sensor model; p is a 3D feature point coordinate, representing the coordinate of the 3D feature point extracted in the first frame of point cloud data; p' is the corresponding 2D feature point coordinate, representing the coordinate of the 2D feature point matched with p in the second frame of point cloud data; Tg and Tg' are the initial pose transformation and the optimized pose transformation between adjacent frames, respectively. Specifically, T is the initial pose transformation obtained by pre-integration of the odometer and IMU data, and T' is the pose transformation optimized by the Levenberg-Marquardt algorithm. The Levenberg-Marquardt algorithm is used to iteratively solve the least squares problem: (J T J+λI)Δξ=J Tf(ξ), where J is the Jacobian matrix of the objective function f with respect to ξ, in this application, J contains the partial derivatives of the ICP matching error term and the motion constraint error term with respect to ξ. λ is the damping coefficient, Δξ is the iterative update, which represents the updated value of the pose transformation ξ in each iteration.

[0026] According to the solved pose transformation, the sparse point cloud is registered to form a globally consistent dense point cloud map; based on the dense point cloud map, a three-dimensional mesh model of the virtual simulation scene is generated through surface reconstruction and texture mapping. Wherein, ξ = (v, ω) represents the optimization variable, which represents the Lie algebra form of the pose transformation. v: translation vector, represents the translation part of the pose transformation. ω: rotation vector, represents the rotation part of the pose transformation. pr represents the function of projecting 3D points to 2D pixel coordinate system. Assuming that the homogeneous coordinates of the 3D point are P = (X, Y, Z, 1), the calculation formula of pr is: Where fx and fy are the focal lengths of the camera, and cx and cy are the principal point coordinates. K: camera intrinsic matrix, describes the internal parameters of the camera. The calculation formula of K is:

[0027] e ξ Convert the Lie algebra ξ into the exponential mapping of the corresponding SE(3) transformation matrix, and the calculation formula is: Where SE(3) represents the special Euclidean group in three-dimensional Euclidean space, that is, the rigid transformation group containing rotation and translation, in visual SLAM, SE(3) is used to represent the pose of the camera or robot in three-dimensional space, that is, the combination of rotation and translation; R is a rotation matrix, which is a 3x3 orthogonal matrix, representing a rotation transformation in three-dimensional space, the rotation matrix R satisfies the properties of R T ×R = I and det(R) = 1, where I is the identity matrix and det(R) represents the determinant of R. The rotation matrix R can rotate a vector from one coordinate system to another. t is a translation vector, which is a 3x1 vector, representing a translation transformation in three-dimensional space. The translation vector t describes the position change of the coordinate system origin. The calculation formula of R and t is: dot(w) represents the dot product of the vector ω with itself. Where ω is a 3x1 vector, representing the rotation part in the Lie algebra SE(3), that is, the rotation vector. The direction of the rotation vector ω represents the rotation axis, and its length represents the rotation angle. skew(ω) is the skew-symmetric matrix of ω, which is used to convert the rotation vector ω into a skew-symmetric matrix. The skew-symmetric matrix skew(ω) satisfies the property of skew(ω) T = -skew(ω), which is used to construct the rotation matrix R in the exponential mapping of the Lie algebra SE(3).

[0028] When θ ≈ 0:

[0029] R = I + skew(ω) + 0.5 x skew(ω) 2

[0030] t = v

[0031] Otherwise:

[0032]

[0033] skew(ω) is the anti-symmetric matrix of ω:

[0034]

[0035] p represents 3D feature point coordinates, represents the coordinates of the 3D feature points extracted in the first frame. p' represents the corresponding 2D feature point coordinates, represents the coordinates of the 2D feature points matched with p in the second frame. T and T' represent the pose transformation matrix between adjacent frames. T represents the initial pose transformation, which is usually obtained by estimation or prediction. T' represents the optimized pose transformation, which is obtained by solving the nonlinear optimization algorithm.

[0036] In visual SLAM, the pose of the camera or robot needs to be optimized to minimize the reprojection error. The optimization problem is usually solved using a nonlinear least squares method, such as the Levenberg-Marquardt algorithm. The solution process requires calculating the Jacobian matrix of the objective function (i.e., the reprojection error) with respect to the optimization variables (i.e., the Lie algebra ξ). Specifically, J in this application represents the Jacobian matrix of the objective function f with respect to ξ, and J represents the Jacobian matrix of the optimization objective function f (including the ICP matching error term and the motion constraint error term) with respect to the optimization variable ξ. The calculation formula of J is:

[0037] where, is the partial derivative of the i-th component of the objective function f with respect to ξ. λ: damping coefficient, used to control the convergence speed and stability of the Levenberg-Marquardt algorithm. The initial value is usually set to a small positive number, such as 0.001. Where, the Jacobian matrix J linearizes the nonlinear optimization problem near the current estimate point, and approximates the original problem as a linear least squares problem. Through the Jacobian matrix J and the objective function value, the increment Δξ of the optimization variable ξ can be calculated, which is used to update the current estimate.

[0038] Further, a hardware-in-the-loop (HIL) simulation platform is built, which includes a host computer, a lower computer, and an actuator. The host computer and the lower computer communicate through Ethernet using the multi-machine communication principle of ROS. The lower computer and the actuator communicate through CAN bus using CAN protocol to send corresponding CAN messages.

[0039] ROS (Robot Operating System) is an open-source software framework for robots that provides mechanisms for multi-computer communication, enabling data exchange and collaboration between different computers. In ROS multi-computer communication, there is a master node and multiple slave nodes. The master node is responsible for managing the communication of the entire ROS system, including node registration, topic publishing and subscription, etc. The slave node is the computer that actually runs the ROS program and exchanges data and collaborates with the master node through communication. Nodes in ROS are processes that perform specific tasks and can run on different computers. Each node has a unique name to identify its identity and function. Nodes communicate with each other through topics, services, parameters, etc. In multi-computer communication, nodes on different computers can communicate with each other to achieve distributed data processing and task execution. Topics are an asynchronous communication mechanism in ROS used to transmit data between nodes. A node can publish messages on a specific topic, and other nodes can subscribe to the topic to receive messages. In multi-computer communication, nodes on different computers can transmit data through topics to achieve distributed data sharing. To achieve multi-computer communication, the computers involved in communication need to be networked. Usually, Ethernet is used, and each computer is assigned a unique IP address. During the ROS startup process, the ROS_MASTER_URI and ROS_HOSTNAME environment variables need to be set to specify the IP address of the master node and the IP address of the local machine to ensure that the nodes can correctly establish network connections. In multi-computer communication, ROS messages need to be converted into a format that can be transmitted over the network, a process called data serialization. ROS uses custom data types and serialization mechanisms to convert messages into binary data streams for transmission over the network. The receiving end deserializes the binary data stream back into ROS messages for processing by the nodes. Through the above mechanisms, ROS realizes multi-computer communication, enabling nodes on different computers to communicate with each other, share data and collaborate. In the communication between the host computer and the slave computer, the multi-computer communication principle of ROS can be used, with the host computer as the master node and the slave computer as the slave node, to achieve data exchange and task collaboration through topics, services, etc. Through reasonable network configuration and node design, an efficient and scalable multi-computer communication system can be built to support complex robot applications and simulation tasks.

[0040] Further, the host computer acts as a server and the lower computer acts as a client. The simulation platform is started for simulation in the following way: the Unreal engine of the host computer is started, and the Carla simulation environment is entered; Roscore is started, and the ROS bridge is run. If sensor data corresponding to the sensor model in the simulation scene is received in the ROS topic, it is judged that the Carla is started successfully. The unmanned algorithm is run on the lower computer for simulation test. The RVIZ tool is started on the host computer, which is used to show the vehicle running track corresponding to the vehicle control signal output by the lower computer and the point cloud data collected by the sensor model. RVIZ (ROS Visualization) is a three-dimensional visualization tool used to display various sensor data and robot state information in ROS. Through RVIZ, the motion track, sensor data, map, and point cloud of the robot can be viewed in real time, which facilitates the debugging and monitoring of the running state of the robot. RVIZ supports multiple data types and display modes, such as lidar point cloud, camera image, and robot model. By subscribing to relevant ROS topics, RVIZ can update the display content in real time and provide an intuitive visualization interface. In the unmanned simulation platform, RVIZ can be used to show the running track of the vehicle and the data collected by the sensor, which facilitates the observation and analysis of the performance and behavior of the unmanned algorithm.

[0041] Further, the host computer is installed with an ubuntu system, and the Unreal engine and Carla simulation software are run. The vehicle control signal of the lower computer is received to control the vehicle model in the simulation scene. Sensor simulation data is generated according to the sensor model in the simulation scene, and the sensor simulation data is sent to the lower computer through the ROS bridge. After the lower computer receives the sensor simulation data sent by the host computer through the ROS bridge, the sensor simulation data is input into the unmanned algorithm to generate vehicle control signals, which are sent to the host computer through the ROS bridge and corresponding CAN messages are sent to the actuator through the CAN bus. The actuator is the vehicle hardware, which includes the driving system, the braking system, and the steering system. According to the received control signal output by the lower computer, the corresponding vehicle hardware action is performed. Ubuntu is a Linux operating system based on Debian, known for its ease of use, stability, and rich software ecosystem. Ubuntu uses GNOME as the default desktop environment, providing a friendly user interface and perfect system management tools. In the field of robots and unmanned vehicles, Ubuntu is a commonly used operating system choice, thanks to its good support for ROS and wide hardware compatibility. Ubuntu provides official installation packages and configuration tools for ROS, making it easy to build a ROS development environment. At the same time, Ubuntu also supports various sensor drivers and hardware interfaces, facilitating the integration of different sensors and actuators.

[0042] In the unmanned simulation platform, the host computer can run the Ubuntu system, and utilize its rich software resources and ROS ecosystem to realize the construction of the simulation environment and data communication. The drive system refers to the power system of the vehicle, including engine, gearbox, transmission shaft and other components, responsible for providing power and driving force for the vehicle. In the unmanned vehicle, the drive system usually adopts motor drive, by controlling the speed and torque of the motor, to realize the acceleration, deceleration and constant speed driving of the vehicle. The drive system adjusts the working state of the motor according to the speed or acceleration instruction output by the upper control algorithm, to achieve the expected vehicle motion effect. The performance and response characteristics of the drive system have important influence on the power, smoothness and energy consumption of the unmanned vehicle. In the hardware-in-the-loop simulation platform, the drive system in the actuator executes the corresponding driving action according to the received control signal, to simulate the motion behavior of the real vehicle. The brake system is a safety-critical system of the vehicle, used to realize the deceleration, stopping and stationary holding of the vehicle. Common brake system types include hydraulic brake, pneumatic brake and electronic brake, etc. In the unmanned vehicle, the brake system usually adopts electronic brake technology, by controlling the brake actuator (such as electronic vacuum booster, electronic parking brake, etc.) to realize the adjustment of brake force. The brake system quickly and accurately controls the brake force of the wheel according to the brake torque or deceleration instruction output by the upper control algorithm, to realize the safe deceleration and stopping of the vehicle. The performance and stability of the brake system are crucial to the safety of the unmanned vehicle. In the hardware-in-the-loop simulation platform, the brake system in the actuator executes the corresponding brake action according to the received control signal, to simulate the braking behavior of the real vehicle. The steering system is the steering control system of the vehicle, used to realize the steering and direction control of the vehicle. Common steering system types include mechanical steering, hydraulic power steering and electric power steering, etc. In the unmanned vehicle, the steering system usually adopts electric power steering technology, by controlling the steering actuator (such as electric power steering motor) to realize the adjustment of steering angle. The steering system accurately controls the steering angle of the wheel according to the steering angle instruction output by the upper control algorithm, to realize the smooth steering and path tracking of the vehicle. The response speed, accuracy and stability of the steering system have important influence on the maneuverability and path control ability of the unmanned vehicle. In the hardware-in-the-loop simulation platform, the steering system in the actuator executes the corresponding steering action according to the received control signal, to simulate the steering behavior of the real vehicle.

[0043] Compared with the prior art, the application has the advantages that:

[0044] The application reconstructs a three-dimensional real downhole environment to build a high-fidelity virtual simulation scene. On this basis, a variety of test scenarios can be flexibly designed to comprehensively simulate the running conditions of vehicles in narrow, dark, muddy and other harsh conditions. Compared with traditional methods, the application is not limited by the actual environment and resources, and can cover more potential extreme conditions, thereby comprehensively evaluating the adaptability and robustness of the unmanned driving system.

[0045] By integrating the unmanned vehicle controller hardware into the simulation environment, the application realizes highly automated hardware-in-the-loop testing. The cumbersome real vehicle road test preparation process is avoided, and the test cycle is greatly shortened.

[0046] The application collects real downhole environment data for scene reconstruction to ensure the consistency of the simulation environment and the actual working conditions. At the same time, the vehicle-mounted controller hardware is integrated into the simulation environment to accurately evaluate the execution effect of the control algorithm. Hardware-in-the-loop simulation not only examines the perception and decision-making algorithm itself, but also tests the running performance of the algorithm on the hardware platform, obtaining more comprehensive and reliable evaluation results.

[0047] Based on the hardware-in-the-loop simulation system, the perception, planning, control and other algorithms can be repeatedly debugged and compared and tested, the performance differences of different algorithm schemes can be quickly evaluated, and the algorithm improvement and parameter optimization can be guided. At the same time, the simulation environment can easily introduce various interference factors to examine the fault tolerance and failure safety mechanism of the system. BRIEF DESCRIPTION OF DRAWINGS

[0048] The application will be further described in the form of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting, and in these embodiments, the same numbers represent the same structures, wherein:

[0049] Figure 1 is a general structure schematic diagram of a hardware-in-the-loop simulation platform according to some embodiments of the application;

[0050] Figure 2 is an exemplary flowchart for building a mine working condition simulation scene based on Carla according to some embodiments of the application;

[0051] Figure 3 is a simulation scene road network schematic diagram according to some embodiments of the application;

[0052] Figure 4 is a simulation scene schematic diagram according to some embodiments of the application. DETAILED DESCRIPTION

[0053] The method and system provided by the embodiments of the application will be described in detail below with reference to the accompanying drawings.

[0054] Figure 1 is a schematic diagram of the overall structure of a hardware-in-the-loop simulation platform according to some embodiments of the present application, the connection of the automatic driving domain controller (lower machine) with the vehicle hardware (actuator), the bench building and assembly of the vehicle hardware, according to the mechanical structure and arrangement of the real vehicle, the corresponding hardware bench is built, the bench needs to contain the core actuator of the vehicle, such as the driving system, the braking system, the steering system, etc., the driving system can adopt motor or engine, and is equipped with corresponding controller and sensor, such as motor controller, throttle position sensor, etc., the braking system can adopt hydraulic braking or electronic braking, and is equipped with pressure sensor, brake actuator, etc., the steering system can adopt electric power steering or hydraulic power steering, and is equipped with angle sensor, steering actuator, etc., each subsystem is assembled according to the arrangement of the real vehicle to form a complete vehicle hardware actuator.

[0055] CAN bus connection and configuration, the lower machine is connected with the actuator through the CAN bus, a suitable CAN transceiver and cable need to be selected, the CAN transceiver needs to match the level standard (such as 3.3V, 5V, etc.) and communication rate of the lower machine and the actuator, the connection cable needs to have sufficient shielding and anti-interference performance, and the length should not be too long to ensure the communication quality, the parameters of the CAN bus, such as baud rate, frame format, frame ID, etc., are configured on the lower machine and the actuator respectively to ensure matching, the termination resistance of the bus is configured, which is usually 120 ohms, to ensure the signal quality of the bus.

[0056] Control instruction transmission based on CAN protocol, according to the CAN protocol, the control instructions of the vehicle are packaged into CAN messages for transmission, the CAN message contains frame ID, data length, data segment and other fields, different control instructions correspond to different frame IDs and data segment formats, common control instructions include throttle control, brake control, steering control, etc., which correspond to a certain range of values respectively. The lower machine generates corresponding CAN messages according to the calculation results of the unmanned driving algorithm, and sends them to the actuator through the CAN bus. The CAN interface of the actuator receives the message and parses the control instruction, and performs corresponding actions, such as throttle opening control, brake pressure control, steering angle control, etc.

[0057] The hardware configuration of the host computer needs to meet the minimum configuration requirements of Carla simulator, such as CPU, memory, graphics card, etc. It is recommended to use a high-performance workstation or gaming desktop, such as Intel i7 / i9, 32G memory, NVIDIA RTX series graphics card, etc. Install the Ubuntu system version compatible with Carla, such as 16.04, 18.04, etc. Install the NVIDIA driver that matches the graphics card to ensure that Carla can run normally. Ethernet connection and configuration, the host computer and the slave computer are connected directly through Ethernet cable, or connected through switch, the network cable needs to have enough transmission bandwidth, it is recommended to use gigabit and above network cable and network port, configure IP address, subnet mask and other network parameters on the host computer and the slave computer respectively, to ensure that both parties are in the same network segment, the ping command can be used to test the connectivity between the host computer and the slave computer.

[0058] Multi-machine communication based on ROS, install the ROS version matched with Ubuntu on the host computer and the slave computer, such as ROS Kinetic, Melodic, etc. Run ROS Core on the host computer as the master node of ROS communication, configure the ROS environment variable on the slave computer to enable it to connect to the ROS Core of the host computer. The host computer serves as the master and the slave computer as the slave, respectively create ROS nodes, subscribe and publish corresponding topics. The host computer sends sensor data generated by Carla to the slave computer through the topic, receives control instructions sent by the slave computer through the topic, and the slave computer receives sensor data through the topic and sends control instructions to the host computer, and sends them to the hardware actuator through the CAN bus. Through the above hardware setup and connection configuration, a complete hardware-in-the-loop simulation platform is formed. The host computer provides a simulation environment and data through Carla simulator, the slave computer runs the automatic driving algorithm and generates control instructions, the hardware actuator executes corresponding actions according to the control instructions, and finally the real-time presentation is realized in the Carla environment of the host computer, realizing the closed-loop simulation test of the entire automatic driving system.

[0059] This hardware-in-the-loop test method can fully verify the feasibility and stability of the automatic driving algorithm and the control effect of the domain controller on the vehicle before the real vehicle road test, greatly improving the test efficiency, reducing the test cost and risk. At the same time, due to the use of standardized interfaces and protocols such as CAN bus, ROS, etc., the platform has good universality and expandability, and can easily access different domain controllers and vehicle models, adapting to various test requirements.

[0060] Software part building: Ubuntu system environment building, installing Ubuntu system version compatible with Carla on the host computer, installing Python compiler, NVIDIA driver and other dependent environments matching the system, installing Carla simulator, cloning or downloading the latest source code of Carla from github, compiling Carla source code, automatically downloading city map resources and checking the availability of dependent libraries, installing Carla provided ROS bridge (Carla_ROS_Bridge) for interaction with unmanned algorithm, building and deploying simulation test scene, according to test requirements, building three-dimensional simulation environment matching real scene in Carla, configuring sensors and vehicle models consistent with real vehicle in simulation environment.

[0061] Hardware-in-the-loop simulation platform workflow: The host computer runs Unreal Engine and Carla simulator to generate sensor data and vehicle state data of the simulation environment. Through the ROS bridge, the host computer sends sensor data to the lower computer, receives control instructions from the lower computer, and displays simulation animation and data in real time. The lower computer runs the unmanned algorithm, receives sensor data from the host computer, calculates vehicle control instructions, and sends the control instructions to the host computer through Ethernet. The control instructions (CAN messages) are sent to the actuator through the CAN bus. The actuator executes corresponding vehicle actions such as acceleration, braking, steering, etc. according to the received CAN messages. The execution effect of the vehicle hardware is fed back to the Carla simulator, which presents the simulation environment in real time on the host computer.

[0062] The sensors provided in Carla include lidar, depth camera, IMU inertial measurement unit, GNSS navigation satellite, etc. In underground working conditions, since there is no GNSS signal underground, underground unmanned driving can only be positioned by SLAM, and high-precision maps are required. When building the underground working condition test scene, the main sensors required are lidar and IMU. The configuration of sensor position parameters and performance parameters is done by importing a json format configuration file. The configurable parameters of lidar include installation position, lidar line number, detection distance, fov angle, rotation frequency, etc. Taking lidar as an example, the json format configuration file is as follows:

[0063] {"type":"sensor.lidar.ray_cast",

[0064] "id":"lidar",

[0065] "spawn_point":{"x":0.0,"y":0.0,"z":2.4,"roll":0.0,"pitch":0.0,"yaw":0.0},

[0066] "range": 50,

[0067] "channels": 32,

[0068] "points_per_second": 640000,

[0069] "upper_fov": 2.0,

[0070] "lower_fov": -26.8,

[0071] "rotation_frequency": 40,

[0072] "noise_stddev": 0.0},

[0073] The vehicle model in Carla provides a vehicle model of a passenger car. The model of the mine truck in the mine working condition scene needs to be modeled and rendered according to the real vehicle model. The import method needs to refer to the official document of Carla, and the vehicle dynamics model also needs to be modified and configured accordingly.

[0074] Figure 2 is an example flowchart for building a mine working condition simulation scene based on Carla according to some embodiments of the present application. The steps of building a mine working condition simulation scene based on Carla are as shown in Figure 2 Collecting pcd point cloud data of the real scene: using a data collection vehicle equipped with a laser radar, collecting omnidirectional point cloud data in the target underground environment. The collected point cloud data should cover typical scenes such as underground tunnels, tankways, and chambers, and as completely as possible reflect the geometric structure and detailed features of the scene. The point cloud data obtained will be the basic data for simulation scene modeling and restoration. Importing the collected point cloud data into Road Runner software: importing the collected pcd format point cloud data into Road Runner software. Road Runner is a tool software specially used for road scene modeling and editing, supporting the import and processing of point cloud data.

[0075] Modeling and restoring the simulation scene in RoadRunner software, according to the imported point cloud data, the underground scene is modeled and restored in RoadRunner: the collected underground environment point cloud data is imported into RoadRunner software. Using the point cloud editing tool provided by RoadRunner, the imported point cloud data is preprocessed, noise is removed, and downsampling is performed to reduce the density of point cloud data and improve the efficiency of subsequent modeling. Using the terrain modeling function of RoadRunner, a three-dimensional terrain model matching the real underground environment is generated according to the point cloud data. By setting appropriate parameters, the accuracy and detail level of the terrain model are controlled to balance the modeling efficiency and restoration degree.

[0076] The scene geometry reflected by the point cloud data is established, and a three-dimensional model highly consistent with the real scene is established, including roadway, gallery, chamber, etc.: on the basis of the generated terrain model, the scene modeling tool of RoadRunner is used to model the key components of the underground scene. For the roadway, the roadway modeling tool of RoadRunner is used to draw a three-dimensional model matching the real roadway according to the direction and width of the roadway in the point cloud data. For the gallery, the shape and size of the gallery reflected by the point cloud data are used to create a three-dimensional model highly consistent with the real gallery using the model library or custom modeling tool of RoadRunner. For the chamber, according to the position, size and shape of the chamber in the point cloud data, the building modeling tool of RoadRunner is used to build a three-dimensional model conforming to the characteristics of the real chamber. In the modeling process, pay attention to the scene geometric details reflected by the point cloud data, such as the corners of the roadway, the connection parts of the gallery, the entrances of the chamber, etc., to improve the authenticity of the model.

[0077] Figure 3is a simulation scene road network schematic diagram according to some embodiments of the application, according to the road part in the point cloud data, build the road network and road in the simulation environment, ensure the consistency with the actual underground road direction and connectivity: in RoadRunner, use the road information reflected by the point cloud data to draw the road network topology structure matched with the real underground road network. Using the road editing tool of Road Runner, according to the width, curvature and slope of the road in the point cloud data, generate three-dimensional road model consistent with the geometric characteristics of the real road. For the complex structure of the road intersection, branch road and other complex structure, refer to the actual situation in the point cloud data, use the intersection editing function of Road Runner, accurately reproduce the connectivity of the road. In the process of road modeling, pay attention to keep consistent with the real underground road network, ensure that the direction, connectivity and detail characteristics of the road match the actual situation. For some special road signs, such as traffic signs, road signs, etc., the signboard library of Road Runner can be used to add three-dimensional models corresponding to the real scene.

[0078] Figure 4 is a simulation scene schematic diagram according to some embodiments of the application, generate CARLA format scene file, and load the scene in CARLA simulator for data acquisition, use the scene export function of Road Runner to export the completed underground simulation scene to CARLA compatible format file: after completing the modeling and editing of the underground simulation scene in RoadRunner software, select the export function of the software. In the export options, select CARLA as the target format, set the corresponding export parameters, such as file path, coordinate system, etc. Execute the export operation, Road Runner will automatically generate a set of CARLA compatible scene files. The exported files include FBX format three-dimensional model file (.fbx), Open DRIVE format road network file (.xodr) and Road Runner scene data file (.rr data.xml): FBX file contains the three-dimensional geometric model of underground simulation scene, including terrain, building, road and other elements. Open DRIVE file describes the road network information in the simulation scene, including the geometric shape of the road, lane attribute, traffic sign, etc. Road Runner scene data file contains other attributes and settings of the scene, such as material, lighting, environment parameters, etc. These files together constitute a complete CARLA format scene description, which can be recognized and loaded by CARLA simulator.

[0079] Compile and import the CARLA format scene file exported by Road Runner into Unreal Engine using the "make import" instruction provided by Unreal Engine: Copy the scene file exported by Road Runner to the corresponding directory of the Unreal Engine project. In the command line interface or batch file of Unreal Engine, execute the "make import" instruction and specify the path of the scene file to be imported. Unreal Engine will automatically parse the scene file and convert it into the internal data format of the engine. The compilation and import process will convert FBX, Open DRIVE and other files into assets formats that can be recognized and used by Unreal Engine: FBX files will be converted into Unreal Engine's Static Mesh assets, representing three-dimensional geometric models in the scene. Open DRIVE files will be converted into Unreal Engine's RoadNetwork assets, containing the geometry and topology information of the road. The properties and settings in the Road Runner scene data file will be applied to the corresponding Unreal Engine assets, such as materials, lighting, etc. After the conversion is completed, the simulation scene will be presented in the scene format of Unreal Engine, which can be viewed and edited in the engine editor.

[0080] Load the compiled underground simulation scene in CARLA simulator: Start CARLA simulator and specify the compiled Unreal Engine scene to load. CARLA simulator will read the scene data and reconstruct a three-dimensional virtual scene corresponding to the real underground environment in the simulation environment. After loading is completed, the underground simulation scene will be presented in CARLA simulator, which can be interacted and data collected.

[0081] Use the previously built unmanned vehicle model and laser radar sensor model to collect data in the simulation scene: In CARLA simulator, load the pre-built unmanned vehicle model and configure the laser radar sensor model. Through the API interface provided by CARLA, control the unmanned vehicle to drive and the sensor to work in the simulation scene. During the driving of the unmanned vehicle, the laser radar sensor model will collect point cloud data of the simulation environment in real time and transmit the data to the data recording module.

[0082] The control vehicle model drives in the simulation scene while recording the point cloud data of the simulation environment through the sensor model to generate a point cloud data package: By writing a control script or using the automatic driving function provided by CARLA, the unmanned vehicle model is controlled to drive in the underground simulation scene according to the driving path and speed. During driving, the laser radar sensor model continuously collects point cloud data of the simulation scene and transmits the data to the data recording module. The data recording module packages and stores the received point cloud data to generate a point cloud data package file for subsequent SLAM mapping and other analysis tasks. By adjusting the driving path of the unmanned vehicle and the collection parameters, full-coverage and high-quality simulation scene point cloud data can be obtained.

[0083] The simulation scene point cloud data is mapped using a SLAM algorithm to generate a high-precision map suitable for unmanned driving algorithms, and the recorded simulation scene point cloud data package is played back using a SLAM algorithm: The collected point cloud data package is imported into the data processing module of the SLAM algorithm. The SLAM algorithm estimates the motion trajectory and pose change of the unmanned vehicle in the simulation scene by analyzing consecutive point cloud frame data. By matching point cloud data at different times, the SLAM algorithm can identify repeated structures in the scene and establish a correlation between data frames. The estimated trajectory and map are optimized using optimization algorithms such as graph optimization or filtering algorithms to reduce cumulative errors and improve mapping accuracy.

[0084] The simulation environment is reconstructed and mapped in three dimensions using a SLAM algorithm to generate a high-precision point cloud map: During playback of the point cloud data, the SLAM algorithm gradually fuses new data frames to update and expand the coverage of the map. By point cloud registration and fusion technology, point cloud data collected from different perspectives and positions are aligned and merged to form a complete three-dimensional point cloud map. The generated point cloud map is denoised and simplified using point cloud filtering and downsampling algorithms to improve the quality and efficiency of the map. After processing by the SLAM algorithm, the generated point cloud map can represent the three-dimensional structure of the simulation scene with high precision and high density.

[0085] The obtained point cloud map has high consistency with the real underground environment and can be used as a reference map for unmanned driving algorithms: Since the simulation scene is modeled based on the point cloud data of the real underground environment, the point cloud map generated by the SLAM algorithm has high geometric consistency with the real environment. The point cloud map can accurately reflect the spatial structure and positional relationship of key areas such as underground tunnels, galleries, and chambers, providing a reliable reference for unmanned driving algorithms. Unmanned driving algorithms can use the point cloud map for positioning, navigation, obstacle avoidance, and other tasks to ensure similar behavior in the simulation environment as in the real environment.

[0086] The high-precision point cloud map obtained by SLAM mapping is further used to produce a simulation scene map suitable for autonomous driving algorithms: post-processing of the point cloud map generated by SLAM extracts important environmental information for autonomous driving algorithms. Through point cloud segmentation and clustering algorithms, key elements such as ground, wall, and obstacles in the point cloud map are identified. Using feature extraction algorithms such as plane fitting and edge detection, important features such as roads, landmarks, and intersections are extracted from the point cloud map. The extracted environmental information is converted into a map format suitable for autonomous driving algorithms, such as vector maps or raster maps.

[0087] Post-processing of the point cloud map extracts key information such as roads and landmarks to generate vector maps or raster maps for autonomous driving algorithms: for vector maps, the extracted road and landmark information is represented as geometric primitives such as line segments and polygons, with corresponding attribute information attached. For raster maps, the point cloud map is divided into regular grids, each representing environmental attributes such as passability and height within a certain range. The generated vector maps or raster maps have the characteristics of structure and good compression, facilitating fast querying and analysis by autonomous driving algorithms. According to requirements, the generated maps are further edited and improved, such as adding semantic information and road network topology, to meet the specific needs of autonomous driving algorithms.

[0088] After building the scene, perform simulation verification: the host computer acts as a server and starts Unreal Engine to enter the CARLA simulation environment: open the Unreal Engine editor on the host computer and load the previously built underground simulation scene. Through the CARLA plugin of Unreal Engine, set the scene to CARLA simulation mode and configure related parameters such as server port and sensor settings. Click the "Play" button to start the CARLA simulation environment, which runs in the background, waiting for the connection of the lower computer.

[0089] Start Ros core and CARLA-ROS bridge on the host computer: open a terminal and enter the ros core command to start the ROS master node, providing support for subsequent ROS communication. In another terminal, enter the ros launch carla_ros_bridge carla_ros_bridge.launch command to start the CARLA-ROS bridge. The CARLA-ROS bridge is responsible for converting sensor data and vehicle state information in the CARLA simulation environment into ROS topics and publishing them to the ROS network. Through the ROS topic, the autonomous driving algorithm of the lower computer can obtain real-time data from the simulation environment and send control instructions to the simulation vehicle.

[0090] Check the ROS topics to confirm that the CARLA simulation environment has started successfully: In the terminal of the host computer, enter the rostopic list command to view the current ROS topic list. Confirm that the topic list contains topics related to the simulation sensors, such as / carla / ego_vehicle / lidar / lidar1 / point_cloud, which represents the laser radar point cloud data topic. At the same time, check the topics of other sensors such as cameras, GPS, etc. to ensure that the data is normally published. If you see the data topics of the relevant sensors, it means that the CARLA simulation environment has been successfully started and connected with the ROS system.

[0091] The lower computer as a client runs the autonomous driving algorithm for simulation testing: On the lower computer, ensure that the ROS environment and dependent libraries required by the autonomous driving algorithm have been installed and configured. Copy the code package of the autonomous driving algorithm to the ROS workspace of the lower computer and compile it to generate an executable file. In the terminal of the lower computer, enter the rosrun command to start the main program of the autonomous driving algorithm and begin simulation testing. The autonomous driving algorithm subscribes to ROS topics to obtain sensor data from the host computer, such as laser radar point cloud, camera image, etc. The algorithm performs environment perception, positioning, path planning and control decision-making based on the acquired sensor data, and generates control instructions. Through the ROS topic, the control instructions are sent back to the host computer to control the motion of the simulation vehicle in the CARLA environment.

[0092] Start RVIZ on the host computer to display the running trajectory of the simulation vehicle and sensor data in real time: In the terminal of the host computer, enter the ros run rviz rviz command to start the RVIZ visualization tool. In RVIZ, add relevant display plugins such as the "Odometry" plugin to display the vehicle's motion trajectory and the "PointCloud2" plugin to display the laser radar point cloud data, etc. Configure the display parameters of RVIZ, such as coordinate system, color, size, etc. to optimize the visualization effect. During the simulation process, RVIZ will update and display the vehicle's motion trajectory and sensor data in real time, making it easy to observe and analyze the performance of the autonomous driving algorithm. In the specific embodiments of the present application, the laser radar of the autonomous vehicle can accurately detect the position of the front obstacle vehicle after adding the obstacle vehicle, and generate a subsequent obstacle avoidance path according to the obstacle position.

Claims

1. A HIL simulation method for the whole process of underground mining, comprising: collecting point cloud data of an underground scene; generating a simulation scene in CARLA format using the collected point cloud data; compiling and importing the generated simulation scene in CARLA format through an Unreal engine, building a vehicle model and a laser radar sensor model in a Carla simulator, and collecting a point cloud data packet of the simulation scene; generating a PCD format point cloud map through a SLAM algorithm according to the collected point cloud data packet of the simulation scene, and constructing a map of the simulation scene according to the PCD point cloud map; building a hardware-in-the-loop (HIL) simulation platform comprising a host computer, a lower computer, and an actuator; loading the constructed map of the simulation scene to the constructed HIL simulation platform, simulating a real mine working condition scene through the simulation platform, verifying an unmanned driving algorithm for the mine working condition, and verifying the computing power, stability, and control effect on hardware of a controller by connecting the controller and vehicle hardware; wherein the point cloud map comprises: optimizing and correcting an inter-frame transformation estimated by an ICP algorithm, comprising: acquiring speed and heading angle information of an unmanned vehicle through an odometer and acquiring attitude angle information of the unmanned vehicle through an IMU, and time-synchronizing the acquired speed, heading angle information, and attitude angle information; pre-integrating the time-synchronized odometer and IMU sensor data to obtain a motion increment between two frames of point cloud data; fusing the motion increment as a motion constraint with the inter-frame transformation estimated by the ICP algorithm, constructing an optimization objective function, and the optimization objective function comprising an ICP matching error term and a motion constraint error term; solving the optimization objective function by using a nonlinear optimization algorithm to obtain an optimized inter-frame transformation; transforming the current frame of point cloud using the optimized inter-frame transformation to obtain point cloud data of the current frame and a previous frame in the same coordinate system for point cloud inter-frame registration; repeating the above steps until all frames of point cloud data are processed to obtain a point cloud inter-frame registration result optimized by a motion constraint; acquiring a globally consistent point cloud map, comprising: extracting a feature descriptor of each frame of point cloud data using a bag-of-words model algorithm and constructing a visual dictionary according to a motion trajectory obtained by inter-frame registration; performing feature matching on each frame of point cloud data using the constructed visual dictionary, and calculating a similarity score between the current frame and a historical frame; when the similarity score exceeds a set threshold, determining that the unmanned vehicle has repeatedly reached the same position, and detecting a loop; establishing a loop constraint between corresponding positions according to the loop detection result, adding the loop constraint to the constructed optimization objective function, and constructing a loop constraint graph; optimizing and solving the constructed loop constraint graph using a G20 graph optimization library to adjust the motion trajectory of the unmanned vehicle; and splicing multiple frames of point cloud data together according to the optimized motion trajectory to obtain a globally consistent point cloud map. 2.The HIL simulation method for the whole process of underground mining according to claim 1, wherein generating a simulation scene in CARLA format comprises: importing the collected point cloud data into software RoadRunner. ​ According to the morphological characteristics of the imported point cloud data, a three-dimensional model of the simulation scene is established to restore the real underground scene; According to the established three-dimensional model, the road network and road corresponding to the real underground scene are built, and the CARLA format simulation scene is generated.

3. The HIL simulation method for the whole process of underground mining according to claim 2, characterized in that: According to the PCD point cloud map, a map of the simulation scene is constructed, including: Importing the collected simulation scene point cloud data packet into the point cloud processing framework PCL, and preprocessing the point cloud data; According to the preprocessed point cloud data, the inter-frame registration and global optimization of the point cloud data are performed through the laser SLAM algorithm, a point cloud map is generated, and the point cloud map is saved as a PCD format point cloud map; The generated PCD format point cloud map is post-processed, and the post-processing includes voxel downsampling, plane extraction and clustering segmentation; the point cloud features of the road in the point cloud map are extracted; According to the extracted point cloud features, a semantic map of the simulation scene is constructed, the road in the point cloud map is converted into a geometric element, and a map matching the real environment of the simulation scene is generated.

4. The HIL simulation method for the whole process of underground mining according to claim 3, characterized in that: Generating a point cloud map includes: Using the preprocessed continuous frame point cloud data, inter-frame registration is performed through the ICP algorithm to estimate the relative motion between adjacent frames and obtain the inter-frame transformation matrix; wherein the ICP algorithm iteratively calculates the optimal transformation matrix by minimizing the distance error between the corresponding points; On the basis of ICP inter-frame registration, motion constraints are set, and the inter-frame transformation estimated by the ICP algorithm is optimized and corrected using odometer and IMU sensor data; On the basis of inter-frame registration and motion constraint optimization, a loop detection algorithm is used to determine whether the unmanned vehicle has repeatedly arrived at the same position, if a loop is detected, a closed loop constraint is added between the corresponding positions, and a global optimization algorithm is used to globally optimize and adjust the entire trajectory to obtain a globally consistent point cloud map; The optimized point cloud map is saved as a PCD format point cloud file.

5. The HIL simulation method for the whole process of underground mining according to claim 1, characterized in that: A nonlinear optimization algorithm is used to solve the optimization objective function to obtain the optimized inter-frame transformation, wherein the nonlinear optimization algorithm uses the Levenberg-Marquardt algorithm.

6. The HIL simulation method for the whole process of underground mining according to claim 1, characterized in that: A hardware-in-the-loop HIL simulation platform including a host computer, a lower computer and an actuator is built, including: The host computer and the lower computer communicate through Ethernet, and use the multi-computer communication principle of ROS for data communication; The lower computer and the actuator communicate through CAN bus and use CAN protocol to send corresponding CAN messages.

7. The HIL simulation method for the whole process of underground mining according to claim 6, characterized in that: It also includes: The host computer acts as a server and the lower computer acts as a client, and the simulation platform is started for simulation in the following way: Start the Unreal Engine of the host computer and enter the Carla simulation environment; Start Roscore and run the ROS bridge. If sensor data corresponding to the sensor model in the simulation scene is received in the ROS topic, it is determined that Carla is successfully started; Run the unmanned driving algorithm on the slave computer and perform simulation testing; Start the RVIZ tool on the host computer to display the vehicle running trajectory corresponding to the vehicle control signal output by the slave computer and the point cloud data collected by the sensor model.

8. The HIL simulation method for the whole process of underground mining according to claim 7, characterized in that: It further comprises: The host computer is installed with an ubuntu system, runs the Unreal Engine and Carla simulation software, receives the vehicle control signal from the slave computer, controls the vehicle model in the simulation scene, and generates sensor simulation data according to the sensor model in the simulation scene, and sends the sensor simulation data to the slave computer through the ROS bridge; After receiving the sensor simulation data sent by the host computer through the ROS bridge, the slave computer inputs the sensor simulation data into the unmanned driving algorithm, generates a vehicle control signal, sends it to the host computer through the ROS bridge, and sends the corresponding CAN message to the actuator through the CAN bus; The actuator is the vehicle hardware, which includes the driving system, the braking system and the steering system, and performs the corresponding vehicle hardware action according to the control signal output by the slave computer.

Images