Simulation System, Method and Storage Medium for Driverless Mining Trucks

Abstract

The present invention provides a simulation system, method and storage medium for an unmanned mining vehicle. The simulation system includes: a mining area environment module that provides a simulated mining area environment; a mining vehicle module that creates a simulated mining vehicle and controls the dynamic behavior of the simulated mining vehicle; a sensor module that collects sensor data during the driving of the simulated mining vehicle; a domain controller that receives the sensor data to generate a mining vehicle control instruction based on the sensor data, and transmits the mining vehicle control instruction to the mining vehicle module to control the dynamic behavior of the simulated mining vehicle; and a protocol parsing module that parses the sensor data and the mining vehicle control instruction according to a first protocol and a second protocol to achieve data transmission between the sensor module and the mining vehicle module and the domain controller.

Description

Technical Field

[0001] The present invention relates to the field of open-pit mine mining, and relates to a simulation system for driverless mining trucks and an operation method thereof. Background Art

[0002] With the country's strong promotion of the construction of mine intelligence and the booming development of driverless technology, the introduction of driverless has gradually become the mainstream. However, during the actual project implementation process, problems such as project delays caused by the lack of a complete simulation test process, difficulty in reproducing road test results, and complex fault troubleshooting processes will greatly increase the project cost and even affect the final acceptance. Summary of the Invention

[0003] The purpose of the present invention is to propose, in view of the deficiencies of the prior art, a simulation system with low cost, high flexibility, and scalability and complete elements suitable for driverless mining trucks, which can provide reliable simulation results, thereby helping engineers evaluate driverless mining trucks, quickly reproduce problems on driverless mining trucks, and at the same time expose potential problems that may exist in advance.

[0004] According to a first aspect of the present invention, there is provided a simulation system, method, and storage medium for driverless mining trucks. The simulation system includes: a mining area environment module that provides a simulated mining area environment; a mining truck module that creates a simulated mining truck and controls the dynamic behavior of the simulated mining truck; a sensor module that collects sensor data during the driving of the simulated mining truck; a domain controller that receives the sensor data to generate a mining truck control instruction based on the sensor data and transmits the mining truck control instruction to the mining truck module to control the dynamic behavior of the simulated mining truck; and a protocol parsing module that parses the sensor data and the mining truck control instruction according to a first protocol and a second protocol to achieve data transmission between the sensor module, the mining truck module, and the domain controller.

[0005] Optionally, the sensor data includes working environment data and vehicle status data.

[0006] Optionally, the first protocol is a working environment protocol suitable for parsing working environment data, and the second protocol is a vehicle status protocol suitable for parsing vehicle status data.

[0007] Optionally, the working environment protocol further parses the working environment data into vehicle positioning data and image data, and the protocol parsing module performs format conversion on the vehicle positioning data, the image data, and the vehicle status data respectively.

[0008] Optionally, the vehicle positioning data, the image data, and the vehicle status data are respectively converted into serial port data, network port data, and CAN port data, and are transmitted to the domain controller in the converted data format.

[0009] Optionally, the protocol parsing module parses the mine car control instructions from the domain controller according to the second protocol.

[0010] Optionally, the simulation system further includes a high-precision map module, which is used to generate a high-precision map providing a global view of the simulated mining area environment, and transmits the generated high-precision map to the mining area environment module and the domain controller respectively.

[0011] Optionally, the mining area environment module performs three-dimensional modeling on the actual mining area scene to provide a simulated mining area environment, or provides a simulated mining area environment by directly receiving the three-dimensional modeling data of the actual mining area scene.

[0012] Optionally, data time synchronization processing is performed by using an interpolation method, and then data fusion is performed on the time-synchronized data to achieve three-dimensional modeling of the actual mining area scene.

[0013] Optionally, the mine car module creates a simulated mine car by using recursive dynamics to simulate the dynamic behavior of the actual mine car.

[0014] Optionally, the mining area environment module, the mine car module, and the sensor module can be integrated into a simulator, the simulator is bridged with the middleware, and the sensor data is transmitted to the protocol parsing module through the middleware.

[0015] According to the second aspect of the present invention, a method for simulating the operation of an unmanned mine car by using a simulation system of an unmanned mine car is provided. The method includes: collecting sensor data by the sensor module during the driving of the simulated mine car; parsing the sensor data by the protocol parsing module according to the first protocol and the second protocol to convert the format of the sensor data and transmit it to the domain controller; generating a mine car control instruction by the domain controller based on the parsed sensor data, and parsing the mine car control instruction by the protocol parsing module according to the second protocol to convert the format of the mine car control instruction and transmit it to the mine car module; and controlling the dynamic behavior of the simulated mine car by the mine car module based on the mine car control instruction.

[0016] Optionally, the sensor data includes working environment data and vehicle status data.

[0017] Optionally, the first protocol is a working environment protocol suitable for parsing working environment data, and the second protocol is a vehicle status protocol suitable for parsing vehicle status data.

[0018] Optionally, the working environment protocol further parses the working environment data into vehicle positioning data and image data, and the protocol parsing module performs format conversion on the vehicle positioning data, the image data, and the vehicle status data respectively.

[0019] Optionally, the vehicle positioning data, image data, and vehicle status data are respectively converted into serial port data, network port data, and CAN port data, and are transmitted to the domain controller in the converted data formats.

[0020] Optionally, the mining truck module controls the simulated mining truck to start driving based on the destination received from the cloud.

[0021] According to a third aspect of the present invention, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method according to the second aspect of the present invention.

[0022] According to a fourth aspect of the present invention, there is provided a computer program product including a computer program, which implements the method according to the second aspect of the present invention when executed by a processor.

[0023] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. Description of the Drawings

[0024] According to the description of those exemplary embodiments with reference to the accompanying drawings, aspects and features of various exemplary embodiments will become more apparent, wherein:

[0025] Figure 1 Shows a simulation system for an unmanned mining truck according to an embodiment of the present invention.

[0026] Figure 2 Shows a mining area environment module in the simulation system according to an embodiment of the present invention.

[0027] Figure 3 Shows an exemplary process in which the mining area environment module according to an embodiment of the present invention synchronizes data using an interpolation method.

[0028] Figure 4 Shows a mining truck module in the simulation system according to an embodiment of the present invention.

[0029] Figure 5 Shows a sensor module in the simulation system according to an embodiment of the present invention.

[0030] Figure 6 Shows a diagram of the communication relationships between the various modules in the simulation system according to an embodiment of the present invention.

[0031] Figure 7 Shows a flowchart of an operation method of the simulation system according to the present invention. Detailed Embodiments

[0032] Figure 1 Fig. Figure 1 shows a simulation system 1000 for driverless mining trucks according to an embodiment of the present invention. Among them, the simulation system 1000 includes a mining area environment module 10, a mining truck module 20, a sensor module 30, a protocol parsing module 40, a domain controller 50, and a high-precision map module 60.

[0033] In the present invention, the mining truck module 20 is used to create a simulated mining truck, the mining area environment module 10 is used to provide a simulated mining area environment, the sensor module 30 is used to collect sensor data SEDA during the driving of the simulated mining truck, and transmit the sensor data SEDA to the domain controller 50 after parsing by the protocol parsing module 40. Thereafter, the domain controller 50 generates a mining truck control instruction INS based on the parsed sensor data SEDA and the high-precision map MAP generated by the high-precision map module 60, and transmits the mining truck control instruction INS to the mining truck module 20 after parsing by the protocol parsing module 40, so that the mining truck module 20 controls the dynamic behavior of the simulated mining truck based on the parsed mining truck control instruction INS and the high-precision map MAP generated by the high-precision map module 60, and can drive along the planned path in the mining truck control instruction INS.

[0034] In addition, in some embodiments, the mining area environment module 10, the mining truck module 20, and the sensor module 30 can be integrated into a simulator VCU. However, the embodiments are not limited thereto. For example, in other embodiments, the mining area environment module 10, the mining truck module 20, and the sensor module 30 can be arranged separately, which can be set according to specific actual needs.

[0035] Figure 2 Fig. Figure 2 shows the mining area environment module 10 in the simulation system 1000 according to an embodiment of the present invention. Before the mining area environment module 10 provides a simulated mining area environment SE, it is necessary for an actual mining truck to collect a variety of sensor data in an actual mining area, so as to comprehensively process these sensor data to realize the three-dimensional reconstruction of the actual mining area scene. Specifically, after selecting the mining area scene to be simulated, at least one perception sensor FS and at least one auxiliary positioning sensor PS are installed on the actual mining truck. After the actual mining truck drives around the actual mining area environment multiple times, perception sensor data SDATA can be obtained from the perception sensor FS and positioning sensor data PDATA can be obtained from the auxiliary positioning sensor PS respectively, and these sensor data are transmitted to the mining area environment module 10. According to the present invention, the mining area environment module 10 can perform time synchronization processing and data fusion processing on these sensor data to realize the three-dimensional reconstruction of the actual mining area scene, thereby generating a simulated mining area environment SE.

[0036] In one embodiment of the present invention, the perception sensor FS may include a lidar sensor, a millimeter-wave radar sensor, and / or a camera, and the auxiliary positioning sensor PS may include an IMU (Inertial Measurement Unit) sensor and / or a GNSS (Global Navigation Satellite System) sensor. However, the embodiments are not limited thereto, and different types of sensors can be selected according to specific actual needs. Additionally, preferably, a GNSS sensor can be used as the auxiliary positioning sensor PS because the GNSS sensor comes with a second pulse generator, which can eliminate the cumulative error caused by the "clock drift" of the clock source, thereby improving the synchronization effect between the data of multiple sensors.

[0037] As Figure 2 shown, according to an embodiment of the present invention, the mining area environment module 10 may adopt an interpolation method to perform time synchronization processing on the received perception sensor data SDATA and positioning sensor data PDATA, so that the sensor data at the same moment from different sensors can be fused in subsequent data fusion steps. Specifically, in the interpolation method according to the present invention, the radar data acquisition moment is used as the inserted time point, so as to obtain the equivalent information at the same moment of other sensor data (such as camera data, positioning sensor data) except radar data. This will be described in detail below with reference to Figure 3 for details.

[0038] Thereafter, according to the time synchronization processing result, time-synchronized radar point cloud data and camera images can be obtained. Then, the time-synchronized radar point cloud data and camera images can be further data-fused to perform 3D reconstruction of the actual mining area scene. Specifically, the radar point cloud data can be used as the basis for understanding the corresponding mining area scene, and the camera image information can be used to assist in positioning the scene information in the current frame, thereby realizing the 3D reconstruction of the actual mining area scene. In this way, the computing power requirement of the server can be greatly reduced, and the target detection and semantic segmentation accuracy of scene elements are significantly improved. In one embodiment of the present invention, the radar data used as the insertion basis can be lidar data collected from a lidar sensor. However, the embodiments are not limited thereto, and different settings can be made according to actual needs. For example, it can also be data collected from a millimeter-wave radar sensor.

[0039] In addition, in other embodiments, it may be different from Figure 2In the manner shown in the figure, the mining area environment module 10 receives the perception sensor data SDATA and the positioning sensor data PDATA, and then performs a three-dimensional modeling process. On the contrary, in other embodiments, a three-dimensional modeling may be performed based on an existing modeling module according to the perception sensor data SDATA and the positioning sensor data PDATA, and three-dimensional modeling data of the actual mining area scene is generated. Then, the three-dimensional modeling data may be directly imported into the mining area environment module 10 or imported after format conversion. Thus, in this way, the mining area environment module 10 may provide a simulated mining area environment SE by directly receiving the three-dimensional modeling data.

[0040] In terms of sensor data acquisition according to the present invention, for the data acquisition of a lidar sensor, it is necessary to collect User Datagram Protocol (UDP) data packets through the lidar in the actual driving mining area environment, and parse the UDP data packets to obtain the point cloud data packets of the lidar in the current frame. For the data acquisition of a millimeter-wave radar sensor, since the millimeter-wave radar sensor transmits data through the CAN port in the Controller Area Network (CAN), the specific work will be divided into two parts: the first part is to use Socket CAN to receive CAN port data, and each communication uses the CAN FRAME structure to encapsulate the data into a frame, and then a custom write function is used to implement data transmission, and a custom read function is used to implement data reception, and a Socket CAN class interface for CAN port data transceiver needs to be customized; the second part is to store the data received by the CAN port and then perform protocol parsing, and the protocol refers to the CAN message type protocol defined by the millimeter-wave radar manufacturer. A UNION union variable for storing data needs to be customized. For the message input and message output parts included in the protocol, it is necessary to define message classes respectively and implement the data acquisition and transmission methods in the message classes. For the millimeter-wave radar, clusters are created based on the reflected signals of the objects. As the number of clusters increases, the detection distance of the millimeter-wave radar also increases at the same time. It is necessary to define a UNION union variable for storing cluster data, and define a message type for cluster output and be responsible for implementation. Based on the clusters, the object (Object) obtained by synthesizing a series of reflected signals detected by the millimeter-wave radar is the message body that really needs to be concerned. Here, a new storage data structure UNION needs to be defined for storing the Object, and a message type for Object output needs to be defined and be responsible for implementation. Finally, it is necessary to implement the splicing work of handing over the data received by the CAN port to each part of the protocol for processing. Here, it mainly involves the encapsulation definition of the socket interface and the implementation of the interface class of the radar protocol.

[0041] Figure 3An exemplary process in which the mining area environment module 10 according to an embodiment of the present invention uses an interpolation method to synchronize data is shown. To obtain the equivalent data information at a certain moment, it is necessary to index and obtain the data of two frames before and after this moment. Therefore, all sensor data SDATA and PDATA can be read from the buffer. Since the radar data does not need to be interpolated, it is independently stored in a specific container in the database, while the other sensor data is stored in a temporary container in the database that has not been synchronized. It is known that since the data in the container is arranged in chronological order, this is equivalent to forming a queue with the input order as the time axis in the container. What the index needs to do is to map the acquisition moment of the radar (such as lidar or millimeter-wave radar) to the corresponding position on the time axis of the other sensor data storage container, so that the data of two frames before and after at a specific time point can be obtained.

[0042] According to an embodiment of the present invention, as Figure 3 shown, the lidar data is taken as an example for illustration. However, it should be noted that this is only an example, and the interpolation method can be executed based on other data according to actual needs. Specifically, after the mining area environment module 10 receives the sensor data, the lidar data can be independently stored in a specific data container, while the other data including positioning data, image data, and optionally millimeter-wave radar data are respectively stored in a temporary data container. Then, each acquisition moment of the lidar data is used as an interpolation to map to the corresponding moments in each other data. For example, when the moment A in the lidar data is used as an interpolation, the corresponding or matching acquisition moment can be searched on the time axis of each other data, so as to map to the corresponding positions on each time axis. As Figure 3 shown, the moment A can be mapped to the moment c1 on the time axis of the positioning data, the moment i2 on the time axis of the image data, and the moment f3 on the time axis of the millimeter-wave radar data respectively. Thereafter, the respective data information at these moments can be used as time-synchronized data, and the time-synchronized data is fused to prepare for the subsequent 3D reconstruction process. Next, a similar interpolation mapping process to the moment A can be performed on the moment B in the lidar data, and so on.

[0043] During the indexing operation, sensors may inevitably encounter situations such as frame loss or problems with timestamps. To avoid such impacts on program functionality, the indexing operation can follow the following rules: 1) If the time of the first data is later than the radar acquisition time (insertion time) to be inserted, that is, there is no data before the insertion time, then there is nowhere to insert and the operation exits directly; 2) If the first data is earlier than the insertion time and the second data is also earlier than the insertion time, then the data at the first time is meaningless and it should continue to search downwards and delete the first data; 3) If the radar acquisition time is already between the first two data, but the time difference between the first data time and the radar acquisition time is too large (e.g., greater than the first threshold), then there must be missing data in the middle and the operation exits directly; 4) If the time difference between the second data time and the radar acquisition time is too large (e.g., greater than the second threshold), then there is also a situation of missing data in the middle and the operation exits directly. As an example, the first threshold and the second threshold can be the same or different. Only when all the above constraint conditions are met can the corresponding insertion position be clearly found. Next, methods such as linear interpolation and spherical interpolation can be used to interpolate the data of each sensor, thereby improving the time synchronization problem of multiple sensors.

[0044] Figure 4 The mine car module 20 in the simulation system 1000 according to an embodiment of the present invention is shown. Among them, the mine car module 20 can create a visual simulation mine car MVE and can control the dynamic behavior of the simulation mine car MVE. In the present invention, after the mine car module 20 receives a mine car control instruction, it can control the dynamic behavior of the simulation mine car MVE according to the instruction so that the simulation mine car MVE has desired vehicle state parameters and travels along the planned path.

[0045] According to some embodiments of the present invention, the mine car module 20 at least includes a basic model part BMO and other system model parts SMO. Specifically, the basic model part BMO at least includes seven degrees of freedom of dynamics (longitudinal, lateral, yaw motion of the vehicle body, and rotational motion of four wheels). When the number of wheels is more than four, the degrees of freedom increase in sequence. If the influence of the suspension needs to be considered, each wheel also needs to increase the degree of freedom of vertical bounce. If the tire model does not perform stability control in the non-linear region, the model can be constructed by looking up a table; otherwise, an analytical model that can better reflect the longitudinal and lateral coupling characteristics should be used. In addition, other system model parts SMO can be established as needed, which can include a motor model, a battery model, a DC-DC model, an engine model, and / or a transmission model, etc.

[0046] In an embodiment of the present invention, the mine car module 20 can use recursive dynamics to simulate the dynamic behavior of an actual mine car to create a simulated mine car MVE. Specifically, given the initial value of the model state, it is then continuously iterated through numerical integration. The calculation process of the model can be divided into three stages: kinematics, statics, and dynamics. Among them, the coordinate systems commonly used in vehicle kinematic modeling include: inertial coordinate system, vehicle body coordinate system, wheel coordinate system, and tire coordinate system. The same physical vector needs to be transformed in different coordinate systems. For example, the speed of the upper suspension fulcrum is calculated through the longitudinal, lateral, vertical, tilt, pitch, and yaw of the vehicle body, and then the wheel center speed is calculated by adding the vertical bounce speed of the wheel, and then the speed of the center of the ground contact patch is calculated by adding the rotational speed of the wheel. Then, in static modeling, the tire force generated by the wheel movement is calculated through the tire model, the spring force and damping force are calculated through the relative movement of the upper and lower suspension fulcrums, and the air resistance is calculated through the speed, etc. Furthermore, in dynamic modeling, the state at the next movement moment is obtained through numerical integration. The above three-step calculations (i.e., kinematic modeling, static modeling, and dynamic modeling) are required at each moment, and then repeated continuously until the predetermined simulation time is reached.

[0047] Figure 5 Figure 4 shows the sensor module 30 in the simulation system 1000 according to an embodiment of the present invention. Among them, the sensor module 30 can include a plurality of analog sensors to collect sensor data SEDA during the driving of the simulated mine car MVE. In one embodiment, the plurality of analog sensors can include a lidar sensor and a camera. Optionally, the plurality of analog sensors can also include a positioning sensor, such as a GNSS positioning sensor. However, the embodiments are not limited thereto, and other types of analog sensors or different arrangements of analog sensors can be included according to actual needs.

[0048] According to some embodiments of the present invention, when running the simulation system 1000, by setting the analog sensors in the sensor module 30 to have the same longitude and latitude as the simulated mine car MVE, during the driving of the simulated mine car MVE, working environment data ENDA is collected in the simulated mining area environment SE, and at the same time, the vehicle state data STDA of the simulated mine car MVE is sensed and obtained. The sensor module 30 transmits the obtained working environment data ENDA and vehicle state data STDA as sensor data SEDA.

[0049] Figure 6A diagram showing the communication relationships among the various modules in the simulation system 1000 according to an embodiment of the present invention. As described above, the sensor module 30 can collect sensor data SEDA during the simulated driving of the mining vehicle MVE and transmit the sensor data SEDA to the domain controller 50. However, the sensor data SEDA obtained at this time cannot be directly received by the domain controller 50 and needs to be parsed by the protocol parsing module 40 before being transmitted to the domain controller 50 via a CAN bus or the like. As Figure 6 shown, the sensor module 30 can transmit the sensor data SEDA to the protocol parsing module 40. According to an embodiment of the present invention, the sensor module 30 can transmit the sensor data SEDA via the middleware 70, where the middleware 70 can be configured to be bridged to the sensor module 30 or the emulator VCU and communicate with the protocol parsing module 40. In other words, the sensor module 30 can transmit the sensor data SEDA to the middleware 70 through the bridge 80, and then the middleware 70 communicates and transmits the sensor data SEDA to the protocol parsing module 40. However, the embodiment is not limited thereto, and the data transmission method and the corresponding components can be set according to specific needs.

[0050] As Figure 6 shown, the protocol parsing module 40 can include a working environment protocol EPA and a vehicle status protocol SPA. As described above with reference to Figure 5 what has been described, the sensor module 30 can generate working environment data ENDA and vehicle status data STDA during the simulated driving of the mining vehicle MVE and transmit these data as sensor data SEDA to the protocol parsing module 40. According to an embodiment of the present invention, after receiving the sensor data SEDA, the protocol parsing module 40 can parse the sensor data SEDA according to the working environment protocol EPA and the vehicle status protocol SPA.

[0051] The working environment protocol EPA can extract the working environment data ENDA in the sensor data SEDA. For example, the working environment protocol EPA can identify and extract the working environment data ENDA according to the identifier (ID) and content of the data header in the working environment data ENDA. According to an embodiment of the present invention, the working environment data ENDA may include vehicle positioning data PODA and image data IMDA. For example, the sensor module 30 can be made to include a combined inertial navigation sensor and a camera (e.g., an industrial camera sensor), and the vehicle positioning data PODA can be obtained through the combined inertial navigation sensor, and the image data IMDA generated by the camera. Another example is that, as described above, the sensor module 30 can include a lidar sensor and a camera. In this case, the working environment data ENDA generated by the sensor module 30 can be the vehicle positioning data PODA generated by the lidar sensor and the image data IMDA generated by the camera. In another embodiment of the present invention, the sensor module 30 can only include a lidar sensor. In this case, the working environment protocol EPA can be a lidar sensor protocol specifically applicable to the data generated by the lidar sensor, and the vehicle positioning data PODA and the image data IMDA can be extracted from the working environment data ENDA generated by the lidar sensor. In yet another embodiment, the sensor module 30 can include a lidar sensor, a camera, and a positioning sensor. At this time, the working environment data ENDA can be the vehicle positioning data PODA generated based on the data of the lidar sensor and the positioning sensor. However, it should be noted that these are only examples, and the present invention is not limited thereto, and the sensor type setting and the corresponding protocol setting can be carried out according to specific actual needs.

[0052] On the other hand, the vehicle state protocol SPA can parse the vehicle state data STDA in the sensor data SEDA. For example, the vehicle state protocol SPA can identify and extract the vehicle state data STDA according to the ID and content of the data header in the vehicle state data STDA. According to an embodiment of the present invention, the vehicle state data STDA can be some parameters related to the dynamic behavior of the simulated mining vehicle. For example, the vehicle state data STDA can include the speed, acceleration, deceleration, steering angle, actuation state, engine state, etc. of the simulated mining vehicle.

[0053] In addition, a suitable vehicle status protocol SPA can generally be selected according to the type and model of the mine car to be simulated. In some embodiments, the J1939 protocol applicable to large vehicles can be selected as the vehicle status protocol SPA. In other embodiments, some parameters in the protocol can be further modified according to the environmental characteristics of the mining area. For example, the characteristics of the mining area scene are usually relatively single elements, closed roads, and obvious slopes. Therefore, it is usually required that the mine car has relatively low speed and acceleration. Therefore, in this case, based on the common large vehicle protocol (such as the J1939 protocol), according to the degree of demand for different parameters in the mining area scene, parameters such as speed, acceleration, and deceleration can be modified or adjusted, such as narrowing the range value. For another example, the speed parameter can be modified to be less than a first threshold, where the first threshold can be the speed at which the mine car is suitable for driving in the mining area scene. And similarly, the acceleration, deceleration, and other required parameters can be modified accordingly. This can make the vehicle status protocol SPA more suitable for application in the mining area scene, thereby more realistically simulating the dynamic behavior of a real mine car and more accurately controlling the driving state and driving path of the simulated mine car. In addition, the embodiments of the present invention are not limited thereto, and the vehicle status protocol can be formulated according to the specific characteristics of the mining area scene and the control requirements of the mine car.

[0054] After the protocol parsing module 40 extracts the working environment data ENDA and the vehicle status data STDA from the sensor data SEDA respectively, the protocol parsing module 40 further performs format conversion on these data, so as to transmit these sensor data to the domain controller 50 in a suitable format. Specifically, as Figure 6 shown, the protocol parsing module 40 converts the vehicle status data STDA into CAN port data, converts the vehicle positioning data PODA into serial port data, and converts the image data IMDA into network port data. Thereafter, the sensor data after format conversion (i.e., the vehicle status data STDA, the vehicle positioning data PODA, and the image data IMDA) can be transmitted to the domain controller 50. According to the embodiments of the present invention, the CAN port data can be transmitted to the domain controller 50 via the CAN bus, the serial port data can be transmitted to the domain controller 50 via the RS232 serial port, and the network port data can be transmitted to the domain controller 50 in the form of tcp / udp data packets.

[0055] According to the embodiments of the present invention, the domain controller 50 can receive the sensor data SEDA and generate a mine car control instruction INS based on the sensor data SEDA. As Figure 6As shown, the domain controller 50 includes a microcontroller unit MCU and an autonomous driving domain controller ACU. The two are connected via Ethernet. Among them, the microcontroller unit MCU is used for the connection control of the peripheral circuit and the interface circuit. In other words, the microcontroller unit MCU receives the vehicle status data STDA, the vehicle positioning data PODA, and the image data IMDA from the protocol parsing module 40 through the CAN port, and packs these data into Ethernet data packets. Thereafter, the microcontroller unit MCU transmits the Ethernet data packets to the autonomous driving domain controller ACU via Ethernet. Then, the autonomous driving domain controller ACU can perform high-performance data processing on the Ethernet data packets. For example, the autonomous driving domain controller ACU can extract the vehicle status data STDA and the working environment data ENDA, and fuse the working environment data ENDA with the high-precision map to generate a mining truck control instruction INS. However, it should be noted that the autonomous driving domain controller ACU can also perform other required data processing operations.

[0056] Specifically, as Figure 1 and Figure 6 shown in, the high-precision map module 60 can generate a high-precision map MAP for providing a global view of the mining area scene, and send the generated high-precision map MAP to the domain controller 50 and the mining area environment module 10 respectively. In this way, the autonomous driving domain controller ACU in the domain controller 50 can plan the subsequent driving path of the mining truck and determine the dynamic behavior parameters that the mining truck needs to adjust based on the high-precision map MAP, combined with the extracted working environment data ENDA (which includes the vehicle positioning data PODA and the image data IMDA), and form a mining truck control instruction INS with the planned path and the determined parameters and transmit it to the mining truck module 20. Since the mining truck module 20 refers to the same high-precision map MAP from the mining area environment module 10, there is a high degree of matching in terms of position positioning, vehicle parameter adjustment, etc., so a mining truck control instruction INS that can accurately control the simulated mining truck MVE can be generated, thereby improving the reliability of the output result of the simulation system 1000.

[0057] According to an embodiment of the present invention, the high-precision map module 60 includes three hierarchical structures: road model, lane model, and positioning model. In addition to the above-mentioned infrastructure model, there are also dynamic information layer and driving behavior analysis layer to supplement some other information. Among them, the road model provides a global view for the mine car, so that the mine car can understand the road information outside the sensor perception range, and can also classify unstructured roads; the lane model provides more accurate information, including lane direction, lane boundary, road sign, road speed limit; the positioning model is the top layer of the model, and it is positioned by sensing the lane lines and other landmarks on both sides of the mine road; the dynamic information layer is used to describe the dynamic conditions on the road, such as accidents and other emergencies, so it has high requirements for real-time performance; the driving behavior analysis layer analyzes the driver's driving behavior in different road scenarios, such as acceleration, deceleration, braking, etc., and converts data into decisions, so that automatic driving can better simulate the driver's behavior.

[0058] The production process of high-precision map MAP can be roughly divided into several steps: data collection, data processing, element identification, and manual verification. Specifically, map data is collected by actual sensors in a mining environment. Optionally, in other embodiments, analog sensors can be used to collect map data in a simulated environment. However, the embodiments are not limited to this, and can be selected according to actual needs. Thereafter, during the collection process, 3-5 laps are collected along the two-way lane with full coverage, the speed is limited to maintain low speed, and a data file (such as a rosbag file) is generated at a default interval, and compressed and packaged after the collection is completed. Point cloud splicing is performed based on the recorded data packets (such as rosbag packages) to generate a base map, and then deep learning is used to perform target recognition and semantic segmentation of scene elements (point clouds) in mining scenes. In the manual verification link, the base map data, image data, and point cloud data are fused and integrated into the final high-precision map MAP.

[0059] Thereafter, after the domain controller 50 generates the mine car control instruction INS based on the high-precision map MAP and the sensor data SEDA, the domain controller 50 may transmit the mine car control instruction INS to the mine car module 20 according to the original path. Specifically, Figure 6As shown in the figure, the autonomous driving domain controller ACU in the domain controller 50 can transmit the generated mine car control instruction INS to the micro control unit MCU via Ethernet, and process it into CAN port data via the micro control unit MCU, so as to transmit it to the protocol parsing module 40 via the CAN bus. As described above, the domain controller 50 cannot directly communicate with the mine car module 20 and the sensor module 30, but needs to perform a protocol parsing process between the two. Therefore, the protocol parsing module 40 parses the mine car control instruction INS according to the vehicle status protocol SPA and converts it into a data format suitable for the mine car module 20. Thereafter, the parsed mine car control instruction INS can be optionally transmitted to the mine car module 20 via the middleware 70, so that the mine car module 20 can control the dynamic behavior of the simulated mine car MVE based on the planned path and vehicle status parameter related information included in the mine car control instruction INS, combined with the high-precision map MAP from the mining area environment module 10, so as to more realistically simulate the actual mine car and make the simulation results used for more reliable evaluation and testing.

[0060] In an embodiment of the present invention, during the driving of the simulated mine car MVE, the sensor module 30 can sense the working environment data ENDA and the vehicle status data STDA in real time to generate sensor data SEDA in real time, and transmit the data to the domain controller 50 in real time, so that the domain controller 50 generates the mine car control instruction INS in real time and issues it to the mine car module 20 in real time, thereby controlling the driving direction, path, speed, etc. of the simulated mine car MVE in real time during driving, so that the simulated mine car MVE reaches the destination end point.

[0061] Figure 7 Shows a flowchart of an operation method of the simulation system 1000 according to the present invention. As Figure 7 shown, the operation method of the simulation system 1000 includes:

[0062] S1: The simulation system 1000 starts to run.

[0063] S2: The sensor module 30 senses the sensor data SEDA of the simulated mine car MVE and transmits the sensor data SEDA to the protocol parsing module 40.

[0064] In an embodiment of the present invention, the sensor module 30 may include a lidar sensor and a camera (optionally may include a positioning sensor) to sense the working environment data ENDA, and at the same time the sensor module 30 can sense the state of the simulated mine car MVE to generate the vehicle status data STDA. Then, the working environment data ENDA and the vehicle status data STDA are transmitted as the sensor data SEDA.

[0065] S3: The protocol parsing module 40 parses the sensor data SEDA according to the first protocol and the second protocol to convert the sensor data SEDA into a data format suitable for communicating with the domain controller 50, and then transmits the parsed sensor data SEDA to the domain controller 50.

[0066] In an embodiment of the present invention, the first protocol is the working environment protocol EPA, and the second protocol is the vehicle status protocol SPA. The protocol parsing module 40 converts the working environment data ENDA into a serial port format and a network port format, and converts the vehicle status data STDA into a CAN port format according to these two protocols.

[0067] S4: The domain controller 50 generates a mine car control instruction INS based on the received sensor data SEDA, and transmits the mine car control instruction INS to the mine car module 20 after parsing by the protocol parsing module 40.

[0068] In an embodiment of the present invention, the protocol parsing module 40 parses the mine car control instruction INS according to the second protocol (i.e., the vehicle status protocol SPA) to convert the mine car control instruction INS into a data format suitable for communicating with the mine car module 20, and then transmits the parsed mine car control instruction INS to the mine car module 20.

[0069] S5: The mine car module 20 controls the dynamic behavior of the simulated mine car MVE based on the received mine car control instruction INS, so that the simulated mine car MVE travels along the planned path.

[0070] According to the above various embodiments of the present invention, the simulation system 1000 and its operation method can provide a complete-element simulation test system suitable for the characteristics of the mining area environment, which can accurately reproduce the driving conditions of the mine car and provide reliable simulation test data. By using the simulation system 1000 according to the present invention, the problem of shortage of road test data in the current mining area can be solved. Engineers can use the simulation system 1000 to generate a large amount of road test data under custom scenarios through online modification, and use it as a training data set for the model training task of the perception algorithm module.

[0071] According to an embodiment of the present invention, the present invention also provides a readable storage medium and a computer program product. In some embodiments, the readable storage medium may be a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method according to any one of the above embodiments. In some embodiments, the computer program product includes a computer program, and the computer program implements the method according to any one of the above embodiments when executed by a processor.

[0072] The various embodiments of the systems and techniques described above in this document can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems-on-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that receives data and instructions from a storage system, at least one input device, and at least one output device, and transmits the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0073] The program code for implementing the methods of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the program codes are executed by the processor or controller, the functions / operations specified in the flowchart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0074] In the context of the present invention, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0075] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and a pointing device (e.g., a mouse or a trackball), by which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, speech input, or tactile input).

[0076] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), and the Internet.

[0077] A computer system can include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship between the client and the server is generated by computer programs running on the respective computers and having a client-server relationship with each other.

[0078] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present invention can be achieved, and no limitation is imposed herein.

[0079] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A simulation system for driverless mining trucks, the simulation system comprises: A mining area environment module, which provides a simulated mining area environment; A mining truck module, which creates a simulated mining truck and controls the dynamic behavior of the simulated mining truck; A sensor module, which collects sensor data during the driving of the simulated mining truck; A domain controller, which receives the sensor data to generate a mining truck control instruction based on the sensor data, and transmits the mining truck control instruction to the mining truck module to control the dynamic behavior of the simulated mining truck; and A protocol parsing module, which parses the sensor data according to a first protocol and a second protocol, and parses the mining truck control instruction according to the second protocol to achieve data transmission between the sensor module, the mining truck module and the domain controller; wherein the sensor data includes working environment data and vehicle status data, the first protocol is a working environment protocol, and is configured to extract the working environment data from the sensor data for parsing, and the second protocol is a vehicle status protocol, and is configured to extract the vehicle status data from the sensor data for parsing; wherein the working environment data includes vehicle positioning data and image data, the working environment protocol is configured to convert the vehicle positioning data into serial port data for transmission to the domain controller via an RS232 serial port, and is also configured to convert the image data into network port data for transmission to the domain controller in the form of tcp / udp data packets; and wherein the vehicle status protocol is configured to convert the vehicle status data into CAN port data for transmission to the domain controller via a CAN bus.

2. The simulation system according to claim 1, wherein, the simulation system further includes a high-precision map module, which is used to generate a high-precision map providing a global view of the simulated mining area environment, and transmits the generated high-precision map to the mining area environment module and the domain controller respectively.

3. The simulation system according to claim 1, wherein, the mining area environment module performs three-dimensional modeling on the actual mining area scene to provide the simulated mining area environment, or provides the simulated mining area environment by directly receiving three-dimensional modeling data of the actual mining area scene.

4. The simulation system according to claim 3, wherein, data time synchronization processing is performed by using an interpolation method, and then data fusion is performed on the time-synchronized data to achieve three-dimensional modeling of the actual mining area scene.

5. The simulation system according to claim 1, wherein, the mining truck module creates the simulated mining truck by using recursive dynamics to simulate the dynamic behavior of the actual mining truck.

6. The simulation system according to claim 1, wherein, the mining area environment module, the mining truck module and the sensor module can be integrated into a simulator, the simulator is bridged with middleware, and the sensor data is transmitted to the protocol parsing module through the middleware.

7. A method for simulating the operation of an unmanned mining vehicle by using a simulation system of the unmanned mining vehicle according to any one of claims 1-6, the method comprises: collecting sensor data by a sensor module during the driving of a simulated mining vehicle; parsing the sensor data by a protocol parsing module according to a first protocol and a second protocol, so as to convert the format of the sensor data and transmit it to a domain controller; generating a mining vehicle control instruction by the domain controller based on the parsed sensor data, parsing the mining vehicle control instruction by the protocol parsing module according to the second protocol, so as to convert the format of the mining vehicle control instruction and transmit it to a mining vehicle module; and controlling the dynamic behavior of the simulated mining vehicle by the mining vehicle module based on the mining vehicle control instruction.

8. The method according to claim 7, wherein, the mining vehicle module controls the simulated mining vehicle to start driving based on a destination end point received from the cloud.

9. A non-transitory computer-readable storage medium storing computer instructions, wherein, the computer instructions are used to cause the computer to execute the method according to any one of claims 7-8.

10. A computer program product comprising a computer program, the computer program implementing the method according to any one of claims 7-8 when executed by a processor.

Images