Unmanned driving guidance method and related equipment in mining areas based on large model system
Through the large model, the inadequate data of unmanned driving guidance in mining areas is solved through the integration of guidance language and environmental sensing data, and the problem of insufficient data volume in unmanned driving guidance in mining areas is achieved, and unmanned driving guidance in multi-mine areas is achieved, which reduces training costs and improves accuracy .
Patent Information
- Application Number
- CN202311862036.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-29
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2043-12-29
AI Technical Summary
The prior art cannot realize unmanned driving guidance solutions suitable for multi-mine areas, and the insufficient data volume of large models in mining areas results in high training costs and low accuracy.
Using a large model-based unmanned driving guidance method in mining areas, multi-modal fusion is carried out by obtaining guidance language text data and environmental sensing data, and combining manual intervention signals to send unmanned driving guidance signals to the on-board control equipment, using pre-training and the secondary training method are adapted to multi-mine scenarios.
It realizes unmanned driving guidance in multi-mine area scenarios, reduces training costs and improves accuracy, and ensures the safe and efficient operation of unmanned driving.
Smart Images

Figure CN118131750B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of unmanned driving technology, and in particular to an unmanned driving guidance method and related equipment in a mining area based on a large model system. Background Art
[0002] Unmanned driving in mining areas can be regarded as a specific application of unmanned driving in a specific scenario. Unmanned driving in mining areas aims to autonomously navigate through multiple types of sensors and navigation systems, and to conduct safe and efficient regional fully unmanned or semi-unmanned transportation on the mining road network. Existing urban unmanned driving can realize the automatic generation of unmanned driving guidance signals through large models, but due to the complex environment of mining areas, which is very different from urban unmanned driving scenarios, the large models obtained through urban unmanned driving training cannot be directly adapted to mining scenarios.
[0003] Mining areas are limited scenarios with little data. Although large models have stronger expression and learning capabilities, they have huge parameters and usually require large-scale data sets for training to perform effective parameter estimation. As an independent and closed scenario, the appearance of the roads and driving rules in mining areas are specific. It is difficult to adapt a unified driving model to all mining areas. The one-mine-one-model approach requires a large amount of specialized training data and training costs.
[0004] Therefore, in the prior art, there is no technical solution that can realize unmanned driving guidance in mining areas. Summary of the invention
[0005] The present invention provides a mine area unmanned driving guidance method based on a large model system, so as to solve the defect that there is no technical solution in the prior art that can realize unmanned driving guidance applicable to multiple mine areas, and realizes mine area unmanned driving guidance applicable to multiple mine areas.
[0006] The present invention provides a large-model-based unmanned driving guidance method for a mining area, comprising:
[0007] Obtaining guidance language text data of vehicles in the target mining area, inputting the guidance language text data into a language processing module in the mining area large model, and obtaining natural language features output by the language processing module;
[0008] Acquire environmental sensor data of vehicles in the target mining area, wherein the environmental sensor data includes at least one of environmental image data and laser radar point cloud data, and the environmental sensor data comes from multiple sensors, input the environmental sensor data and the natural language features into a multimodal fusion module in the large mining area model, and acquire fusion features output by the multimodal fusion module;
[0009] Determine the current target unmanned driving task, input the fusion feature into the task output head corresponding to the target unmanned driving task in the large model of the mining area, and obtain the task output output by the task output head;
[0010] Determine an unmanned driving guidance signal based on the task output, and send the unmanned driving guidance signal to an on-board control device of a vehicle in the target mining area;
[0011] A manual intervention signal is obtained and sent to the vehicle-mounted control device, wherein the manual intervention data has a higher priority than the unmanned driving guidance signal.
[0012] According to a large-model-based unmanned driving guidance method for a mining area provided by the present invention, the unmanned driving guidance signal is determined based on the task output, comprising:
[0013] Acquire a collaborative task corresponding to the target mining area vehicle, and determine a collaborative vehicle for the collaborative task, wherein the collaborative vehicle includes the target mining area vehicle;
[0014] Input the communication relationship data between the cooperative vehicles, the vehicle information data of the cooperative vehicles and the task data of the cooperative task into the cooperative planning module of the mining area large model, and obtain the cooperative planning data output by the cooperative planning module;
[0015] The unmanned driving guidance signal is determined based on the collaborative planning data and the task output.
[0016] According to a large-model-based unmanned driving guidance method for a mine area provided by the present invention, after sending the unmanned driving guidance signal to the on-board control device of the target mine area vehicle, the method comprises:
[0017] Acquire the driving data of the target mining area vehicle, and update the visual simulation model of the mining area where the target mining area vehicle is located based on the driving data.
[0018] According to a large-model-based unmanned driving guidance method for a mining area provided by the present invention, after sending the manual intervention data to the vehicle-mounted control device, the method further includes:
[0019] The large mining area model is updated based on the manual intervention data.
[0020] According to a large-model-based unmanned driving guidance method for a mining area provided by the present invention, the training process of the large-model mining area includes:
[0021] The mining area large model is trained based on multiple groups of first training data, and the multimodal fusion module and the task output head in the mining area large model are updated to obtain the pre-trained mining area large model, wherein the first training data includes sample environment sensor data and corresponding task output labels, and the first training data is obtained based on an urban driving environment;
[0022] Training the pre-trained large mining area model based on multiple sets of second training data, wherein the second training data includes sample input data, the sample input data includes sample guiding language text data, sample environmental sensor data, sample communication relationship data, sample vehicle information data and sample task data, and the second training data is obtained based on the mining area driving environment;
[0023] Among them, when the large model of the mining area is trained, the corresponding task output head is updated according to the unmanned driving subtask corresponding to the input into the large model of the mining area.
[0024] According to a large-model-based unmanned driving guidance method for a mining area provided by the present invention, the pre-trained large-model of the mining area is trained based on multiple groups of the second training data, including:
[0025] Inputting the second training data into the large model of the mining area, and obtaining a training unmanned driving guidance signal determined based on an output of the large model of the mining area;
[0026] Acquiring manual evaluation data for the training unmanned driving guidance signal;
[0027] The large mining area model is updated based on the manual evaluation data.
[0028] The present invention also provides a large-scale model-based unmanned driving guidance device for a mining area, comprising:
[0029] A language processing unit, used to obtain the guidance language text data of the target mining area vehicle, input the guidance language text data into the language processing module in the mining area large model, and obtain the natural language features output by the language processing module;
[0030] A multimodal fusion unit is used to obtain environmental sensor data of vehicles in the target mining area, wherein the environmental sensor data includes environmental image data and sensor signal data, and the sensor signal data comes from multiple sensors, and input the environmental sensor data and the natural language features into the multimodal fusion module in the large mining area model to obtain fusion features output by the multimodal fusion module;
[0031] A subtask processing unit, used to determine the current target unmanned driving subtask, input the fusion feature into the task output head corresponding to the target unmanned driving subtask in the large mining area model, and obtain the task output output by the task output head;
[0032] An unmanned driving guidance unit, configured to determine an unmanned driving guidance signal based on the task output, and send the unmanned driving guidance signal to an on-board control device of a vehicle in the target mining area;
[0033] The manual intervention unit is used to obtain manual intervention data and send the manual intervention data to the vehicle-mounted control device, and the priority of the manual intervention data is higher than the unmanned driving guidance signal.
[0034] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an unmanned driving guidance method for a mining area based on a large model as described above is implemented.
[0035] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the unmanned driving guidance method for a mining area based on a large model as described in any one of the above is implemented.
[0036] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned large-model-based unmanned driving guidance methods for mining areas.
[0037] The large-model-based unmanned driving guidance method and related equipment for a mining area provided by the present invention extract natural language features of guidance language text data through a language processing module in the large model of the mining area, and input the natural language features and environmental sensor data including environmental image data and sensor signal data into a multimodal fusion module in the large model of the mining area for fusion to obtain fusion features. According to the current target unmanned driving subtask, the fusion features are input into a task output head corresponding to the target unmanned driving subtask in the large model of the mining area, the task output output by the task output head is obtained, the unmanned driving guidance signal is determined based on the task output, the unmanned driving guidance signal is sent to the on-board control device of the target mining area vehicle, and when the manual intervention data is obtained, the manual intervention data is also sent to the on-board control device, the priority of the manual intervention data is higher than the unmanned driving guidance signal, and the unmanned driving guidance signal output by the large model and the manual intervention are combined. By adding the guidance language text and the manual intervention, the problem that the large model cannot be trained with high precision for each mining area due to insufficient mining area scene data can be effectively solved, and the unmanned driving guidance that can be adapted to multiple mining areas can be realized. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0039] Figure 1 It is a flow chart of the unmanned driving guidance method for a mining area based on a large model provided by the present invention;
[0040] Figure 2 It is a functional structure diagram of a mine large model expert system of a mine area unmanned driving guidance method based on a large model provided by the present invention;
[0041] Figure 3 It is a schematic diagram of the process of constructing a large model of a mining area in the large model-based unmanned driving guidance method for a mining area provided by the present invention;
[0042] Figure 4 It is a schematic diagram of the training process of the large model of the mining area in the large model-based unmanned driving guidance method for the mining area provided by the present invention;
[0043] Figure 5 It is a structural schematic diagram of an unmanned driving guidance device for a mining area based on a large model provided by the present invention;
[0044] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0045] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0046] Unmanned driving technology refers to the use of automation and intelligent technology to enable vehicles to drive and operate autonomously without the intervention of human drivers. This technology uses advanced technologies such as various sensors, computer vision, machine learning and decision-making algorithms to enable vehicles to perceive and understand the surrounding environment and make corresponding decisions and control actions. From a technical perspective, unmanned driving can be divided into perception, decision-making and control. The perception system uses sensors such as lidar, cameras, radar and ultrasound to obtain real-time information about the vehicle's surroundings, including road conditions, obstacles, traffic signs and other vehicles. The decision-making system conducts decision-making processes such as route planning, traffic signal recognition and vehicle behavior prediction based on the information provided by the perception system, combined with map data and rules. The control system is responsible for executing decisions and controlling the vehicle's acceleration, braking, steering and obstacle avoidance operations.
[0047] In machine learning, a large model refers to a deep learning model with a large number of parameters. These models usually have hundreds of millions of parameters and are capable of handling more complex and richer tasks. The number of parameters of a large model is usually larger than that of a traditional small model or a traditional machine learning algorithm, so more computing resources and storage space are required for training and inference. Its rise is due to the advancement of hardware technology, the emergence of unsupervised neural networks, and the availability of large-scale data sets. Large models first achieved widespread applications in natural language processing, including machine translation, text generation, question-answering systems, etc. For example, GPT-3 (developed by OpenAI) is a well-known large model with 175 billion parameters. Models of this size perform well in generating text and can produce coherent, semantically understandable sentences.
[0048] Large models have the following characteristics: 1) The number of parameters in large models is huge, which makes them more powerful in expression and learning. 2) Large models usually require large data sets for training, because the number of parameters is huge and sufficient samples are needed for effective parameter estimation. 3) They can accept feedback from human evaluation and introduce reinforcement learning mechanisms to enable the model to have logical reasoning capabilities; 4) Large models require a lot of computing resources and storage space for training and inference.
[0049] At present, unmanned mine technology solutions have been widely studied and applied in the mining industry. They include: 1) Modified unmanned vehicles, which can navigate autonomously through multiple types of sensors and navigation systems, and can carry out safe and efficient regional fully unmanned or semi-unmanned transportation on the mine road network; 2) Auxiliary robot technology is used in mines, especially underground mines, for various tasks such as inspection, maintenance and cleaning. Robots can carry sensors and cameras to monitor equipment status and environmental conditions in real time, and can also perform some repetitive, dangerous or heavy work; 3) The widespread use of unmanned aerial vehicles is mainly used for measurement, exploration and inspection tasks. They can quickly obtain images and data of the mining area, conduct topographic surveys, mineral exploration and environmental monitoring, and improve exploration efficiency and safety; 4) By widely deploying sensor networks in mines, various parameters of mining equipment and environment, such as temperature, humidity, vibration, etc., can be monitored in real time. These sensors and monitoring systems can provide timely data and early warning information to optimize the production process and reduce the risk of accidents; 5) Through automation technology and remote operation systems, some processes and equipment in mines can be autonomously controlled and remotely monitored. This allows mine operators to remotely manage and control mining equipment, reducing the need for personnel to enter dangerous areas; 6) Using big data analysis and artificial intelligence technology, data in mines can be deeply mined and analyzed.
[0050] Unmanned driving in mining areas can be seen as the application of unmanned driving in specific scenarios, and mining areas can also be seen as the edge scenarios of unmanned driving. However, the large model obtained through urban unmanned driving training does not seem to be directly adapted to mining area scenarios. This is mainly determined by the typical characteristics and difficulties of mining areas. Unmanned driving in mining areas has the following characteristics:
[0051] 1. It is difficult for large models based on urban scenarios to define and provide correct guidance for complex and dangerous situations in mines under unmanned driving. This is mainly because the mining area is a limited scenario, and its data volume is scarce. The data on edge problems in the mining area scenario is even more scarce. Due to the existence of extreme long-tail distribution of data, the edge situations in the mining area require additional methods to supplement.
[0052] 2. Due to the large size of unmanned mining vehicles and the diverse sensors installed, the traditional single-vehicle sensor multimodal fusion strategy is not suitable for mining vehicles.
[0053] 3. Unmanned mining vehicles need to realize many functions, including drivable area recognition, roadside recognition, autonomous positioning in the mining area, etc. The independent existence of these functions not only increases the time cost but also is not conducive to the development of intelligent algorithms.
[0054] 4. The mining area is a scene involving the collaborative operation of multiple vehicles, including the collaboration of excavators and mining trucks, the collaboration of pushers and mining trucks, the collaboration of multiple mining trucks, etc. The interactive intelligence of these multiple vehicles is rarely seen in urban scenes. Therefore, the large model system of the mining area needs to solve the problem of collaborative operation of multiple vehicles.
[0055] 5. As an independent and closed scene, the road appearance and driving rules of a mining area may be unique. Looking at the numerous mining areas in my country, it is difficult to adapt a unified driving model to all mining areas, but the one-mine-one-model solution requires a lot of training time and economic expenses.
[0056] In view of at least one of the above problems, the present invention provides a method for guiding unmanned driving in a mining area based on a large model. Figure 1-Figure 4 The unmanned driving guidance method for a mine area based on a large model provided by the present invention is described as follows: Figure 1 As shown, the method comprises the steps of:
[0057] S110, obtaining guidance language text data of vehicles in the target mining area, inputting the guidance language text data into a language processing module in the mining area large model, and obtaining natural language features output by the language processing module;
[0058] S120, obtaining environmental sensor data of vehicles in the target mining area, the environmental sensor data including environmental image data and sensor signal data, the sensor signal data coming from a variety of sensors, inputting the environmental sensor data and natural language features into a multimodal fusion module in the mining area large model, and obtaining fusion features output by the multimodal fusion model;
[0059] S130, determining the current target unmanned driving subtask, inputting the fusion feature into the task output header corresponding to the target unmanned driving subtask in the large model of the mining area, and obtaining the task output output by the task output header;
[0060] S140, determining an unmanned driving guidance signal based on the task output, and sending the unmanned driving guidance signal to an on-board control device of a vehicle in the target mining area;
[0061] S150, obtaining manual intervention data, and sending the manual intervention data to the vehicle-mounted control device, wherein the manual intervention data has a higher priority than the unmanned driving guidance signal.
[0062] The method provided by the present invention is implemented based on a large mine model expert system. Figure 2 As shown in the figure, the mine model expert system consists of four parts, namely, parallel simulation system, single-vehicle intelligent system, multi-vehicle collaborative intelligent system and human-machine hybrid intelligent interaction system.
[0063] Parallel simulation system: responsible for the virtual modeling of the mining area, the kinematics and dynamics construction of mining vehicles, and end-to-end data collection and model training. It is the basic training platform for large models, involving spatial modeling technology, robot modeling technology and reinforcement learning training strategies.
[0064] Single-vehicle intelligent system: It is trained by a large model through supervised and semi-supervised methods, and the model is deployed on a cloud platform. Real vehicles and simulated vehicles are controlled in real time through fast communication technology. Different large model heads correspond to related tasks according to different tasks, and can be subdivided into positioning unit, perception unit, planning unit and control unit.
[0065] Multi-vehicle collaborative intelligent system: The multi-vehicle collaborative intelligent system is responsible for the collaborative control of multiple vehicles in the mining area. The large model provides joint training and control of collaborative vehicles. When the vehicles are in an interactive state, the collaborative operation system and the single vehicle collaborate to control the vehicles simultaneously. Depending on the tasks, it can be subdivided into excavation and truck collaboration, transportation collaboration, and unloading area collaboration (push machine-mine truck collaboration) operations.
[0066] Human-machine hybrid intelligent interaction system: This system is an important embodiment of human participation in large model training. ChatGPT training requires human evaluation. For large mine models, both commentators and guides need to provide change information reminders and strong human intervention. The former hopes that the model can be quickly deployed in a new environment, while the latter aims to quickly avoid accidents. Their feedback will intervene in the training iteration of the model, and then correct the model output to improve its accuracy.
[0067] In the method provided by the present invention, the above-mentioned system functions are realized by various modules of the large model of the mining area. The large model of the mining area is used to output unmanned driving guidance signals. In the method provided by the present invention, the output of the unmanned driving guidance signal by the large model of the mining area is combined with manual intervention. By adding guidance language text and manual intervention, the problem that the large model with high accuracy cannot be trained for each mining area due to insufficient mining area scene data can be effectively solved, and unmanned driving guidance that can be adapted to multiple mining areas can be realized. The various steps of the method provided by the present invention are described in detail below.
[0068] Specifically, the guiding language text data in step S110 may be a guiding language text manually input based on the situation of the mining area, such as "there are obstacles ahead that need to be avoided". Figure 3 As shown, the language processing module in the mining area large model can be an existing language large model.
[0069] The multimodal fusion module in step S120 is used to unify the visual and text representations as well as the signal data of more sensors in the mining scene. Through the multimodal fusion module, an expandable multimodal fusion interface can be realized to adapt the multi-sensor data so as to adapt to the mining vehicles.
[0070] For the task output head in step S130, in the method provided by the present invention, different task output heads are set for different unmanned driving subtasks. Specifically, unmanned driving subtasks may include positioning tasks, route planning tasks, perception tasks, etc. Setting different output heads for different unmanned driving subtasks allows different unmanned driving subtasks to share the model except for the output head, that is, it can realize the function of multi-head multi-task adaptation system, realize multi-task adaptation based on the same large model, and share subject parameters, which not only saves time and space overhead, but also reduces the complexity of the large model of the mining area and increases the scalability of the model.
[0071] In step S140, determining an unmanned driving guidance signal based on the task output includes:
[0072] Acquire the collaborative task corresponding to the target mining area vehicle, determine the collaborative vehicle of the collaborative task, and the collaborative vehicle includes the target mining area vehicle;
[0073] Input the communication relationship data between the cooperative vehicles, the vehicle information data of the cooperative vehicles and the task data of the cooperative tasks into the cooperative planning module of the mining area large model, and obtain the cooperative planning data output by the cooperative planning module;
[0074] Determine unmanned driving guidance signals based on collaborative planning data and mission outputs.
[0075] In the method provided by the present invention, the collaborative planning of collaborative vehicles is realized through a large model, which is more suitable for mining scenarios. The collaborative planning module can perform secondary training and learning parameters on the large model through training data based on mining scenarios, which will be described in detail later.
[0076] For step S150, the unmanned driving guidance signal is data for guiding the target mining area vehicle to travel, and the unmanned driving guidance signal includes the travel speed and travel direction of the target mining area vehicle. The method provided by the present invention, after obtaining the unmanned driving guidance signal, sends the unmanned driving guidance signal to the on-board control device of the target mining area vehicle, and the on-board control device controls the target mining area vehicle to travel in the form, speed and direction corresponding to the unmanned driving guidance signal based on the unmanned driving guidance signal.
[0077] The large-scale model algorithm of the mining area in the method provided by the present invention is deployed in the cloud. Rongguang aggregates a variety of unmanned driving subtasks and uses the cloud-to-vehicle approach to remotely deploy the algorithm, thereby releasing the burden on the on-board processor and reducing the resource allocation requirements for a single vehicle.
[0078] For step S160, the method provided by the present invention is intended to solve the problem of adapting to different mining areas through human intervention and intelligent guidance. Intelligent guidance is reflected in the guiding language text data processing process in the previous text. For the human intervention part, in the method provided by the present invention, a parallel simulation system is established. In the simulation platform system, the mining area is virtually modeled, and the kinematics and dynamics of the mining area vehicles are constructed. In addition, the driving data of the mining area vehicles are collected in real time. The driving data reflects the position, driving speed and direction of the mining area vehicles, and the visual simulation model of the mining area where the target mining area vehicle is located is updated based on the driving data. In this way, the management personnel can obtain the situation of the vehicles in the mining area in time by observing the visual simulation model. When human intervention is required, the human intervention data is issued, and the human intervention data is sent to the vehicle-mounted control device of the mining area vehicle. After the vehicle-mounted control device receives the human intervention data, it preferentially controls the driving of the target mining area vehicle according to the human intervention data. That is to say, when there are both human intervention data and unmanned driving guidance signals, the driving of the target mining area vehicle is controlled according to the human intervention data. This can effectively avoid accidents caused by insufficient training data, which results in the large model of the mining area being unable to adapt well to the actual situation of the mining area after switching the mining area, and the output unmanned driving guidance signal being inaccurate.
[0079] Furthermore, when manual intervention data is received, it indicates that the unmanned driving guidance signal output by the large model of the mining area is not accurate enough. The method provided by the present invention updates the large model of the mining area based on the manual intervention data after receiving the manual intervention data, so that iterative training and updating based on the actual application mining area can be realized during the application of the large model of the mining area. The mining area feedback module based on remote control and manual guidance embodies the idea of human in the loop, provides a transmission interface for human intervention, evaluation, and guidance, and manual guidance provides fast-changing signals and directional guidance for iterative training, which makes it possible to optimize and iterate the large model of the mining area, and finally makes the large model of the mining area more and more adapted to the current mining area and outputs more accurate unmanned driving guidance signals.
[0080] The training process of the large mining area model is described below.
[0081] The training process of the mining area large model in the method provided by the present invention includes:
[0082] The mining area large model is trained based on multiple groups of first training data, and the multimodal fusion module and the task output head in the mining area large model are updated to obtain a pre-trained mining area large model, wherein the first training data includes sample environmental sensor data and corresponding task output labels, and the first training data is obtained based on an urban driving environment;
[0083] The pre-trained mining area large model is trained based on multiple sets of second training data, the second training data includes sample input data, the sample input data includes sample guiding language text data, sample environmental sensor data, sample communication relationship data, sample vehicle information data and sample task data, and the second training data is obtained based on the mining area driving environment;
[0084] Among them, when the large mining area model is trained, the corresponding task output head is updated according to the output data of the large mining area model.
[0085] like Figure 4 As shown, the training of the mining area model in the method provided by the present invention is based on the training of the urban unmanned driving multimodal large model, in which there are two parts that are different from the urban unmanned driving multimodal large model: the mining area environment platform interaction and the guidance of the mine truck driver (supervisor). The mining area environment platform provides a large amount of data and training environment for mining area scenes. For multi-vehicle collaborative tasks and texts that do not exist in the urban unmanned driving scene, when the mining area model is trained based on the first training data of the urban scene, only the modules corresponding to the urban unmanned driving scene in the mining area model (multimodal fusion module and task output head) are called, and other modules do not work or update (language processing module and collaborative planning module). When training the large model of the mining area, the corresponding unmanned driving subtasks are different according to the different sample data, so different output heads output data. Each time the large model of the mining area is updated based on a set of training data, in addition to updating other modules, for the output head part, only the output head corresponding to the unmanned driving subtask of this training data is updated, that is, only the output head with output data this time is updated, and the output head without output data this time is not updated. In this way, the various output heads of the large model of the mining area can be applied to different unmanned driving subtasks respectively, but the parameters of other main parts of the large model of the mining area can be universal.
[0086] The pre-trained mining area large model is trained based on multiple sets of second training data, including:
[0087] Inputting the second training data into the large model of the mining area, and obtaining a training unmanned driving guidance signal determined based on the output of the large model of the mining area;
[0088] Obtaining human evaluation data for training unmanned driving guidance signals;
[0089] Update the mining area model based on manual evaluation data.
[0090] When the pre-trained large model of the mining area is trained based on multiple sets of second training data, all modules of the large model of the mining area participate in data processing, and the parameters of all modules of the large model of the mining area are updated. Specifically, in a possible implementation method, the process of training the pre-trained large model of the mining area based on multiple sets of second training data can be divided into two stages. In the first stage, the parameters of the fixed multimodal fusion module and the task output head are unchanged, and only the parameters of the language processing module and the collaborative planning module are updated, so that the parameters of the language processing module and the collaborative planning module can be quickly optimized. In the second stage, the parameters of all modules of the large model of the mining area (language processing module, collaborative planning module, language processing module and collaborative planning module) are updated, and finally the optimization of the parameters of all modules is realized. In this way, the training efficiency can be improved and the training cost can be reduced. The second training data used in the first stage and the second stage can be repeated, so that the utilization rate of the second training data can be improved, the influence due to the small number of mining area scene data can be reduced, and the utilization of mining area scene data can be maximized.
[0091] The method provided by the present invention first uses urban scene data to train the large model of the mining area, and then performs secondary training based on the mining area performance, so that the large model of the mining area can adapt to the multi-cooperative scene of the mining area. In the process of secondary training, further, since the amount of unmanned driving label data in the mining area scene is small, in order to avoid the cost of a large amount of labeling, the method provided by the present invention obtains the training loss update model through manual evaluation.
[0092] The following is a description of the unmanned driving guidance device for a mine area based on a large model system provided by the present invention. The unmanned driving guidance device for a mine area based on a large model system described below and the unmanned driving guidance method for a mine area based on a large model system described above can be referred to each other. Figure 5 As shown, the device provided by the present invention includes:
[0093] The language processing unit 510 is used to obtain the guidance language text data of the target mining area vehicle, input the guidance language text data into the language processing module in the mining area large model, and obtain the natural language features output by the language processing module;
[0094] The multimodal fusion unit 520 is used to obtain the environmental sensor data of the target mining area vehicle, the environmental sensor data includes environmental image data and sensor signal data, the sensor signal data comes from multiple sensors, and inputs the environmental sensor data and natural language features into the multimodal fusion module in the mining area large model to obtain the fusion features output by the multimodal fusion module;
[0095] The subtask processing unit 530 is used to determine the current target unmanned driving subtask, input the fusion feature into the task output head corresponding to the target unmanned driving subtask in the mining area large model, and obtain the task output output by the task output head;
[0096] An unmanned driving guidance unit 540, configured to determine an unmanned driving guidance signal based on the task output, and send the unmanned driving guidance signal to an on-board control device of a target mining area vehicle;
[0097] The manual intervention unit 550 is used to obtain manual intervention data and send the manual intervention data to the vehicle-mounted control device. The priority of the manual intervention data is higher than the unmanned driving guidance signal.
[0098] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor (processor) 610, a communication interface (Communications Interface) 620, a memory (memory) 630 and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call the logic instructions in the memory 630 to execute the unmanned driving guidance method for the mine area based on the large model system, which includes: obtaining the guidance language text data of the target mine area vehicle, inputting the guidance language text data into the language processing module in the mine area large model, and obtaining the natural language features output by the language processing module; obtaining the environmental sensor data of the target mine area vehicle, the environmental sensor data includes environmental image data and sensor signal data, and the sensor signal data comes from multiple sensors, inputting the environmental sensor data and natural language features into the multimodal fusion module in the mine area large model, and obtaining the fusion features output by the multimodal fusion module; determining the current target unmanned driving subtask, inputting the fusion features into the task output head corresponding to the target unmanned driving subtask in the mine area large model, and obtaining the task output output by the task output head; determining the unmanned driving guidance signal based on the task output, and sending the unmanned driving guidance signal to the on-board control device of the target mine area vehicle; obtaining manual intervention data, and sending the manual intervention data to the on-board control device, and the priority of the manual intervention data is higher than the unmanned driving guidance signal.
[0099] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0100] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the unmanned driving guidance method for a mine area based on a large model system provided by the above methods. The method includes: obtaining guidance language text data of a target mine area vehicle, inputting the guidance language text data into a language processing module in the mine area large model, and obtaining natural language features output by the language processing module; obtaining environmental sensor data of the target mine area vehicle, the environmental sensor data including environmental image data and sensor signal data , the sensor signal data comes from a variety of sensors, and the environmental sensor data and natural language features are input into the multimodal fusion module in the large model of the mining area to obtain the fusion features output by the multimodal fusion module; the current target unmanned driving subtask is determined, and the fusion features are input into the task output head corresponding to the target unmanned driving subtask in the large model of the mining area to obtain the task output output by the task output head; the unmanned driving guidance signal is determined based on the task output, and the unmanned driving guidance signal is sent to the on-board control device of the target mining area vehicle; the manual intervention data is obtained, and the manual intervention data is sent to the on-board control device, and the priority of the manual intervention data is higher than the unmanned driving guidance signal.
[0101] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented by a processor to execute the unmanned driving guidance method for a mine area based on a large model system provided by the above methods when the computer program is executed, the method comprising: obtaining guidance language text data of a target mine area vehicle, inputting the guidance language text data into a language processing module in a large model of the mine area, and obtaining natural language features output by the language processing module; obtaining environmental sensor data of the target mine area vehicle, the environmental sensor data including environmental image data and sensor signal data, the sensor signal data coming from a variety of sensors, inputting the environmental sensor data and natural language features into a multimodal fusion module in a large model of the mine area, and obtaining fusion features output by the multimodal fusion module; determining the current target unmanned driving subtask, inputting the fusion features into a task output header corresponding to the target unmanned driving subtask in the large model of the mine area, and obtaining the task output output by the task output header; determining an unmanned driving guidance signal based on the task output, and sending the unmanned driving guidance signal to a vehicle-mounted control device of the target mine area vehicle; obtaining manual intervention data, and sending the manual intervention data to the vehicle-mounted control device, the manual intervention data having a higher priority than the unmanned driving guidance signal.
[0102] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0103] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for unmanned driving guidance in a mine area based on a large model system, characterized in that: include: Obtaining guidance language text data of vehicles in the target mining area, inputting the guidance language text data into a language processing module in the mining area large model, and obtaining natural language features output by the language processing module; Acquire environmental sensor data of vehicles in the target mining area, wherein the environmental sensor data includes at least one of environmental image data and laser radar point cloud data, and the environmental sensor data comes from multiple sensors, input the environmental sensor data and the natural language features into a multimodal fusion module in the large mining area model, and acquire fusion features output by the multimodal fusion module; Determine the current target unmanned driving task, input the fusion feature into the task output head corresponding to the target unmanned driving task in the large model of the mining area, and obtain the task output output by the task output head; Determine an unmanned driving guidance signal based on the task output, and send the unmanned driving guidance signal to an on-board control device of a vehicle in the target mining area; A manual intervention signal is obtained and sent to the vehicle-mounted control device, wherein the manual intervention signal has a higher priority than the unmanned driving guidance signal.
2. The unmanned driving guidance method for a mine area based on a large model system according to claim 1 is characterized in that: The step of determining an unmanned driving guidance signal based on the task output comprises: Acquire a collaborative task corresponding to the target mining area vehicle, and determine a collaborative vehicle for the collaborative task, wherein the collaborative vehicle includes the target mining area vehicle; Input the communication relationship data between the cooperative vehicles, the vehicle information data of the cooperative vehicles and the task data of the cooperative task into the cooperative planning module of the mining area large model, and obtain the cooperative planning data output by the cooperative planning module; The unmanned driving guidance signal is determined based on the collaborative planning data and the task output.
3. The unmanned driving guidance method for a mine area based on a large model system according to claim 1 is characterized in that: After the unmanned driving guidance signal is sent to the vehicle-mounted control device of the target mining area vehicle, the method includes: Acquire the driving data of the target mining area vehicle, and update the visual simulation model of the mining area where the target mining area vehicle is located based on the driving data.
4. The method for unmanned driving guidance in a mine area based on a large model system according to claim 1 is characterized in that: After sending the manual intervention signal to the vehicle-mounted control device, the method further includes: The large mining area model is updated based on the manual intervention signal.
5. The method for unmanned driving guidance in a mine area based on a large model system according to claim 2 is characterized in that: The training process of the mining area large model includes: The mining area large model is trained based on multiple groups of first training data, and the multimodal fusion module and the task output head in the mining area large model are updated to obtain the pre-trained mining area large model, wherein the first training data includes sample environment sensor data and corresponding task output labels, and the first training data is obtained based on an urban driving environment; Training the pre-trained large mining area model based on multiple sets of second training data, wherein the second training data includes sample input data, the sample input data includes sample guiding language text data, sample environmental sensor data, sample communication relationship data, sample vehicle information data and sample task data, and the second training data is obtained based on the mining area driving environment; Wherein, when the large mining area model is trained, the corresponding task output head is updated according to the unmanned driving task corresponding to the input into the large mining area model.
6. The method for unmanned driving guidance in a mine area based on a large model system according to claim 5 is characterized in that: The step of training the pre-trained large mining area model based on the plurality of sets of the second training data comprises: Inputting the second training data into the large model of the mining area, and obtaining a training unmanned driving guidance signal determined based on an output of the large model of the mining area; Acquiring manual evaluation data for the training unmanned driving guidance signal; The large mining area model is updated based on the manual evaluation data.
7. A mining area confirmation driving guidance device based on a large model system, characterized in that: include: A language processing unit, used to obtain the guidance language text data of the target mining area vehicle, input the guidance language text data into the language processing module in the mining area large model, and obtain the natural language features output by the language processing module; A multimodal fusion unit is used to obtain environmental sensor data of vehicles in the target mining area, wherein the environmental sensor data includes at least one of environmental image data and laser radar point cloud data, and the environmental sensor data comes from multiple sensors, and inputs the environmental sensor data and the natural language features into the multimodal fusion module in the large mining area model to obtain fusion features output by the multimodal fusion module; A subtask processing unit, used for determining a current target unmanned driving task, inputting the fusion feature into a task output header corresponding to the target unmanned driving task in the large mining area model, and obtaining a task output output by the task output header; An unmanned driving guidance unit, configured to determine an unmanned driving guidance signal based on the task output, and send the unmanned driving guidance signal to an on-board control device of a vehicle in the target mining area; The manual intervention unit is used to obtain a manual intervention signal and send the manual intervention signal to the vehicle-mounted control device, wherein the priority of the manual intervention signal is higher than the unmanned driving guidance signal.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the mining area confirmation driving guidance method based on the large model system as described in any one of claims 1 to 6 is implemented.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the mining area confirmation driving guidance method based on a large model system as described in any one of claims 1 to 6 is implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the mining area confirmation driving guidance method based on a large model system as described in any one of claims 1 to 6 is implemented.
