Mine unmanned driving virtual scene generation method based on large model

Abstract

The invention belongs to the field of mine unmanned driving virtual scene generation, and discloses a mine unmanned driving virtual scene generation method based on a large model, which comprises the following steps: collecting network information and mine data, fusing the network data and a mine data set, forming a pre-training data set, and obtaining a sample Token; the sample Token is subjected to two stages of pre-training and weight cyclic fine tuning in sequence, feedback correction and cyclic tuning until a final model generated by the mine unmanned driving virtual scene is output; and inputting a to-be-trained unmanned driving algorithm and a required text to the final model, automatically generating a marked mine unmanned driving data set, scoring a data set generation result, feeding back to the weight cycle fine tuning stage, providing a scoring reference for the data set, and performing cycle tuning to obtain a mine virtual generation final model. According to the method, the problems of data shortage, high data labeling cost, insufficient training data diversity, insufficient virtual scene fidelity, high equipment cost and the like of the mine unmanned driving technology are solved.

Description

Technical Field

[0001] The present invention relates to the technical field related to generation of unmanned driving virtual scenes in mines, and in particular to a method for generating unmanned driving virtual scenes in mines based on a large model. Background Art

[0002] Unlike road unmanned driving, the mining environment faced by unmanned driving technology in mines is complex and changeable. The geology, climate, and operating procedures of different mines vary significantly, making it difficult to obtain comprehensive and diverse data. At the same time, the complexity of the mining environment also increases the difficulty of data labeling. The unmanned driving system in mines also needs to adapt to a variety of working conditions and scenarios, but the existing training data is often limited to specific scenarios and lacks diversity. In addition, the existing simulation environment for unmanned driving in mines often cannot fully simulate the complexity and variability of real mines, resulting in the performance of models trained in the simulation environment may be degraded in actual applications.

[0003] Therefore, the current unmanned driving technology in mines faces difficulties such as lack of data, high data annotation costs, insufficient diversity of training data, insufficient fidelity of virtual scenes, and high equipment costs, which greatly restrict its promotion and application in actual mining scenarios. At present, it is urgent to design a method for generating virtual scenes for unmanned driving in mines based on large models, so as to utilize deep learning large model technology to generate highly realistic virtual scenes and image data of mining environments, and to dynamically generate training data that meets actual needs by simulating diverse mining environments (such as terrain, climate, equipment, lighting, etc.), thereby reducing dependence on real mining environment data and reducing collection and annotation costs. Summary of the invention

[0004] In response to the above defects or improvement needs of the prior art, the present invention provides a method for generating a virtual scene for unmanned driving in mines based on a large model, thereby solving the technical problems faced by current unmanned driving technology in mines, such as lack of data, high data annotation cost, insufficient diversity of training data, and insufficient realism of virtual scenes.

[0005] To achieve the above object, according to one aspect of the present invention, a method for generating a virtual scene of an unmanned mine driving based on a large model is provided, comprising the following steps:

[0006] S1: Data collection, collecting network information and mine data. Network information comes from crawling network data of communication networks. Mine data is obtained through vehicle-mounted vision, surveillance cameras and manual collection to form a mine data set. The network data and the mine data set are integrated to form a pre-training data set and obtain sample tokens.

[0007] S2: Model training. Based on the sample tokens obtained in step S1, the models are pre-trained in sequence to obtain the basic model for virtual generation of mines. The basic model enters the weight cycle fine-tuning stage, and feedback correction and cycle tuning are performed until the final model for generating the virtual scene of unmanned driving in mines is output;

[0008] S3: Data generation, input the unmanned driving algorithm to be trained and the required text into the final model obtained in step S2. The final model automatically generates a dataset with labeled mine unmanned driving; the dataset generation results are scored and fed back to the weight cycle fine-tuning stage in step S2 to provide a scoring reference for it and obtain the final model of mine virtual generation after cyclic optimization.

[0009] S4: In actual use, the final model of the mine virtualization generated in step S3 is combined to obtain a dataset of labeled mine unmanned driving and combined with scores. The content of this dataset covers differentiated images simulating changes in the mine environment, images of different points in the mine, and simulated images of various obstacles during operation, in order to cope with various standard problems faced by mine unmanned driving.

[0010] Preferably, the network data in step S1 is multimodal information related to unmanned driving in mines;

[0011] The method for obtaining the sample Token in step S1 is to classify and sort the fused network information and the mining data set, unitize the samples, differentiate them into the smallest units that can be processed by the model, and obtain the sample Token of the corresponding class.

[0012] Preferably, the method further includes data set screening before pre-training, wherein the sample tokens include audio tokens and image tokens, and the data set screening includes the following steps:

[0013] S211: Convert audio tokens into text tokens for comparison and screening to remove duplicate and invalid texts;

[0014] S212: Perform scene recognition on image tokens based on semantic tags to exclude non-mine scenes;

[0015] S213: The two types of tokens selected in steps S211 and S212 are respectively imported into the text or image encoder, and hybrid decoding is performed by the hybrid decoder to obtain a pre-training data set.

[0016] Preferably, in step S2, the pre-training outputs the basic model for virtual generation of the mine, and the weight loop fine-tuning stage consists of three parts: supervised fine-tuning, reward modeling, and reinforcement learning, and the model weights are fine-tuned through a loop structure.

[0017] Preferably, the specific steps of model training in step S2 are:

[0018] Based on the sample Token obtained in step S1, the data set is screened to obtain a pre-trained data set, the pre-trained data set is pre-trained, and the basic model of mine virtual generation is iteratively output. The weights are fine-tuned through a loop structure by supervised fine-tuning, reward modeling and reinforcement learning. The mine data set is manually labeled to form a high-quality mine data set. The supervised fine-tuning receives its information, learns the human labeling instructions, and outputs weights to reward modeling. Reward modeling learns the excellent values ​​of human beings for data through feedback, and further outputs weights to reinforcement learning. Reinforcement learning evaluates the quality of the basic model, and feedback corrections and cyclic tuning are performed until the final model of mine virtual generation is output.

[0019] Preferably, the supervised fine-tuning sets a generation function on the framework of the basic model, obtains a high-quality mining data set, and generates virtual text and corresponding virtual images by the data generator to form a virtual data set F, which is mixed with the real data set R for simulation discrimination. The relationship between the two is:

[0020]

[0021] Among them, i is the same type of text, R i With F i Respectively represent the image matching probability distribution of real data and virtual data on text i;

[0022] If the L(R,F) value gradually decreases to convergence and is lower than the preset threshold, the weight is output; otherwise, it means distortion occurs, that is, the gap between the virtual data and the real data is very large, then the model weight is returned for fine-tuning.

[0023] Preferably, the reward modeling sets a logical association function on the framework of the basic model, receives model weights to form a logical association network H, the logical association network is initially self-generated by the model, and is corrected by manual scoring results, performs logical judgment on the generated text and image, and performs logical association evaluation on the quality of image generation. If it is a weak association, the logical associator is returned to fine-tune the weight; otherwise, the weight is output.

[0024] Preferably, the reinforcement learning is performed as a cyclic connection structure to iteratively perform supervised fine-tuning and reward modeling, optimize the model weights, score the comprehensive quality of the generated text and image according to L(R, F), feedback the low-quality weight greater than the preset threshold and output the high-quality weight, and its adjustment strategy is:

[0025] D=Q′(s,a,s′)-Q(s,a,s′)=H(s′)-H(s)

[0026] Among them, s is the current weight state, a is the weight adjustment change, s′ is the next weight state, D represents the offset value of the model from the ideal state, Q represents the weight adjustment strategy, H represents the potential energy function of the logical association network, representing the distance between the current state and the target state, and the model iteration goal is to make D approach 0, and finally obtain the final model of mine virtual generation.

[0027] In general, the above technical solutions conceived by the present invention have the following beneficial effects compared with the prior art:

[0028] 1. The present invention proposes a method for generating a virtual scene for unmanned driving in mines based on a large model. This system is aimed at unmanned driving algorithms in mining scenarios, and provides them with high-quality, diverse and annotated image data sets. The system adopts a large model framework, and with its excellent comprehensive performance, it makes the generated images more in line with the needs of unmanned driving algorithms in mines. In addition, the large model has a unique advantage in natural language understanding, and can imitate and learn human logical behavior to generate image data that is more in line with human logic. At the same time, the generation of virtual scenes is not affected by the external environment, and can quickly and massively produce the required images, and can annotate them by themselves, greatly reducing the manpower and time costs of data collection. In actual use, you only need to enter the required text to generate annotated image data sets by yourself.

[0029] 2 The present invention proposes a model pre-training scheme based on classification coding-hybrid decoding. Different from the traditional classification coding-decoding, this system performs hybrid decoding on the text and image data based on classification decoding. On the one hand, hybrid decoding takes into account both traditional decoding and data association, improving the efficiency of pre-training; on the other hand, it logically associates text and image data for the first time in the process, laying the foundation for subsequent association reinforcement. In addition, in order to reduce the cost of pre-training, this scheme uses a fixed number of iterations instead of the common loss function iteration. Therefore, the output basic model does not have the ability to work independently, and some optimizations are allocated to subsequent steps to improve the continuity of model training.

[0030] 3 The present invention proposes a weight cyclic fine-tuning system based on a reward mechanism, which constructs a cyclic structure consisting of supervised fine-tuning, reward modeling, and reinforcement learning, and uses a reward mechanism based on external feedback to allocate the three-stage capability optimization. Supervised fine-tuning achieves realistic generation of text / images through reward feedback of high-quality data sets; reward modeling receives manual scoring and imitates human logic to judge the quality of generated data; reinforcement learning comprehensively evaluates the quality of data generation and performs iterative optimization as a cyclic connection structure. Allocating the model's capability optimization to different sections can effectively optimize details for different types of problems and facilitate the professional adjustment of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flow chart of a method for generating a virtual scene of an unmanned mine driving based on a large model constructed according to a preferred embodiment of the present invention;

[0032] Figure 2 It is a schematic diagram of a model pre-training process based on classification coding-hybrid decoding constructed according to a preferred embodiment of the present invention;

[0033] Figure 3 It is a schematic diagram of the structure of a weight cycle fine-tuning system based on a reward mechanism constructed according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0035] like Figure 1 As shown, a method and working method for generating a virtual scene of unmanned driving in a mine based on a large model include:

[0036] S1: Data collection, collecting network information and mine data. Network information comes from crawling network data of communication networks. Mine data is obtained through vehicle-mounted vision, surveillance cameras and manual collection to form a mine data set. The network data and the mine data set are integrated to form a pre-training data set and obtain sample tokens.

[0037] The network data in step S1 is multimodal information related to unmanned driving in mines, including text, audio, image and other multimodal data. The network information comes from crawling network data of communication networks to obtain relevant papers, public data sets, related codes and other multimodal information.

[0038] The mine data in step S1 is obtained by obtaining visual information through vehicle-mounted vision, surveillance cameras, and manual collection (drone aerial photography, handheld camera photography, etc.), forming a mine data set. This type of data is all image-based;

[0039] The method for obtaining the sample Token in step S1 is to classify and sort the fused network information and the mining data set, unitize the samples, differentiate them into the smallest units that can be processed by the model, and obtain the sample Token of the corresponding class.

[0040] S2: Model training, such as Figure 2As shown, based on the sample tokens obtained in step S1, the models are pre-trained in sequence to obtain the basic model for virtual generation of the mine, and the basic model enters the weight cycle fine-tuning stage, and feedback correction and cycle tuning are performed until the final model for generating the virtual scene of unmanned driving in the mine is output;

[0041] Sample tokens include audio tokens and image tokens. Before pre-training, the sample tokens are first screened for data sets.

[0042] Dataset screening includes the following steps:

[0043] S211: Convert audio tokens into text tokens for comparison and screening to remove duplicate and invalid texts;

[0044] S212: Perform scene recognition on image tokens based on semantic tags to exclude non-mine scenes;

[0045] S213: The two types of tokens selected in steps S211 and S212 are respectively imported into the text or image encoder, and hybrid decoding is performed by the hybrid decoder to obtain a pre-training data set.

[0046] In step S2, the pre-training outputs the basic model for virtual generation of the mine. The weight loop fine-tuning stage consists of three parts: supervised fine-tuning, reward modeling, and reinforcement learning. The model weights are fine-tuned through a loop structure to form a loop structure. The three are each assigned different task objectives. Supervised fine-tuning performs human instruction understanding and compares and associates human descriptions with graphics; reward modeling performs human value understanding, performs logical association based on the feedback of manual ratings, and performs logical judgment on the quality of generated images; reinforcement learning is used as a loop connection structure for iterative optimization.

[0047] That is, the specific steps of model training in step S2 are:

[0048] Based on the sample Token obtained in step S1, the data set is screened to obtain a pre-trained data set, the pre-trained data set is pre-trained, and the basic model of mine virtual generation is iteratively output. The weights are fine-tuned through a loop structure by supervised fine-tuning, reward modeling and reinforcement learning. The mine data set is manually labeled to form a high-quality mine data set. The supervised fine-tuning receives its information, learns the human labeling instructions, and outputs weights to reward modeling. Reward modeling learns the excellent values ​​of human beings for data through feedback, and further outputs weights to reinforcement learning. Reinforcement learning evaluates the quality of the basic model, and feedback corrections and cyclic tuning are performed until the final model of mine virtual generation is output.

[0049] The supervised fine-tuning sets a generation function on the framework of the basic model. The mine data set obtained in step S1 is manually annotated to obtain a high-quality mine data set, which is divided into two types of data: text and image.

[0050] The high-quality mining data set is obtained by generating virtual text and corresponding virtual images by the data generator to form a virtual data set (F), which is mixed with the real data set (R) for simulation discrimination. The relationship between the two is:

[0051]

[0052] Among them, i is the same type of text, R i With F i They represent the image matching probability distribution of real data and virtual data on text i respectively; if the L(R,F) value gradually decreases to convergence and is lower than the preset threshold, which is preferably 0.3, the weight is output; otherwise, it means distortion occurs, that is, the gap between virtual data and real data is very large, then the model weight is returned for fine-tuning.

[0053] like Figure 3 As shown, the reward modeling sets a logical association function on the framework of the basic model, receives the model weights to form a logical association network H, the logical association network is initially self-generated by the model, and is modified by the manual scoring results, and logical judgments are made on the generated text and images, and logical association evaluation is performed on the quality of image generation. If it is a weak association, the logical associator is returned to fine-tune the weight; otherwise, the weight is output.

[0054] The reinforcement learning is used as a cyclic connection structure to iteratively perform supervised fine-tuning and reward modeling, optimize the model weights, score the comprehensive quality of the generated text and image according to L(R, F), feedback the low-quality weight greater than the preset threshold, which is preferably 0.15, and output the high-quality weight, and its adjustment strategy is:

[0055] D=Q′(s,a,s′)-Q(s,a,s′)=H(s′)-H(s)

[0056] Among them, s is the current weight state, a is the weight adjustment change, s′ is the next weight state, D represents the offset value of the model from the ideal state, Q represents the weight adjustment strategy, H represents the potential energy function of the logical association network, representing the distance between the current state and the target state, and the model iteration goal is to make D approach 0, and finally obtain the final model generated by the mine unmanned driving virtual scene.

[0057] S3: Data generation, input the unmanned driving algorithm to be trained and the required text into the final model obtained in step S2. The final model automatically generates a dataset with labeled mine unmanned driving; the dataset generation results are scored and fed back to the weight cycle fine-tuning stage in step S2 to provide a split reference and obtain the final model of mine virtual generation after cyclic tuning.

[0058] The data generation process only needs to input the required text to generate the final model of the mine virtualization, and then generate the labeled image data set. The data set is scored by humans or machines and fed back to the logical association network.

[0059] S4: In actual use, combined with the final model generated by the mine virtualization in step S3, a dataset of labeled mine unmanned driving and combined with scores is obtained. The content of this dataset covers differentiated images simulating changes in the mine environment, images of different points in the mine, and simulated images of various obstacles during operation, which correspond to various standard problems faced by mine unmanned driving. Corresponding to the three important problems of harsh environmental interference, equipment spatial positioning, and task path planning in mine unmanned driving, the dataset images include three types of images: simulated environmental changes, different mine points, and operation obstacle simulation.

[0060] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating a virtual scene for unmanned driving in a mine based on a large model, characterized in that: The following steps are involved: S1: Data collection, collecting network information and mine data. Network information comes from crawling network data in the communication network. Mine data is obtained through vehicle-mounted vision, surveillance cameras, and manual collection to form a mine data set. The network data and the mine data set are integrated to form a pre-training data set and obtain sample tokens. S2: Model training. Based on the sample tokens obtained in step S1, the models are pre-trained in sequence to obtain the basic model for virtual generation of mines. The basic model enters the weight cycle fine-tuning stage, and feedback correction and cycle tuning are performed until the final model for generating the virtual scene of unmanned driving in mines is output; S3: Data generation, input the unmanned driving algorithm to be trained and the required text into the final model obtained in step S2, and the final model automatically generates a dataset with labeled mine unmanned driving; score the dataset generation results and feed them back to the weight cycle fine-tuning stage in step S2 to provide a scoring reference for it and obtain the final model of mine virtual generation after cyclic tuning; S4: In actual use, the final model of the mine virtualization generated in step S3 is combined to obtain a dataset of labeled mine unmanned driving and combined with scores. The content of this dataset covers differentiated images simulating changes in the mine environment, images of different points in the mine, and simulated images of various obstacles during operation, in order to cope with various standard problems faced by mine unmanned driving.

2. The method for generating a large-scale model-based unmanned driving virtual scene in a mine as claimed in claim 1, characterized in that: The network data in step S1 is multimodal information related to unmanned driving in mines; The method for obtaining the sample Token in step S1 is to classify and sort the fused network information and the mining data set, unitize the samples, differentiate them into the smallest units that can be processed by the model, and obtain the sample Token of the corresponding class.

3. The method for generating a large-scale model-based unmanned driving virtual scene in a mine as claimed in claim 1, characterized in that: It also includes the data set screening before pre-training. The sample tokens include audio tokens and image tokens. The data set screening includes the following steps: S211: Convert audio tokens into text tokens for comparison and screening to remove duplicate and invalid texts; S212: Perform scene recognition on image tokens based on semantic tags to exclude non-mine scenes; S213: The two types of tokens selected in steps S211 and S212 are respectively imported into the text or image encoder, and hybrid decoding is performed by the hybrid decoder to obtain a pre-training data set.

4. The method for generating a large-scale model-based unmanned driving virtual scene in a mine as claimed in claim 3, characterized in that: In step S2, the pre-training output mine virtual generation basic model, the weight loop fine-tuning stage consists of three parts: supervised fine-tuning, reward modeling and reinforcement learning, and the model weights are fine-tuned through the loop structure.

5. The method for generating a large-scale model-based unmanned driving virtual scene in a mine as claimed in claim 4, characterized in that: The specific steps of model training in step S2 are: Based on the sample Token obtained in step S1, the data set is screened to obtain a pre-trained data set, the pre-trained data set is pre-trained, and the basic model of mine virtual generation is iteratively output. The weights are fine-tuned through a loop structure by supervised fine-tuning, reward modeling and reinforcement learning. The mine data set is manually labeled to form a high-quality mine data set. The supervised fine-tuning receives its information, learns the human labeling instructions, and outputs weights to reward modeling. Reward modeling learns the excellent values ​​of human beings for data through feedback, and further outputs weights to reinforcement learning. Reinforcement learning evaluates the quality of the basic model, and feedback corrections and cyclic tuning are performed until the final model of mine virtual generation is output.

6. The method for generating a large-scale model-based unmanned driving virtual scene in a mine as claimed in claim 5, characterized in that: The supervised fine-tuning sets a generation function on the framework of the basic model, and obtains a high-quality mining data set. The data generator generates virtual text and corresponding virtual images to form a virtual data set F, which is mixed with the real data set R for simulation discrimination. The relationship between the two is: Among them, i is the same type of text, R i With F i Respectively represent the image matching probability distribution of real data and virtual data on text i; If the L(R,F) value gradually decreases to convergence and is lower than the preset threshold, the weight is output; otherwise, it means distortion occurs, that is, the gap between the virtual data and the real data is very large, then the model weight is returned for fine-tuning.

7. The method for generating a large-scale model-based unmanned driving virtual scene in a mine as claimed in claim 6, characterized in that: The reward modeling sets a logical association function on the framework of the basic model, receives the model weights to form a logical association network H, which is initially self-generated by the model and modified by the manual scoring results, performs logical discrimination on the generated text and image, and performs logical association evaluation on the quality of image generation. If it is a weak association, the logical associator is returned to fine-tune the weight; Otherwise, the output weight is given.

8. The method for generating a large-scale model-based unmanned driving virtual scene in a mine as claimed in claim 6, characterized in that: The reinforcement learning is a cyclic connection structure that iteratively performs supervised fine-tuning and reward modeling, optimizes model weights, scores the comprehensive quality of generated text and images according to L(R, F), feeds back low-quality weights greater than a preset threshold and outputs high-quality weights. Its adjustment strategy is: D=Q′(s,a,s′)-Q(s,a,s′)=H(s′)-H(s) Among them, s is the current weight state, a is the weight adjustment change, s′ is the next weight state, D represents the offset value of the model from the ideal state, Q represents the weight adjustment strategy, H represents the potential energy function of the logical association network, representing the distance between the current state and the target state, and the model iteration goal is to make D approach 0, and finally obtain the final model of mine virtual generation.

Images