A method for generating virtual scenes for unmanned driving in mines based on large models

Abstract

The present invention belongs to the field of generating virtual scenes for unmanned driving in mines. It discloses a method for generating virtual scenes for unmanned driving in mines based on a large model, including collecting network information and mine data, fusing the network data with the mine data set to form a pre-training data set, and obtaining sample tokens; the sample tokens are sequentially pre-trained and weighted in two stages, followed by feedback correction and cyclic optimization until the final model for generating the virtual scene for unmanned driving in mines is output; the unmanned driving algorithm to be trained and the required text are input into the final model, and a data set with labeled unmanned driving in mines is automatically generated. The data set generation results are scored and fed back to the weighted cyclic optimization stage to provide a scoring reference for it. After cyclic optimization, the final model for virtual generation of mines is obtained. The present invention solves the difficulties faced by unmanned driving technology in mines, such as lack of data, high data annotation costs, insufficient training data diversity, insufficient virtual scene fidelity, and high equipment costs.

Description

Technical Field

[0001] The present invention relates to the technical field related to generation of unmanned driving virtual scenes in mines, and in particular to a method for generating unmanned driving virtual scenes in mines based on a large model. Background Art

[0002] Unlike road-based autonomous driving, autonomous driving technology in mines faces a complex and volatile environment. The geology, climate, and operational processes of different mines vary significantly, making it difficult to obtain comprehensive and diverse data. The complexity of the mining environment also increases the difficulty of data labeling. Autonomous driving systems in mines also need to adapt to a variety of working conditions and scenarios, but existing training data is often limited to specific scenarios and lacks diversity. Furthermore, existing simulation environments for autonomous driving in mines often fail to fully simulate the complexity and variability of real mines, resulting in models trained in these environments potentially experiencing performance degradation in real-world applications.

[0003] Therefore, current unmanned mining technology faces challenges such as data scarcity, high data annotation costs, insufficient training data diversity, insufficient virtual scene fidelity, and high equipment costs, significantly limiting its promotion and application in actual mining scenarios. There is an urgent need to design a large-scale model-based virtual scene generation method for unmanned mining. This method, which leverages deep learning large-scale model technology to generate highly realistic virtual scenes and image data of mining environments, can dynamically generate training data that meets actual needs by simulating diverse mining environments (such as terrain, climate, equipment, and lighting), reducing reliance on real-world mining environment data and lowering acquisition and annotation costs. Summary of the Invention

[0004] In response to the above-mentioned defects or improvement needs of the existing technology, the present invention provides a method for generating virtual scenes for unmanned driving in mines based on a large model, thereby solving the technical problems faced by current unmanned driving technology in mines, such as lack of data, high data annotation costs, insufficient diversity of training data, and insufficient realism of virtual scenes.

[0005] To achieve the above object, according to one aspect of the present invention, a method for generating a virtual scene for unmanned driving in a mine based on a large model is provided, comprising the following steps:

[0006] S1: Data collection: collect network information and mine data. Network information comes from crawling network data from the communication network. Mine data is obtained through vehicle-mounted vision, surveillance cameras, and manual collection to form a mine data set. The network data and the mine data set are integrated to form a pre-training data set and obtain sample tokens.

[0007] S2: Model training: Based on the sample tokens obtained in step S1, the model is pre-trained in sequence to obtain the basic model for virtual mine generation. The basic model enters the weight cyclic fine-tuning stage, and feedback correction and cyclic tuning are performed until the final model for generating the mine unmanned driving virtual scene is output;

[0008] S3: Data generation: Input the unmanned driving algorithm to be trained and the required text into the final model obtained in step S2. The final model automatically generates a dataset with labeled mine unmanned driving; the dataset generation results are scored and fed back to the weight cyclic fine-tuning stage in step S2 to provide a scoring reference and obtain the final model of mine virtual generation after cyclic tuning.

[0009] S4: In actual use, combined with the final model of the mine virtualization generated in step S3, a dataset of labeled mine unmanned driving and combined with scores is obtained. The content of this dataset covers differentiated images simulating changes in the mine environment, images of different points in the mine, and simulated images of various obstacles during operation, in order to deal with various standard problems faced by mine unmanned driving.

[0010] Preferably, the network data in step S1 is multimodal information related to unmanned driving in mines;

[0011] The method for obtaining sample tokens in step S1 is to classify and organize the fused network information and the mine dataset, unitize the samples, differentiate them into the smallest units that can be processed by the model, and obtain sample tokens of the corresponding class.

[0012] Preferably, the method further includes data set screening before pre-training, wherein the sample tokens include audio tokens and image tokens, and the data set screening includes the following steps:

[0013] S211: Convert audio tokens into text tokens for comparison and screening to remove duplicate and invalid texts;

[0014] S212: Perform scene recognition on the image token based on the semantic label, excluding non-mining scenes;

[0015] S213: The two types of tokens selected in steps S211 and S212 are respectively imported into the text or image encoder, and hybrid decoding is performed by the hybrid decoder to obtain a pre-training dataset.

[0016] Preferably, in step S2, the pre-training outputs the basic model for virtual generation of the mine, and the weight loop fine-tuning stage consists of three parts: supervised fine-tuning, reward modeling, and reinforcement learning, and the model weights are fine-tuned through a loop structure.

[0017] Preferably, the specific steps of model training in step S2 are:

[0018] Based on the sample Token obtained in step S1, the data set is screened to obtain a pre-training data set, the pre-training data set is pre-trained, and the basic model of mine virtual generation is iteratively output. The weights are fine-tuned through a loop structure by supervised fine-tuning, reward modeling and reinforcement learning. The mine data set is manually labeled to form a high-quality mine data set. The supervised fine-tuning receives its information, learns the human labeling instructions, and outputs weights to the reward modeling. The reward modeling learns the human value of data excellence through feedback, and further outputs the weights to the reinforcement learning. The reinforcement learning evaluates the quality of the basic model, and feedback correction and cyclic tuning are performed until the final model of mine virtual generation is output.

[0019] Preferably, the supervised fine-tuning sets a generation function on the framework of the basic model, and obtains a high-quality mining data set. The data generator generates virtual text and corresponding virtual images to form a virtual data set F, which is then mixed with the real data set R for realistic discrimination. The relationship between the two is:

[0020]

[0021] Among them, i is the same type of text, R i With F i Represent the image matching probability distribution of real data and virtual data on text i respectively;

[0022] If the L(R,F) value gradually decreases to convergence and is lower than the preset threshold, the weight is output; otherwise, it means distortion occurs, that is, the gap between the virtual data and the real data is very large, and the model weight is returned for fine-tuning.

[0023] Preferably, the reward modeling sets a logical association function on the framework of the basic model, receives the model weights to form a logical association network H, the logical association network is initially self-generated by the model, and is modified by the manual scoring results, and logical judgments are made on the generated text and image, and a logical association evaluation is made on the quality of the image generation. If it is a weak association, the logical associator is returned to fine-tune the weight; otherwise, the weight is output.

[0024] Preferably, the reinforcement learning is performed as a cyclic connection structure to iteratively perform supervised fine-tuning and reward modeling, optimize the model weights, score the comprehensive quality of the generated text and image according to L(R, F), feedback the low-quality weight greater than the preset threshold and output the high-quality weight, and its adjustment strategy is:

[0025] D=Q′(s,a,s′)-Q(s,a,s′)=H(s′)-H(s)

[0026] Among them, s is the current weight state, a is the weight adjustment change, s′ is the next weight state, D represents the offset value of the model from the ideal state, Q represents the weight adjustment strategy, H represents the potential energy function of the logical association network, and represents the distance between the current state and the target state. The model iteration goal is to make D approach 0, and finally obtain the final model of mine virtual generation.

[0027] In general, the above technical solutions conceived by the present invention have the following beneficial effects compared with the prior art:

[0028] 1. The present invention proposes a method for generating a virtual scene for unmanned driving in mines based on a large model. This system is aimed at unmanned driving algorithms in mining scenarios, and provides them with high-quality, diverse and annotated image data sets. The system adopts a large model framework, and with its excellent comprehensive performance, it makes the generated images more in line with the needs of unmanned driving algorithms in mines. In addition, large models have unique advantages in natural language understanding, and can imitate and learn human logical behavior to generate image data that is more in line with human logic. At the same time, the generation of virtual scenes is not affected by the external environment, and can quickly and massively produce the required images, and can self-annotate, which greatly reduces the manpower and time costs of data collection. In actual use, you only need to enter the required text to automatically generate annotated image data sets.

[0029] 2. This invention proposes a model pre-training scheme based on classification coding and hybrid decoding. Unlike traditional classification coding and decoding, this system performs hybrid decoding of text and image data on top of classification decoding. On the one hand, hybrid decoding takes into account both traditional decoding and data association, improving pre-training efficiency. On the other hand, it logically associates text and image data for the first time in the process, laying the foundation for subsequent association reinforcement. In addition, to reduce pre-training costs, this scheme uses a fixed number of iterations rather than the common loss function iteration. Therefore, the output basic model does not have the ability to operate independently, and some optimization is distributed to subsequent steps, improving the consistency of model training.

[0030] This paper proposes a reward-based, cyclical weight fine-tuning system. It constructs a three-part loop structure consisting of supervised fine-tuning, reward modeling, and reinforcement learning. A reward mechanism based on external feedback is used to allocate capacity optimization across these three stages. Supervised fine-tuning achieves realistic text / image generation through reward feedback from high-quality datasets. Reward modeling receives human ratings, imitating human logic to judge the quality of generated data. Reinforcement learning comprehensively evaluates data generation quality, and as a cyclical, connected structure, iteratively optimizes the model. Distributing model capacity optimization to different components allows for effective, detailed optimization for different problem types and facilitates specialized model adjustments. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flowchart of a method for generating a large-scale model-based unmanned driving virtual scene in a mine constructed according to a preferred embodiment of the present invention;

[0032] Figure 2 This is a schematic diagram of a model pre-training process based on classification coding-hybrid decoding constructed according to a preferred embodiment of the present invention;

[0033] Figure 3 It is a structural diagram of a weight cycle fine-tuning system based on a reward mechanism constructed according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0035] like Figure 1 As shown, the method and working method for generating a virtual scene of unmanned driving in a mine based on a large model include:

[0036] S1: Data collection: collect network information and mine data. Network information comes from crawling network data from the communication network. Mine data is obtained through vehicle-mounted vision, surveillance cameras, and manual collection to form a mine data set. The network data and the mine data set are integrated to form a pre-training data set and obtain sample tokens.

[0037] The network data in step S1 is multimodal information related to unmanned driving in mines, specifically including multimodal data such as text, audio, and images. The network information comes from crawling network data of communication networks to obtain multimodal information such as relevant papers, public data sets, and related codes;

[0038] The mine data in step S1 is obtained by obtaining visual information through vehicle-mounted vision, surveillance cameras, and manual collection (drone aerial photography, handheld camera photography, etc.), forming a mine dataset. This type of data is all image-based;

[0039] The method for obtaining sample tokens in step S1 is to classify and organize the fused network information and the mine dataset, unitize the samples, differentiate them into the smallest units that can be processed by the model, and obtain sample tokens of the corresponding class.

[0040] S2: Model training, such as Figure 2As shown, based on the sample tokens obtained in step S1, the model is pre-trained in sequence to obtain the basic model for virtual generation of the mine. The basic model enters the weight cycle fine-tuning stage, and feedback correction and cycle tuning are performed until the final model for generating the unmanned driving virtual scene of the mine is output;

[0041] Sample Tokens include audio Tokens and image Tokens. Before pre-training, the sample Tokens are first screened for data sets.

[0042] Dataset screening includes the following steps:

[0043] S211: Convert audio tokens into text tokens for comparison and screening to remove duplicate and invalid texts;

[0044] S212: Perform scene recognition on the image token based on the semantic label, excluding non-mining scenes;

[0045] S213: The two types of tokens selected in steps S211 and S212 are respectively imported into the text or image encoder, and hybrid decoding is performed by the hybrid decoder to obtain a pre-training dataset.

[0046] In step S2, the pre-training outputs the basic model for virtual generation of the mine. The weight cyclic fine-tuning stage consists of three parts: supervised fine-tuning, reward modeling, and reinforcement learning. The model weights are fine-tuned through a cyclic structure to form a cyclic structure. The three are each assigned different task objectives. Supervised fine-tuning performs human instruction understanding and compares and associates human descriptions with graphics; reward modeling performs human value understanding, makes logical associations based on the feedback of manual ratings, and makes logical judgments on the quality of generated images; reinforcement learning serves as a cyclic connection structure for iterative optimization.

[0047] That is, the specific steps of model training in step S2 are:

[0048] Based on the sample Token obtained in step S1, the data set is screened to obtain a pre-training data set, the pre-training data set is pre-trained, and the basic model of mine virtual generation is iteratively output. The weights are fine-tuned through a loop structure by supervised fine-tuning, reward modeling and reinforcement learning. The mine data set is manually labeled to form a high-quality mine data set. The supervised fine-tuning receives its information, learns the human labeling instructions, and outputs weights to the reward modeling. The reward modeling learns the human value of data excellence through feedback, and further outputs the weights to the reinforcement learning. The reinforcement learning evaluates the quality of the basic model, and feedback correction and cyclic tuning are performed until the final model of mine virtual generation is output.

[0049] The supervised fine-tuning sets a generation function on the framework of the basic model. The mine dataset obtained in step S1 is manually labeled to obtain a high-quality mine dataset, which is divided into two types of data: text and image.

[0050] The high-quality mining data set is obtained by generating virtual text and corresponding virtual images by the data generator to form a virtual data set (F), which is then mixed with the real data set (R) for simulation discrimination. The relationship between the two is:

[0051]

[0052] Among them, i is the same type of text, R i With F i Represent the image matching probability distribution of real data and virtual data on text i respectively; if the L(R,F) value gradually decreases to convergence and is lower than the preset threshold, which is preferably 0.3, the weight is output; otherwise, it means distortion occurs, that is, the gap between virtual data and real data is very large, then the model weight is returned for fine-tuning.

[0053] like Figure 3 As shown, the reward modeling sets a logical association function on the framework of the basic model, receives the model weights to form a logical association network H, which is initially self-generated by the model and modified by the manual scoring results. It performs logical judgment on the generated text and image, and performs logical association evaluation on the quality of image generation. If it is a weak association, the logical associator is returned to fine-tune the weight; otherwise, the weight is output.

[0054] The reinforcement learning is a cyclic connection structure that iteratively performs supervised fine-tuning and reward modeling, optimizes model weights, and scores the comprehensive quality of generated text and images according to L(R, F). Feedback is given when the low-quality weight is greater than a preset threshold, which is preferably 0.15, and outputs a high-quality weight. The adjustment strategy is as follows:

[0055] D=Q′(s,a,s′)-Q(s,a,s′)=H(s′)-H(s)

[0056] Among them, s is the current weight state, a is the weight adjustment change, s′ is the next weight state, D represents the offset value of the model from the ideal state, Q represents the weight adjustment strategy, H represents the potential energy function of the logical association network, and represents the distance between the current state and the target state. The model iteration goal is to make D approach 0, and finally obtain the final model generated by the mine unmanned driving virtual scene.

[0057] S3: Data generation: Input the unmanned driving algorithm to be trained and the required text into the final model obtained in step S2. The final model automatically generates a dataset with labeled mine unmanned driving; the dataset generation results are scored and fed back to the weight cyclic fine-tuning stage in step S2 to provide it with a reference for equal division and cyclic tuning to obtain the final model of mine virtual generation.

[0058] The data generation process simply requires inputting the required text into the final model of the mine virtualization to generate an annotated image dataset. The dataset is then scored by humans or machines and fed back into the logical association network.

[0059] S4: In actual use, combining the final model generated by the virtual mine in step S3, a labeled and scored dataset of autonomous mine driving is obtained. This dataset includes differentiated images simulating changes in the mine environment, images of different mine locations, and simulated images of various obstacles during operation. These images address various standard issues faced by autonomous mine driving. The dataset addresses three key issues in autonomous mine driving: harsh environmental interference, equipment spatial positioning, and task path planning. The images in the dataset include simulated environmental changes, images of different mine locations, and simulated operational obstacles.

[0060] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for generating a virtual scene for unmanned driving in a mine based on a large model, characterized in that: The following steps are involved: S1: Data collection: collect network information and mine data. Network information comes from crawling network data in the communication network. Mine data is obtained through vehicle-mounted vision, surveillance cameras, and manual collection to form a mine data set. The network data and the mine data set are integrated to form a pre-training data set and obtain sample tokens. S2: Model training: Based on the sample tokens obtained in step S1, the model is pre-trained in sequence to obtain the basic model for virtual mine generation. The basic model enters the weight cyclic fine-tuning stage, and feedback correction and cyclic tuning are performed until the final model for generating the mine unmanned driving virtual scene is output; S3: Data generation: Input the unmanned driving algorithm to be trained and the required text into the final model obtained in step S2. The final model automatically generates a dataset with labeled mine unmanned driving. The dataset generation results are scored and fed back to the weight cyclic fine-tuning stage in step S2 to provide a scoring reference. After cyclic fine-tuning, the final model of mine virtual generation is obtained. S4: In actual use, combined with the final model of the mine virtualization generated in step S3, a dataset of labeled mine unmanned driving and combined with scores is obtained. The content of this dataset covers differentiated images simulating changes in the mine environment, images of different points in the mine, and simulated images of various obstacles during operation, in order to deal with various standard problems faced by mine unmanned driving.

2. The method for generating a large-scale model-based unmanned driving virtual scene in a mine according to claim 1, characterized in that: The network data in step S1 is multimodal information related to unmanned driving in mines; The method for obtaining sample tokens in step S1 is to classify and organize the fused network information and the mine dataset, unitize the samples, differentiate them into the smallest units that can be processed by the model, and obtain sample tokens of the corresponding class.

3. The method for generating a large-scale model-based unmanned driving virtual scene in a mine according to claim 1, characterized in that: It also includes data set screening before pre-training. Sample tokens include audio tokens and image tokens. Data set screening includes the following steps: S211: Convert audio tokens into text tokens for comparison and screening to remove duplicate and invalid texts; S212: Perform scene recognition on the image token based on the semantic label, excluding non-mining scenes; S213: The two types of tokens selected in steps S211 and S212 are respectively imported into the text or image encoder, and hybrid decoding is performed by the hybrid decoder to obtain a pre-training dataset.

4. The method for generating a large-scale model-based unmanned driving virtual scene in a mine according to claim 3, characterized in that: In step S2, the pre-training outputs the basic model for virtual generation of the mine. The weight loop fine-tuning stage consists of three parts: supervised fine-tuning, reward modeling, and reinforcement learning, and the model weights are fine-tuned through a loop structure.

5. The method for generating a large-scale model-based unmanned driving virtual scene in a mine according to claim 4, characterized in that: The specific steps of model training in step S2 are: Based on the sample Token obtained in step S1, the data set is screened to obtain a pre-training data set, the pre-training data set is pre-trained, and the basic model of mine virtual generation is iteratively output. The weights are fine-tuned through a loop structure by supervised fine-tuning, reward modeling and reinforcement learning. The mine data set is manually labeled to form a high-quality mine data set. The supervised fine-tuning receives its information, learns the human labeling instructions, and outputs weights to the reward modeling. The reward modeling learns the human value of data excellence through feedback, and further outputs the weights to the reinforcement learning. The reinforcement learning evaluates the quality of the basic model, and feedback correction and cyclic tuning are performed until the final model of mine virtual generation is output.

6. The method for generating a large-scale model-based unmanned driving virtual scene in a mine according to claim 5, characterized in that: The supervised fine-tuning sets a generation function on the framework of the basic model to obtain a high-quality mining data set. The data generator generates virtual text and corresponding virtual images to form a virtual data set F, which is then mixed with the real data set R for realistic discrimination. The relationship between the two is: Among them, i is the same type of text, R i With F i Represent the image matching probability distribution of real data and virtual data on text i respectively; If the L(R,F) value gradually decreases to convergence and is lower than the preset threshold, the weight is output; otherwise, it means distortion occurs, that is, the gap between the virtual data and the real data is very large, and the model weight is returned for fine-tuning.

7. The method for generating a large-scale model-based unmanned driving virtual scene in a mine according to claim 6, characterized in that: The reward modeling sets up a logical association function on the framework of the basic model, receives the model weights to form a logical association network H, which is initially generated by the model itself and modified by the manual scoring results. It performs logical judgment on the generated text and image, and performs logical association evaluation on the quality of image generation. If there is a weak association, it returns to the logical associator for weight fine-tuning. Otherwise, the output weight is given.

8. The method for generating a large-scale model-based unmanned driving virtual scene in a mine according to claim 6, characterized in that: The reinforcement learning is a cyclic connection structure that iteratively performs supervised fine-tuning and reward modeling, optimizes model weights, scores the comprehensive quality of generated text and images based on L(R,F), and outputs high-quality weights when the low-quality weights are greater than a preset threshold. The adjustment strategy is as follows: D=Q′(s,a,s′)-Q(s,a,s′)=H(s′)-H(s) Among them, s is the current weight state, a is the weight adjustment change, s′ is the next weight state, D represents the offset value of the model from the ideal state, Q represents the weight adjustment strategy, H represents the potential energy function of the logical association network, and represents the distance between the current state and the target state. The model iteration goal is to make D approach 0, and finally obtain the final model of mine virtual generation.

Images