Automatic path planning method for parts in fixed virtual environment based on large language model
By building an automatic path planning system including large language model, physics engine and three-dimensional environment module, combined with LoRA fine-tuning and quantization technology, the technical problem of achieving high-precision and low-latency automatic path planning in a three-dimensional environment is solved, and efficient and real-time automatic pathfinding function is realized, which is suitable for intelligent control in multiple fields.
Patent Information
- Application Number
- CN202411737493.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-29
- Publication Date
- 2025-03-11
AI Technical Summary
Implementing high-precision, low-latency automatic path planning based on large language models in a three-dimensional environment still faces technical difficulties in how to effectively compress and deploy large language models, transform the model's language understanding capabilities into effective mobile control instructions, and meet real-time requirements while ensuring pathfinding accuracy.
Build an automatic path planning system for parts including three-dimensional environment modules, control logic modules, large language modules and physical engine modules. Use large language modules to process current position information, generate speed vectors, and update part position information through physical engine modules to realize automatic path planning. LoRA fine-tuning technology and Q4_K_M quantization technology are used to fine-tune and compress large language models to ensure real-time inference requirements.
实现了高效的自动寻路功能,具有良好的实时性能和扩展性,适用于游戏开发、机器人导航等领域,为复杂环境中的智能控制提供了新的解决方案。
Smart Images
Figure CN119596741A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the fields of artificial intelligence, computer graphics and automatic control, and in particular to a method for automatic part path planning in a fixed virtual environment based on a large language model. Background Art
[0002] Automatic path planning of parts in a fixed virtual environment in a three-dimensional environment is an important research direction in the fields of computer graphics, artificial intelligence and automatic control. Traditional path planning methods mainly rely on rule-based algorithms, such as the Dijkstra algorithm. These methods perform well in known static environments, but often have difficulty adapting to dynamic changes or achieving smooth mobile control when real-time decision-making or complex paths are required.
[0003] The emergence of Large Language Models (LLMs) has brought new possibilities for solving complex tasks. LLMs, represented by GPT and LLAMA, have shown amazing capabilities and flexibility in natural language processing tasks. However, applying these models to non-language tasks, especially in the field of automatic pathfinding on fixed paths, is still an emerging research direction. The application potential of large language models in such tasks has not yet been fully explored and utilized.
[0004] Currently, there are still many technical challenges in achieving high-precision, low-latency automatic path planning based on large language models on fixed paths in three-dimensional environments. The main challenges include how to effectively compress and deploy large language models to meet real-time reasoning requirements, how to convert the model's language understanding capabilities into effective mobile control instructions, and how to meet real-time requirements while ensuring pathfinding accuracy. Summary of the invention
[0005] To solve the above problems, the present invention provides a method for automatic path planning of parts in a fixed virtual environment based on a large language model, comprising the following steps:
[0006] S1. Build a parts automatic path planning system, which includes a three-dimensional environment module, a control logic module, a large language module and a physical engine module;
[0007] S2. Using a three-dimensional environment module to load a three-dimensional virtual environment, the three-dimensional virtual environment includes parts;
[0008] S3. Obtain the current position information of the part in the three-dimensional virtual environment through the control logic module and send it to the large language module;
[0009] S4. The large language module uses the fine-tuned converted large language model to process the current position information and returns the velocity vector to the control logic module;
[0010] S5. The control logic module sends the velocity vector to the physical engine module to calculate the movement information of the part, and sends the movement information to the three-dimensional environment module to update the position information of the part;
[0011] S6. If the part reaches the end of the path or deviates from the path, the control logic loop ends, otherwise steps S3-S6 are repeated.
[0012] Furthermore, the process of obtaining the fine-tuned converted large language model by the large language module includes:
[0013] S11. Collecting raw data: constructing a target three-dimensional virtual environment, manipulating the parts to move in the target three-dimensional virtual environment, collecting a set of position data at fixed time intervals, and obtaining multiple sets of position data as raw data; a set of position data includes the three-dimensional coordinates of the current position of the part and the instantaneous speed in three directions, and the instantaneous speed in three directions includes the instantaneous speed of the part in the x, y, and z directions at the current position;
[0014] S12. Preprocessing the original data to obtain training data;
[0015] S13. Use the LLAMA model as the pre-trained language model;
[0016] S14. Based on the training data, the pre-trained language model is fine-tuned using the LoRA fine-tuning technology to obtain the fine-tuned LoRA parameters; the benchmark large language model and the fine-tuned LoRA parameters are combined to obtain the fine-tuned large language model;
[0017] S15. Convert the fine-tuned large language model into the GGUF format, and use the Q4_K_M quantization technology to compress it to obtain the fine-tuned converted large language model.
[0018] Furthermore, step S12 pre-processes the original data to obtain training data that conforms to the LLAMA model input format, including:
[0019] Perform noise filtering and normalization on the original data to obtain preprocessed data;
[0020] Convert the preprocessed data formats into the format that meets the LLAMA model input;
[0021] The preprocessed data after format conversion is divided into training data and verification data in a ratio of 8:2.
[0022] Furthermore, the specific processing of the control logic module includes:
[0023] S21. Set the frequency of requesting large speech model inference results;
[0024] S22. Obtaining the current position information of the part from the three-dimensional virtual environment;
[0025] S23. Construct an HTTP POST request according to the current location information, send an HTTP POST request to the large language model of the large language module through the API interface, and set the request timeout;
[0026] S24. If a JSON response is received from the large language model within the request timeout period, the velocity vector is extracted from the JSON response and input into the physics engine module; otherwise, the last velocity vector is used to input into the physics engine module;
[0027] S25. Obtain movement information from the physical engine module to update the position information of the part; if the part reaches the end of the path or deviates from the path, the control logic loop ends, otherwise the next control logic loop starts and returns to step S22.
[0028] Furthermore, the frequency of requesting the large language model inference result is no less than 4 times per second.
[0029] Furthermore, the calculation process of the physics engine module is expressed as
[0030] P_new=P_current+V×△t+0.5×a×(△t) 2
[0031] Among them, P_new represents movement information, P_current represents the current position information of the part, V represents velocity vector, a represents acceleration, and △t represents time step.
[0032] Furthermore, the automatic part path planning system also includes:
[0033] Optimization module for caching part position information and velocity vectors.
[0034] Furthermore, the dynamic adjustment process of requesting the frequency of large language model inference results is expressed as
[0035]
[0036] Among them, f new represents the updated frequency of requesting large language model inference results, f current represents the frequency of requesting large language model inference results before updating, t target represents the target delay, t actual represents the actual delay, and α represents the adjustment coefficient.
[0037] Beneficial effects of the present invention:
[0038] The present invention combines a large language model, a physical engine and a three-dimensional virtual environment to achieve an efficient automatic path-finding function, has good real-time performance and scalability, can be applied to multiple fields such as game development and robot navigation, and provides a new solution for intelligent control in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 The automatic path planning system for parts of the present invention is implemented as a flow chart;
[0040] Figure 2 This is a schematic diagram of the connection modules of the automatic path planning system for parts of the present invention. DETAILED DESCRIPTION
[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0042] The present invention provides a method for automatic path planning of parts in a fixed virtual environment based on a large language model. Figure 1 , Figure 2 As shown, including:
[0043] S1. Build a parts automatic path planning system, which includes a three-dimensional environment module, a control logic module, a large language module and a physical engine module;
[0044] S2. Using a three-dimensional environment module to load a three-dimensional virtual environment, the three-dimensional virtual environment includes parts;
[0045] S3. Obtain the current position information of the part in the three-dimensional virtual environment through the control logic module and send it to the large language module;
[0046] S4. The large language module uses the fine-tuned converted large language model to process the current position information and returns the velocity vector to the control logic module;
[0047] S5. The control logic module sends the velocity vector to the physical engine module to calculate the movement information of the part, and sends the movement information to the three-dimensional environment module to update the position information of the part;
[0048] S6. If the part reaches the end of the path or deviates from the path, the control logic loop ends, otherwise steps S3-S6 are repeated.
[0049] Specifically, in the automatic part path planning system:
[0050] The 3D environment module is used to load the 3D virtual environment based on the WebGL-based graphics rendering library;
[0051] The control logic module is used to obtain the current position information of the part in the 3D virtual environment and dynamically adjust the frequency of requesting the inference results of the large language model;
[0052] The large language module is used to obtain the large language model after fine-tuning and conversion. The large language model outputs a velocity vector according to the current position information.
[0053] The physics engine module is used to calculate the movement information of parts based on the velocity vector.
[0054] The system workflow is as follows Figure 1 As shown in the figure, the 3D environment module is used to initialize the 3D virtual environment and parts, the control logic module is used to obtain the current position information of the parts in the 3D virtual environment, the large language module is used to output the velocity vector according to the current position information, and the physical engine module is used to calculate the movement information of the parts according to the velocity vector. The above operations are repeated until the termination condition is met. The optimization module implements a local cache mechanism to store the model output of the commonly used path segments.
[0055] Specifically, the present invention uses a 3D environment module to construct the required 3D virtual environment, which includes a fixed virtual environment path, and initializes the position information of the parts in the fixed virtual environment path. At the same time, the 3D environment module can also update and render the environment status in real time.
[0056] Specifically, in the large language module, a large language model based on a multi-layer Transformer architecture is selected to take advantage of its powerful language understanding and generation capabilities. The LoRA (Low-Rank Adaptation) fine-tuning technology is used for efficient fine-tuning to adapt the model to the automatic path-finding task.
[0057] In order to enable the large language model to effectively learn the path-finding strategy in a three-dimensional environment, the present invention adopts the following steps to collect training data:
[0058] S11. Collecting raw data: constructing a target three-dimensional virtual environment, manipulating the part to move in the target three-dimensional virtual environment, collecting a set of position data at a fixed time interval (such as 50ms), and obtaining multiple sets of position data as raw data; a set of position data includes the three-dimensional coordinates (x, y, z) of the current position of the part and the instantaneous speed in three directions, and the instantaneous speed in three directions includes the instantaneous speed (vx, vy, vz) of the part in the three directions of x, y, and z at the current position;
[0059] S12. Preprocessing the original data to obtain training data, including:
[0060] Filter the raw data for noise and remove jitter caused by manual operation; calculate the position change between adjacent time points to generate more accurate speed data;
[0061] Normalize the original data after noise filtering so that all input features are in the same numerical range to obtain preprocessed data;
[0062] Convert the preprocessed data formats into the format that meets the LLAMA model input;
[0063] The preprocessed data after format conversion is divided into training data and validation data in a ratio of 8:2. Make sure that both data sets contain all parts of the complete path to avoid overfitting.
[0064] The present invention trains and fine-tunes the large language model including:
[0065] Initialize LoRA parameters: the rank of the low-rank matrix, the scaling factor alpha, and the target_modules modules to which LoRA needs to be applied.
[0066] Set training parameters: learning_rate, batch_size, num_epochs (dynamically adjusted according to the performance of the validation set), use AdamW optimizer. Apply gradient clipping to prevent gradient explosion. Implement early stopping mechanism to avoid overfitting.
[0067] Training loop: Input the training data into the large language model to obtain the prediction results; use the mean square error loss function to calculate the loss for back propagation and update the LoRA parameters; evaluate the model performance on the validation set.
[0068] Model saving: Save the fine-tuned LoRA parameters, merge the LoRA parameters with the benchmark large language model, and obtain the fine-tuned large language model.
[0069] Model conversion: Convert the fine-tuned large language model into GGUF (GPT-Generated Unified Format) format, apply Q4_K_M quantization technology to further compress the model size, significantly reduce the model inference time, implement a standardized local inference interface, and support asynchronous requests and error handling.
[0070] Model deployment: Prepare deployment configuration files, specify model parameters and quantization settings. Use the Big Prophet model deployment framework to load the fine-tuned converted big language model. Set up API interfaces to allow the system to interact with the fine-tuned converted big language model through HTTP requests.
[0071] The use of LoRA fine-tuning technology significantly reduces the number of parameters that need to be updated, speeds up training, and maintains the generalization ability of the model. The final model conversion and quantization steps ensure that the model can run efficiently on the local device, providing the basis for real-time control.
[0072] Specifically, the control logic module is the core part of the system, responsible for coordinating the interaction between the large language model, the physical engine module, and the three-dimensional environment module. The specific implementation steps are as follows:
[0073] S21. Set the frequency of requesting large language model inference results to ensure smooth visual effects;
[0074] S22. Obtaining the current position information of the part from the three-dimensional virtual environment;
[0075] S23. Construct an HTTP POST request according to the current location information, send an HTTP POST request to the large language model of the large language module through the API interface, and set the request timeout; the request timeout is generally 5s or 10s, which is the timeout value of the large model interface service.
[0076] Among them, construct the query string: "Current position: (x, y, z), please predict the velocity vector of the next step.", and construct the HTTP POST request according to the query string;
[0077] The large language model reasoning process can be expressed as:
[0078]
[0079] V=f(Inorm)·γ+β
[0080] Where Inorm is the normalized input, μ is the mean of the training data, σ is the standard deviation of the training data, γ is the speed scaling factor, β is the speed bias vector, I represents the model input, V represents the model output, and f represents the large language model function.
[0081] S24. If a JSON response is received from the large language model within the request timeout period, the velocity vector is extracted from the JSON response and input into the physics engine module; otherwise, the valid velocity vector obtained from the last request is used to input into the physics engine module;
[0082] S25. Obtain movement information from the physical engine module to update the position information of the part; if the part reaches the end of the path or deviates from the path, the control logic loop ends, otherwise the next control logic loop starts and returns to step S22.
[0083] Specifically, the calculation process of the physics engine module is expressed as
[0084] P_new=P_current+V×△t+0.5×a×(△t) 2
[0085] Among them, P_new represents the movement information (that is, the position information of the part after update), P_current represents the current position information of the part, V represents the velocity vector, △t represents the time step, and a represents the acceleration calculated by the engine that supports rigid body dynamics simulation and has real-time collision detection capabilities.
[0086] Specifically, during the operation of the system, the frequency of requesting the inference results of the large language model needs to be dynamically adjusted according to the delay of the large language model returning the response, which is expressed as
[0087]
[0088] Among them, f new represents the updated frequency of requesting large language model inference results, f current represents the frequency of requesting large language model inference results before updating, t target represents the target delay, t actual represents the actual delay, and α represents the adjustment coefficient.
[0089] In order to improve the real-time performance and response speed of the system, the automatic path planning system for parts also includes:
[0090] Optimization module for caching part position information and velocity vectors.
[0091] Specifically, an LRU (Least Recently Used) cache is implemented in the optimization module:
[0092] Key: Discretize the part position to a grid, such as (floor(x / 0.1),floor(y / 0.1),floor(z / 0.1))
[0093] Value: stores the corresponding velocity vector.
[0094] Before each request to the large language model, first check whether there is a corresponding key in the cache. If so, use the cached velocity vector directly. If not, request the large language model and update the cache.
[0095] In addition, when the system starts, the model responses for common path segments are preloaded. Before officially starting pathfinding, several "warm-up" queries are performed to ensure that the model and API are in the best condition.
[0096] Further prune and distill knowledge of large language models to reduce the inference time of large language models while maintaining performance. Explore different quantization strategies (such as INT4 quantization) to find the best balance between accuracy and speed.
[0097] The present invention significantly improves the response speed and stability of the system, making real-time pathfinding control based on a large language model possible with limited computing resources. These optimizations not only improve the user experience, but also provide valuable practical experience for similar real-time AI control systems.
[0098] In the present invention, unless otherwise clearly stipulated and limited, the terms such as "installation", "setting", "connection", "fixation" and "rotation" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, ordinary technicians in this field can understand the specific meanings of the above terms in the present invention according to the specific circumstances.
[0099] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A method for automatic path planning of parts in a fixed virtual environment based on a large language model, characterized in that: The following steps are involved: S1. Build a parts automatic path planning system, which includes a three-dimensional environment module, a control logic module, a large language module and a physical engine module; S2. Using a three-dimensional environment module to load a three-dimensional virtual environment, the three-dimensional virtual environment includes parts; S3. Obtain the current position information of the part in the three-dimensional virtual environment through the control logic module and send it to the large language module; S4. The large language module uses the fine-tuned converted large language model to process the current position information and returns the velocity vector to the control logic module; S5. The control logic module sends the velocity vector to the physical engine module to calculate the movement information of the part, and sends the movement information to the three-dimensional environment module to update the position information of the part; S6. If the part reaches the end of the path or deviates from the path, the control logic loop ends, otherwise steps S3-S6 are repeated.
2. The method for automatic part path planning in a fixed virtual environment based on a large language model according to claim 1 is characterized in that: The process of obtaining the fine-tuned converted large language model by the large language module includes: S11. Collecting raw data: constructing a target three-dimensional virtual environment, manipulating the parts to move in the target three-dimensional virtual environment, collecting a set of position data at fixed time intervals, and obtaining multiple sets of position data as raw data; a set of position data includes the three-dimensional coordinates of the current position of the part and the instantaneous speed in three directions, and the instantaneous speed in three directions includes the instantaneous speed of the part in the x, y, and z directions at the current position; S12. Preprocessing the original data to obtain training data; S13. Use the LLAMA model as the pre-trained language model; S14. Based on the training data, the pre-trained language model is fine-tuned using the LoRA fine-tuning technology to obtain the fine-tuned LoRA parameters; the benchmark large language model and the fine-tuned LoRA parameters are combined to obtain the fine-tuned large language model; S15. Convert the fine-tuned large language model into the GGUF format, and then use the Q4_K_M quantization technology to compress it to obtain the fine-tuned converted large language model.
3. The method for automatic path planning of parts in a fixed virtual environment based on a large language model according to claim 2 is characterized in that: Step S12 pre-processes the original data, including: Perform noise filtering and normalization on the original data to obtain preprocessed data; Convert the preprocessed data formats into the format that meets the LLAMA model input; The preprocessed data after format conversion is divided into training data and verification data in a ratio of 8:
2.
4. The method for automatic part path planning in a fixed virtual environment based on a large language model according to claim 1, characterized in that: The specific processing of the control logic module includes: S21. Setting the frequency of requesting large language model inference results; S22. Obtaining the current position information of the part from the three-dimensional virtual environment; S23. Construct an HTTP POST request according to the current location information, send an HTTP POST request to the large language model of the large language module through the API interface, and set the request timeout; S24. If a JSON response is received from the large language model within the request timeout period, the velocity vector is extracted from the JSON response and input into the physics engine module; otherwise, the last velocity vector is used to input into the physics engine module; S25. Obtain movement information from the physical engine module to update the position information of the part; if the part reaches the end of the path or deviates from the path, end the control logic loop, otherwise return to step S22.
5. The method for automatic path planning of parts in a fixed virtual environment based on a large language model according to claim 4 is characterized in that: The frequency of requesting large language model inference results is no less than 4 times per second.
6. The method for automatic part path planning in a fixed virtual environment based on a large language model according to claim 1, characterized in that: The calculation process of the physics engine module is expressed as P_new=P_current+V×△t+0.5×a×(△t) 2 Among them, P_new represents movement information, P_current represents the current position information of the part, V represents velocity vector, a represents acceleration, and △t represents time step.
7. The method for automatic part path planning in a fixed virtual environment based on a large language model according to claim 1, characterized in that: The automatic part path planning system also includes: Optimization module for caching part position information and velocity vectors.
8. The method for automatic part path planning in a fixed virtual environment based on a large language model according to claim 1, characterized in that: The dynamic adjustment process of requesting the frequency of large language model inference results is expressed as Among them, f new represents the updated frequency of requesting large language model inference results, f current represents the frequency of requesting large language model inference results before updating, t target represents the target delay, t actual represents the actual delay, and α represents the adjustment coefficient.