Virtual scene unmanned platform motion control method based on large language model

Abstract

The present application relates to a kind of virtual scene unmanned platform motion control method based on large language model, belong to natural language processing and virtual simulation technical field.The present application implementation method is:1, the system of virtual scene unmanned platform motion control based on large language model is used to the natural language recognition;Specifically, the natural language of user is input into large language model;The natural language of user is voice data;2, large model obtains scene information using visual recognition system;Further, the visual recognition system based on YOLO is fused with large language model, for improving the perception ability of large language model;3, large language model decomposes natural language into subtask and structured language by training4, the structured language after encapsulation is sent to unmanned platform to execute the motion task of patrol positioning;Compared with prior art, the present application has good universality for unmanned platform motion control in different scenarios.

Description

Technical Field

[0001] This invention relates to a motion control method for unmanned platforms in virtual scenes based on a large language model, belonging to the field of natural language processing and virtual simulation technology. Background Technology

[0002] An unmanned platform refers to a system that can autonomously perform tasks in a specific environment without direct human control, through remote control, preset programs, or intelligent decision-making systems. Existing unmanned platform operation methods include: (1) Operators use remote controls or control terminals to remotely control the unmanned platform, controlling parameters such as movement direction, altitude, and speed via joysticks and buttons, while also receiving data such as images transmitted back from the unmanned platform. This method is mainly used in consumer drones and small unmanned ground vehicles. (2) Operation using preset programs. Before performing a task, the operator writes the task requirements and action path into a program and inputs it into the unmanned platform's control system. The unmanned platform automatically runs according to the preset program when performing the task. This operation method is mainly used in unmanned transport vehicles on industrial production lines and agricultural plant protection drones. (3) Intelligent autonomous operation. Unmanned platforms are equipped with advanced sensors, processors, and algorithms, enabling autonomous perception, decision-making, and control, and are commonly used in autonomous vehicles.

[0003] The existing unmanned platforms have the following shortcomings: (1) Remote control requires high operator skills. Operators need to undergo systematic learning before they can operate the unmanned platform. They also need to maintain focus at all times to manually operate the joystick and buttons. It requires high reaction speed and operating skills. Operators of average skill level find it difficult to complete complex actions, and the control is not simple and convenient enough. (2) Using preset programs for operation cannot cope with sudden situations and complex and ever-changing environments. Once a situation that the program has not considered occurs, the unmanned platform may not be able to respond correctly. The control program must be rewritten for each new task, wasting a lot of time and energy. (3) Intelligent autonomous operation is very expensive. It requires a large number of high-precision and expensive sensors, and a lot of resources and data are needed to train the decision-making algorithm.

[0004] However, the proposed methods are limited to a single application scenario and cannot be adapted to multiple scenarios. Therefore, applying unmanned platforms to multiple scenarios to make them universally applicable has become an urgent technical problem to be solved. Summary of the Invention

[0005] The purpose of this invention is to address the technical problem of the universality of unmanned platforms applicable to different scenarios and environments by proposing a motion control method for unmanned platforms in virtual scenarios based on a large language model.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] This invention discloses a motion control method for an unmanned platform in a virtual scene based on a large language model, comprising the following steps:

[0008] Step 1: Utilize a motion control system for a virtual scene unmanned platform based on a large language model to recognize natural language; specifically, input the user's natural language into the large language model; the user's natural language is speech data.

[0009] Step 2: The large model uses a visual recognition system to acquire scene information; furthermore, a YOLO-based visual recognition system is fused with the large language model to improve the large language model's perception capabilities.

[0010] Step 2.1: Train YOLO using a dataset of objects in the scene;

[0011] Step 2.2: Perform entity recognition on objects in the scene, and calculate the three-dimensional coordinates of the object in the scene based on multi-angle perspectives and imaging principles;

[0012] Step 2.3: Store the name and coordinate information of each object in the form of a tuple; obtain the tuple encapsulated information;

[0013] Step 2.4: Input the encapsulated information of the binary tuples obtained in Step 2.3 into the large model so that the large model can acquire scene information;

[0014] Step 3: The large language model decomposes natural language into sub-tasks and structured language through training;

[0015] Step 3.1: The large language model is trained through prompt word engineering to perform user natural language recognition and analysis;

[0016] Step 3.2: The large language model transforms the user's natural language into a set of subtasks in the form of tuples.

[0017] The subtask set input into the large model is subjected to secondary dimensionality reduction in the form of tuples.

[0018] Step 3.3: The large language model decomposes the subtask set and successively calls the control function with path planning function according to the subtask to encapsulate the structured language; the control function with path planning function is used to make autonomous decisions on the path planning of the UAV, thereby enabling the UAV to avoid obstacles.

[0019] Step 4: Send the encapsulated structured language to the unmanned platform to perform the roving positioning motion task;

[0020] Step 4.1: When the large model detects abnormal user execution of instructions, return to step 1;

[0021] Step 4.2: When the large model does not detect the corresponding object, guide the drone to the designated safe area and return to step 1;

[0022] This invention discloses a motion control system for an unmanned platform in a virtual scene based on a large language model, used to implement the aforementioned method. The motion control system for an unmanned platform in a virtual scene based on a large language model disclosed in this invention includes a natural language processing module, a scene understanding module, and a motion control module.

[0023] The scene understanding module uses a visual recognition system to capture and identify objects and then extract the position information of objects in the scene. It collects the two-dimensional coordinates of objects in the scene image where the unmanned platform is located in a frame-by-frame manner, forms three-dimensional coordinates through imaging principles, and stores the scene information in the form of tuples, which will be used as the input of the natural language processing module.

[0024] The natural language processing module is used to semantically parse the user's voice data, decompose it into a set of subtasks in the form of tuples, and perform structured processing of natural language by combining scene information with the set of subtasks; this will be used as the input of the motion control module.

[0025] The motion control module is used to drive the unmanned platform to make autonomous decisions on path planning, and realize the position control of the unmanned platform in multiple scenarios.

[0026] Beneficial effects:

[0027] Compared with existing technologies, it has the following beneficial effects:

[0028] (1) It enables operators to control the movement of unmanned platforms using natural language. Compared to using joysticks and buttons for complex manual operations, natural language control is more intuitive and easier to understand, greatly reducing the operating threshold.

[0029] (2) Compared with operating using preset programs, using natural language control allows for changes in task instructions at any time based on the actual situation, without being limited to pre-set programs, and can better cope with changes and unexpected situations in the task. If the task requirements change, there is no need to reprogram; instructions can simply be issued again using natural language, thus enhancing the level of intelligence.

[0030] (3) Compared with intelligent autonomous control systems, our unmanned platform motion control system based on large language models has a low cost and good universality in a large number of different scenarios. Attached Figure Description

[0031] Figure 1 This is a sequence diagram of the method of the present invention;

[0032] Figure 2 This is a flowchart illustrating the present invention;

[0033] Figure 3 This is a system functional block diagram of the present invention;

[0034] Figure 4 This is an experimental diagram of the present invention. Detailed Implementation

[0035] To better illustrate the purpose and advantages of this invention, the invention will be further described below with reference to the accompanying drawings and examples. It should be noted that the implementation of this invention is not limited to the following embodiments, and any modifications or alterations made to this invention will fall within the scope of protection of this invention.

[0036] This embodiment describes an experiment in which a drone was controlled in a simulated island environment using the method described in this invention.

[0037] Example

[0038] like Figure 1 As shown, the motion control method for an unmanned platform in a virtual scene based on a large language model according to the present invention has the following specific implementation steps:

[0039] Step 1: Utilize a motion control system for a virtual scene unmanned platform based on a large language model to recognize natural language; specifically, input the user's natural language into the large language model; the user's natural language is speech data.

[0040] In this example, the user inputs their requirements into a visual interactive interface built on the Flask framework. For example, the user inputs the following: "Take off, find the red tent, circle around it, ascend 10m, then head to the waterfall, and then return at a speed of 5m / s."

[0041] Step 2: The large model uses a visual recognition system to acquire scene information; furthermore, a YOLO-based visual recognition system is fused with the large language model to improve the large language model's perception capabilities.

[0042] Step 2.1: Train YOLO using a dataset of objects in the scene;

[0043] Step 2.2: Perform entity recognition on objects in the scene, and calculate the three-dimensional coordinates of the object in the scene based on multi-angle perspectives and imaging principles;

[0044] Step 2.3: Store the name and coordinate information of each object in the form of a tuple; obtain the tuple encapsulated information;

[0045] Step 2.4: Input the encapsulated information of the binary tuples obtained in Step 2.3 into the large model so that the large model can acquire scene information;

[0046] In this embodiment, a YOLO model is trained using the COCO dataset. Then, an unmanned platform captures images from different angles, analyzes the captured scene using the trained YOLO model, identifies each object, and marks its position in the 2D image. Finally, its 3D coordinates are calculated using imaging principles. The object names and 3D coordinates are then encapsulated in the form of ["object name", "3D coordinates"] and input into the larger model.

[0047] Step 3: The large language model decomposes natural language into sub-tasks and structured language through training;

[0048] Step 3.1: The large language model is trained through prompt word engineering to perform user natural language recognition and analysis;

[0049] Step 3.2: The large language model transforms the user's natural language into a set of subtasks in the form of tuples.

[0050] The subtask set input into the large model is subjected to secondary dimensionality reduction in the form of tuples.

[0051] Step 3.3: The large language model decomposes the subtask set and successively calls the control function with path planning function according to the subtask to encapsulate the structured language; the control function with path planning function is used to make autonomous decisions on the path planning of the UAV, thereby enabling the UAV to avoid obstacles.

[0052] In this embodiment, the large language model, after being trained using the prompt, parses the above user input into the following sub-tasks:

[0053] [Task 1: Execute takeoff command]

[0054] [Task 2: Find the coordinates of the red tent]

[0055] [Mission 3, Fly to the Red Tent]

[0056] [Task 4: Circle the red tent once, maintaining an appropriate altitude and distance above the tent, and return to the original position after completing one circle.]

[0057] [Task 5: Control the drone to ascend vertically by 10 meters]

[0058] [Task Six: Find the coordinates of the waterfall]

[0059] [Task Seven, Fly to the Waterfall]

[0060] [Task 8, return at a speed of 5 m / s]

[0061] The large model then calls a single function based on the simple task described above. The motion function contains path planning algorithms for RRT and A*, which are encapsulated in a structured language.

[0062] Step 4: Send the encapsulated structured language to the unmanned platform to perform the roving positioning motion task;

[0063] Step 4.1: When the large model detects abnormal user execution of instructions, return to step 1;

[0064] Step 4.2: When the large model does not detect the corresponding object, guide the drone to the designated safe area and return to step 1;

[0065] In this embodiment, the large model performs task understanding on the solution. If the input is unclear, it will request the user in text form, requiring the user to re-enter the command to clarify the task. If the corresponding item cannot be found, it flies to another safe area in the current scene and adds the current scene information. If feasible, the structured code described above is sent to the drone, which executes the functions sequentially.

[0066] This invention discloses a motion control system for an unmanned platform in a virtual scene based on a large language model, used to implement the aforementioned method. The motion control system for an unmanned platform in a virtual scene based on a large language model disclosed in this invention includes a natural language processing module, a scene understanding module, and a motion control module.

[0067] The scene understanding module uses a visual recognition system to capture and identify objects and then extract the position information of objects in the scene. It collects the two-dimensional coordinates of objects in the scene image where the unmanned platform is located in a frame-by-frame manner, forms three-dimensional coordinates through imaging principles, and stores the scene information in the form of tuples, which will be used as the input of the natural language processing module.

[0068] The natural language processing module is used to semantically parse the user's voice data, decompose it into a set of subtasks in the form of tuples, and perform structured processing of natural language by combining scene information with the set of subtasks; this will be used as the input of the motion control module.

[0069] The motion control module is used to drive the unmanned platform to make autonomous decisions on path planning, and realize the position control of the unmanned platform in multiple scenarios.

[0070] To further illustrate the advantages of this invention, the explanation is provided in conjunction with experimental data. For example... Figure 4 As shown, different application scenarios are processed by a visual recognition system and then input into a large model. The large model's perception capabilities are used to process the scenarios and input the model. The model receives natural language input from the user, decomposes the task, and generates structured language. The structured language is then input into the unmanned platform for motion and position control, and the process results of the unmanned platform's motion control task are obtained.

[0071] The results show that the present invention achieves universality for unmanned platforms applicable to different scenarios and environments.

Claims

1. A motion control method for an unmanned platform in a virtual scene based on a large language model, characterized in that: Includes the following steps, Step 1: Utilize a motion control system for a virtual scene unmanned platform based on a large language model to recognize natural language; specifically, input the user's natural language into the large language model; the user's natural language is speech data. Step 2: The large model uses a visual recognition system to acquire scene information; furthermore, a YOLO-based visual recognition system is fused with the large language model to improve the large language model's perception capabilities. Step 2.1: Train YOLO using a dataset of objects in the scene; Step 2.2: Perform entity recognition on objects in the scene, and calculate the three-dimensional coordinates of the object in the scene based on multi-angle perspectives and imaging principles; Step 2.3: Store the name and coordinate information of each object in the form of a tuple; Obtain the encapsulation information of the binary tuple; Step 2.4: Input the encapsulated information of the binary tuples obtained in Step 2.3 into the large model so that the large model can acquire scene information; Step 3: The large language model decomposes natural language into sub-tasks and structured language through training; Step 3.1: The large language model is trained through prompt word engineering to perform user natural language recognition and analysis; Step 3.2: The large language model transforms the user's natural language into a set of subtasks in the form of tuples. The subtask set input into the large model is subjected to secondary dimensionality reduction in the form of tuples. Step 3.3: The large language model decomposes the set of subtasks and successively calls the control function with path planning function according to the subtask to encapsulate the structured language; The control function with path planning capability is used to make autonomous decisions on the path planning of the UAV, thereby enabling the UAV to avoid obstacles. Step 4: Send the encapsulated structured language to the unmanned platform to perform the roving positioning motion task; Step 4.1: When the large model detects abnormal user execution of instructions, return to step 1; Step 4.2: When the large model does not detect the corresponding object, drive the drone to the designated safe area and return to step 1.

2. A motion control system for a virtual scene unmanned platform based on a large language model, as described in claim 1, characterized in that: It includes a natural language processing module, a scene understanding module, and a motion control module; The scene understanding module uses a visual recognition system to capture and identify objects and then extract the position information of objects in the scene. It collects the two-dimensional coordinates of objects in the scene image where the unmanned platform is located in a frame-by-frame manner, forms three-dimensional coordinates through imaging principles, and stores the scene information in the form of tuples, which will be used as the input of the natural language processing module. The natural language processing module is used to semantically parse the user's voice data, decompose it into a set of subtasks in the form of tuples, and perform structured processing of natural language by combining scene information with the set of subtasks; this will be used as the input of the motion control module. The motion control module is used to drive the unmanned platform to make autonomous decisions on path planning, and to realize the position control of the unmanned platform in multiple scenarios.

Images