Virtual scene unmanned platform motion control method based on large language model

Abstract

The invention relates to a virtual scene unmanned platform motion control method based on a large language model, and belongs to the technical field of natural language processing and virtual simulation. The implementation method comprises the following steps of: 1, identifying a natural language by using a virtual scene unmanned platform motion control system based on a large language model; specifically, a natural language of a user is input into a large language model; the natural language of the user is voice data; 2, the large model obtains scene information by using a visual identification system; furthermore, a YOLO-based visual identification system is fused with the large language model, so that the perception capability of the large language model is improved; 3, the big language model decomposes a natural language into subtasks and a structured language through training; 4, the packaged structured language is sent to the unmanned platform to execute a motion task of cyclic positioning; compared with the prior art, the method has the advantages that the motion control of the unmanned platform is realized in different scenes, and the method has good universality.

Description

Technical Field

[0001] The invention relates to a virtual scene unmanned platform motion control method based on a large language model, and belongs to the technical field of natural language processing and virtual simulation. Background Art

[0002] An unmanned platform refers to a system that can autonomously perform tasks in a specific environment through remote control, preset programs or intelligent decision-making systems without the need for direct human control. There are several existing methods for operating unmanned platforms: (1) The operator uses a remote control or control terminal to remotely control the unmanned platform, and controls the movement direction, height, speed and other parameters of the unmanned platform by controlling joysticks, buttons, etc., and can also receive data such as images sent back by the unmanned platform. This method is mainly used in consumer drones, small unmanned ground vehicles, etc. (2) Use preset programs for operation. Before performing a task, the operator writes information such as task requirements and action paths into a program and inputs it into the control system of the unmanned platform. The unmanned platform automatically runs according to the preset program when performing the task. This operation method is mainly used in unmanned transport vehicles on industrial production lines, agricultural plant protection drones, etc. (3) Intelligent autonomous operation. The unmanned platform is equipped with advanced sensors, processors and algorithms, and can perform autonomous perception, decision-making and control, and is often used in autonomous driving vehicles.

[0003] Existing unmanned platforms have the following shortcomings: (1) Remote control requires high requirements for operators. Operators need to undergo systematic learning before they can operate the unmanned platform, and they need to stay focused at all times to manually operate the joystick and buttons. This requires high reaction speed and operating skills. It is difficult for average operators to complete complex actions, and the control is not simple and convenient enough. (2) Using preset programs for operation cannot cope with emergencies and complex and changing environments. Once a situation that is not considered by the program occurs, the unmanned platform may not be able to respond correctly. When facing new tasks, the control program must be rewritten, wasting a lot of time and energy. (3) The cost of intelligent autonomous control is very high, requiring a large number of high-precision and expensive sensors, and requiring a large amount of resources and data to train the decision-making algorithm.

[0004] However, the application scenarios of the above-mentioned methods are relatively single and cannot adapt to multi-scenario conditions. Therefore, applying unmanned platforms to multi-scenario conditions and making them universal has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of the present invention is to propose a motion control method for an unmanned platform in a virtual scene based on a large language model in order to solve the technical problem of the universality of unmanned platforms in different scene environments.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] The present invention discloses a method for controlling motion of an unmanned platform in a virtual scene based on a large language model, comprising the following steps:

[0008] Step 1: using a virtual scene unmanned platform motion control system based on a large language model to recognize natural language; specifically, inputting the user's natural language into the large language model; the user's natural language is voice data;

[0009] Step 2: The large model uses the visual recognition system to obtain scene information. Furthermore, the YOLO-based visual recognition system is integrated with the large language model to improve the perception ability of the large language model.

[0010] Step 2.1: Train YOLO using a dataset of objects in the scene.

[0011] Step 2.2: Perform entity recognition on the objects in the scene, and calculate the corresponding three-dimensional coordinates of the object in the scene based on multi-directional viewing angles and imaging principles;

[0012] Step 2.3: Store the name and coordinate information of each object in the form of a tuple; obtain the tuple encapsulation information;

[0013] Step 2.4: Input the binary encapsulation information obtained in step 2.3 into the large model, so that the large model can obtain the scene information;

[0014] Step 3: The large language model decomposes natural language into subtasks and structured language through training;

[0015] Step 3.1: The large language model is trained to recognize and analyze the user's natural language through the prompt word project;

[0016] Step 3.2: The large language model converts the user's natural language into a sub-task set in the form of bigrams.

[0017] Perform secondary dimensionality reduction on the subtask set input into the large model in the form of binary groups;

[0018] Step 3.3: The large language model disassembles the subtask set, and successively calls the control function with the path planning function according to the subtask, and encapsulates the structured language; the control function with the path planning function is used to make autonomous decisions on the path planning of the UAV, so that the UAV can avoid obstacles;

[0019] Step 4: Send the encapsulated structured language to the unmanned platform to perform the patrol positioning motion task;

[0020] Step 4.1: When the large model detects that the user executes instructions abnormally, return to step 1;

[0021] Step 4.2: When the large model does not detect the corresponding object, drive the drone to the designated safe area and return to step 1;

[0022] The invention discloses a virtual scene unmanned platform motion control system based on a large language model, which is used to implement the above method. The invention discloses a virtual scene unmanned platform motion control system based on a large language model, which includes a natural language processing module, a scene understanding module, and a motion control module.

[0023] The scene understanding module uses a visual recognition system to capture and identify objects and then extract the location information of objects in the scene, collects the two-dimensional coordinates of objects in the scene image where the unmanned platform is located frame by frame, forms three-dimensional coordinates through the imaging principle, and stores the scene information in the form of two-tuples, which will serve as the input of the natural language processing module;

[0024] The natural language processing module is used to semantically analyze the user's voice data and decompose it into subtask sets in the form of two-tuples, and to perform structured processing on the natural language in combination with the scene information and the subtask sets; this will serve as the input of the motion control module;

[0025] The motion control module is used to drive the unmanned platform to make autonomous decisions on path planning and realize the position control of the unmanned platform in multiple scenarios;

[0026] Beneficial effects:

[0027] Compared with the prior art, it has the following beneficial effects:

[0028] (1) It enables operators to control the movement of unmanned platforms using natural language. Compared with complex manual operations using joysticks and buttons, natural language control is more intuitive and easier to understand, greatly reducing the operational threshold.

[0029] (2) Compared with using preset programs, using natural language control can change task instructions at any time according to actual conditions, not limited to pre-set programs, and can better cope with changes and unexpected situations in tasks. If the task requirements change, there is no need to reprogram, just use natural language to re-issue instructions, which enhances the degree of intelligence.

[0030] (3) Compared with intelligent autonomous control systems, our unmanned platform motion control system based on a large language model is very low cost and has good universality in a large number of different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a method sequence diagram of the present invention;

[0032] Figure 2 It is a schematic diagram of the process of the present invention;

[0033] Figure 3 It is a system function module diagram of the present invention;

[0034] Figure 4 It is an experimental diagram of the present invention. DETAILED DESCRIPTION

[0035] In order to better illustrate the purpose and advantages of the present invention, the invention is further described below in conjunction with the accompanying drawings and examples. It should be noted that the implementation of the present invention is not limited to the following embodiments, and any form of modification or change made to the present invention will fall within the protection scope of the present invention.

[0036] This embodiment describes an experiment of controlling a UAV in a simulated island environment using the method of the present invention.

[0037] Example

[0038] like Figure 1 As shown, a virtual scene unmanned platform motion control method based on a large language model of the present invention is specifically implemented in the following steps:

[0039] Step 1: using a virtual scene unmanned platform motion control system based on a large language model to recognize natural language; specifically, inputting the user's natural language into the large language model; the user's natural language is voice data;

[0040] In the embodiment, the user inputs the requirements in the visual interactive interface built based on the flask framework. For example, the user inputs the following content: "Take off, find the red tent, circle around, rise 10m, then go to the waterfall, and then return at a speed of 5m / s".

[0041] Step 2: The large model uses the visual recognition system to obtain scene information. Furthermore, the YOLO-based visual recognition system is integrated with the large language model to improve the perception ability of the large language model.

[0042] Step 2.1: Train YOLO using a dataset of objects in the scene.

[0043] Step 2.2: Perform entity recognition on the objects in the scene, and calculate the corresponding three-dimensional coordinates of the object in the scene based on multi-directional viewing angles and imaging principles;

[0044] Step 2.3: Store the name and coordinate information of each object in the form of a tuple; obtain the tuple encapsulation information;

[0045] Step 2.4: Input the binary encapsulation information obtained in step 2.3 into the large model, so that the large model can obtain the scene information;

[0046] In the embodiment, the COCO dataset is used to train the YOLO model. Then, the unmanned platform captures images from different angles, uses the trained YOLO to analyze the captured scene, identifies each object, and marks its position in the two-dimensional image, and then calculates its three-dimensional coordinates through the imaging principle. The above object name and three-dimensional coordinates are encapsulated in the form of ["object name", "three-dimensional coordinates"] and input into the large model.

[0047] Step 3: The large language model decomposes natural language into subtasks and structured language through training;

[0048] Step 3.1: The large language model is trained to recognize and analyze the user's natural language through the prompt word project;

[0049] Step 3.2: The large language model converts the user's natural language into a sub-task set in the form of bigrams.

[0050] Perform secondary dimensionality reduction on the subtask set input into the large model in the form of binary groups;

[0051] Step 3.3: The large language model disassembles the subtask set, and successively calls the control function with the path planning function according to the subtask, and encapsulates the structured language; the control function with the path planning function is used to make autonomous decisions on the path planning of the UAV, so that the UAV can avoid obstacles;

[0052] In the embodiment, the large language model is trained through prompt to parse the above user input into the following subtasks:

[0053] [Task 1, execute take-off command]

[0054] [Task 2, find the coordinates of the red tent]

[0055] [Task 3, fly to the red tent]

[0056] [Task 4: Fly around the red tent, maintain a proper height and distance above the tent, perform a circling flight, and return to the original position after completing a circle]

[0057] [Task 5, control the drone to rise vertically 10 meters]

[0058] [Task 6, find the coordinates of the waterfall]

[0059] [Mission 7, Fly to the Waterfall]

[0060] [Task 8, return at a speed of 5m / s]

[0061] The large model then calls a single function based on the simple task above, where the motion function includes the path planning algorithms of RRT and A*, encapsulated as a structured language.

[0062] Step 4: Send the encapsulated structured language to the unmanned platform to perform the patrol positioning motion task;

[0063] Step 4.1: When the large model detects that the user executes instructions abnormally, return to step 1;

[0064] Step 4.2: When the large model does not detect the corresponding object, drive the drone to the designated safe area and return to step 1;

[0065] In the embodiment, the large model understands the task of the solution. If the input is unclear, it will ask the user in text form, and the user needs to re-enter the instruction to clarify the task. If the corresponding object cannot be found, it will fly to another safe area of ​​the current scene and add the current scene information. If feasible, the above structured code is sent to the drone, and the drone executes the function in sequence.

[0066] The invention discloses a virtual scene unmanned platform motion control system based on a large language model, which is used to implement the above method. The invention discloses a virtual scene unmanned platform motion control system based on a large language model, which includes a natural language processing module, a scene understanding module, and a motion control module.

[0067] The scene understanding module uses a visual recognition system to capture and identify objects and then extract the location information of objects in the scene, collects the two-dimensional coordinates of objects in the scene image where the unmanned platform is located frame by frame, forms three-dimensional coordinates through the imaging principle, and stores the scene information in the form of two-tuples, which will serve as the input of the natural language processing module;

[0068] The natural language processing module is used to semantically analyze the user's voice data and decompose it into subtask sets in the form of two-tuples, and to perform structured processing on the natural language in combination with the scene information and the subtask sets; this will serve as the input of the motion control module;

[0069] The motion control module is used to drive the unmanned platform to make autonomous decisions on path planning and realize the position control of the unmanned platform in multiple scenarios;

[0070] In order to further illustrate the advantages of the present invention, the following is described in combination with experimental data. Figure 4 As shown, different application scenarios are processed by the visual recognition system and then input into the large model. The large model's perception capability is used to process the scene and then input into the model. The natural language input by the user is received to perform task decomposition and generate structured language. The structured language is input into the unmanned platform for motion position control to obtain the process results of the unmanned platform motion control task.

[0071] The results show that the present invention achieves universal applicability for unmanned platforms in different scenario environments.

Claims

1. A motion control method for an unmanned platform in a virtual scene based on a large language model, characterized in that: The following steps are included: Step 1: using a virtual scene unmanned platform motion control system based on a large language model to recognize natural language; specifically, inputting the user's natural language into the large language model; the user's natural language is voice data; Step 2: The large model uses the visual recognition system to obtain scene information. Furthermore, the YOLO-based visual recognition system is integrated with the large language model to improve the perception ability of the large language model. Step 2.1: Train YOLO using a dataset of objects in the scene. Step 2.2: Perform entity recognition on the objects in the scene, and calculate the corresponding three-dimensional coordinates of the object in the scene based on multi-directional viewing angles and imaging principles; Step 2.3: Store the name and coordinate information of each object in the form of a tuple; Get the binary encapsulation information; Step 2.4: Input the binary encapsulation information obtained in step 2.3 into the large model, so that the large model can obtain the scene information; Step 3: The large language model decomposes natural language into subtasks and structured language through training; Step 3.1: The large language model is trained to recognize and analyze the user's natural language through the prompt word project; Step 3.2: The large language model converts the user's natural language into a sub-task set in the form of bigrams. Perform secondary dimensionality reduction on the subtask set input into the large model in the form of binary groups; Step 3.3: The large language model disassembles the subtask set, calls the control function with path planning function according to the subtask one by one, and encapsulates the structured language; The control function with path planning function is used to make autonomous decisions on the path planning of the UAV, thereby enabling the UAV to avoid obstacles; Step 4: Send the encapsulated structured language to the unmanned platform to perform the patrol positioning motion task; Step 4.1: When the large model detects that the user executes instructions abnormally, return to step 1; Step 4.2: When the large model does not detect the corresponding object, drive the drone to the designated safe area and return to step 1.

2. A virtual scene unmanned platform motion control system based on a large language model that implements the method as claimed in claim 1, characterized in that: Includes natural language processing module, scene understanding module, and motion control module; The scene understanding module uses a visual recognition system to capture and identify objects and then extract the location information of objects in the scene, collects the two-dimensional coordinates of objects in the scene image where the unmanned platform is located frame by frame, forms three-dimensional coordinates through the imaging principle, and stores the scene information in the form of two-tuples, which will serve as the input of the natural language processing module; The natural language processing module is used to semantically analyze the user's voice data and decompose it into subtask sets in the form of two-tuples, and to perform structured processing on the natural language in combination with the scene information and the subtask sets; this will serve as the input of the motion control module; The motion control module is used to drive the unmanned platform to make autonomous decisions on path planning and realize the position control of the unmanned platform in multiple scenarios.

Images