Unmanned vehicle and unmanned vehicle control method and device

Abstract

The invention discloses an unmanned vehicle and an unmanned vehicle control method and device, relates to the field of intelligent mechanical control, solves the problem of low task execution efficiency caused by unreasonable control of the unmanned vehicle, and comprises the following steps: decomposing a first task instruction of a current unmanned vehicle to be operated according to a first large language model to obtain a second task instruction; the first large language model is obtained based on the initial large language model in combination with training of an underlying control interface of the unmanned vehicle; the first task instruction is a natural language instruction generated in response to user input; the second task instruction comprises a task target and a task action; if the task target is not empty, matching the task target according to the semantic navigation map to obtain a navigation target, and generating a task path plan according to the coordinates of the unmanned vehicle and the navigation target in the semantic navigation map; the semantic navigation map is generated by pre-scanning a current task scene by the unmanned vehicle; and controlling the unmanned vehicle to arrive at the navigation target and complete the task action according to the task path planning.

Description

Technical Field

[0001] The present application relates to the field of intelligent mechanical control, and in particular to an unmanned vehicle, and an unmanned vehicle control method and device. Background Art

[0002] In the process of performing tasks, unmanned vehicles need to plan the task path. In the prior art, the task planning path of unmanned vehicles is mainly generated based on target navigation algorithms and models. Traditional target navigation algorithms and models rely on complex mathematical models and rules, have low adaptability in complex environments, and can only perceive local scenes. It takes multiple perceptions and modeling to complete the task at the target point, resulting in low efficiency in task execution. At the same time, users may issue task instructions to unmanned vehicles in the form of natural language, and unmanned vehicles must understand natural language instructions and convert them into task instructions. The prior art generally uses a large model to convert natural language instructions, but due to the opacity of the model and the poor compatibility of the model with the underlying control actions of the unmanned vehicle, the interpretability of the converted task instructions is not high, and it is difficult for the unmanned vehicle to recognize the task instructions, so it is difficult to reasonably control the unmanned vehicle, and then it is difficult to complete the task to be performed, thereby reducing the efficiency of task execution. Therefore, how to reasonably control the unmanned vehicle to improve the task execution efficiency of the unmanned vehicle is an urgent problem to be solved. Summary of the invention

[0003] The present application provides an unmanned vehicle, an unmanned vehicle control method and a device to solve the technical problem of low task execution efficiency caused by unreasonable control of unmanned vehicles in the prior art.

[0004] According to a first aspect of the embodiments of the present application, there is provided a method for controlling an unmanned vehicle, comprising:

[0005] According to the first large language model, the first task instruction of the unmanned vehicle to be operated is decomposed to obtain a second task instruction; wherein the first large language model is obtained based on the initial large language model and combined with the training of the bottom control interface of the unmanned vehicle; the first task instruction is a natural language instruction generated in response to user input; the second task instruction includes a task goal and a task action;

[0006] If the task target is not empty, the task target is matched according to the semantic navigation map to obtain the corresponding navigation target, and the task path planning is generated according to the coordinates of the unmanned vehicle and the navigation target in the semantic navigation map; wherein the semantic navigation map is generated by the unmanned vehicle by pre-scanning the current task scene;

[0007] According to the mission path planning, the unmanned vehicle is controlled to reach the navigation target and complete the mission action.

[0008] The present application first decomposes the first task instruction in natural language format according to the first large language model obtained by training based on the underlying control interface of the unmanned vehicle, and obtains the second task instruction including the task target and the task action. The large language model is trained based on the underlying control interface of the unmanned vehicle, which can improve the interpretability of the model for the task instructions of the unmanned vehicle, thereby reducing the deviation of the control of the unmanned vehicle and improving the rationality of the control of the unmanned vehicle; then, when the task target is not empty, the task target is matched according to the semantic navigation map generated by pre-scanning the current task scene to obtain the navigation target, and then the task path planning is generated according to the coordinates of the unmanned vehicle and the navigation target, which can improve the overall perception of the unmanned vehicle to the current task scene. Compared with the prior art, the unmanned vehicle only needs to perform task path planning to reach the navigation target, and does not need to perform multiple perception modeling as in the prior art. It can more reasonably control the unmanned vehicle to reach the navigation target and complete the task action, thereby improving the task execution efficiency of the unmanned vehicle.

[0009] In certain embodiments of the present application, the first large language model is obtained based on the initial large language model and combined with the training of the underlying control interface of the unmanned vehicle, and specifically includes:

[0010] Obtaining historical mission data and underlying control interface of the unmanned vehicle;

[0011] According to the historical task data and the underlying control interface, an initial large language model with preset initial parameters is iteratively trained, and the initial parameters are updated during iteration until the initial parameters meet the preset values, and the training is completed to obtain a first large language model.

[0012] This application first obtains the historical mission data and underlying control interface of the unmanned vehicle, and then iteratively trains the initial large language model based on the historical mission data and the underlying control interface to obtain the first large language model, which can improve the compatibility of the model with the underlying control interface of the unmanned vehicle, thereby improving the model's interpretability of the unmanned vehicle's mission instructions, reducing the deviation of the unmanned vehicle's control, and improving the rationality of the unmanned vehicle's control, thereby improving the unmanned vehicle's task execution efficiency.

[0013] In certain embodiments of the present application, matching the mission target according to the semantic navigation map to obtain the corresponding navigation target, and generating the mission path planning according to the coordinates of the unmanned vehicle and the navigation target in the semantic navigation map, specifically includes:

[0014] Encoding the semantic features of the task target to obtain a task feature vector;

[0015] Based on a preset similarity algorithm, matching each semantic feature vector of the semantic navigation map according to the task feature vector to obtain a navigation target;

[0016] Based on a preset path-finding algorithm, a task path planning is generated according to the coordinates of the unmanned vehicle and the navigation target in the semantic navigation map.

[0017] This application first performs semantic feature encoding on the task target to obtain a task feature vector, which can convert the task target into a task feature vector that is easier to match, thereby improving the matching efficiency. Then, based on a similarity algorithm, the task feature vector is matched with each semantic feature vector of the semantic navigation map to obtain a navigation target, thereby improving the accuracy of finding the navigation target. Then, based on a preset path-finding algorithm, the task path planning is generated, which can improve the rationality of the task path planning, and then improve the rationality of the control of the unmanned vehicle, thereby improving the task execution efficiency of the unmanned vehicle.

[0018] In certain embodiments of the present application, the semantic navigation map is generated by the unmanned vehicle by pre-scanning the current mission scene, specifically including:

[0019] Obtain observation data of the current mission scene based on binocular camera and inertial navigation acquisition; wherein the observation data includes camera pose, scene depth and scene image;

[0020] Corresponding to the camera pose and scene depth of the same frame, the unmanned vehicle is used as the origin of the semantic navigation map to construct global point cloud data;

[0021] Performing pixel-level visual segmentation encoding on the scene image to obtain pixel semantic features corresponding to each pixel point;

[0022] The global point cloud data is aligned with the pixel semantic features corresponding to each pixel point, and a ground projection is performed corresponding to the origin to obtain a semantic navigation map.

[0023] This application first obtains observation data of the current mission scene based on binocular cameras and inertial navigation, and then constructs global point cloud data according to the camera pose and scene depth. It can pre-construct the global scene and improve the unmanned vehicle's perception of the current mission scene. It then performs pixel-level visual segmentation encoding on the scene image to obtain the pixel semantic features corresponding to each pixel point, and aligns them with the global point cloud data. At the same time, it performs corresponding ground projection to obtain a semantic navigation map, which can integrate the pixel semantic features into the global point cloud data to enhance the unmanned vehicle's overall perception of the current mission scene. Compared with the existing technology, the unmanned vehicle does not need multiple perception modeling to reach the navigation target, so that the unmanned vehicle can be more reasonably controlled to reach the navigation target and complete the mission action, thereby improving the unmanned vehicle's mission execution efficiency.

[0024] In certain embodiments of the present application, it further includes:

[0025] If the task target is empty, combining the semantic navigation map, the task action and the current position of the unmanned vehicle to generate a task action plan;

[0026] The unmanned vehicle is controlled according to the mission action plan so that the unmanned vehicle completes the mission action.

[0027] When the current mission target is empty, this application first generates a mission action plan by combining the semantic navigation map, mission action and the current position of the unmanned vehicle, and controls the unmanned vehicle according to the mission action plan to enable the unmanned vehicle to complete the mission action. Different processing methods are set according to whether the mission target is empty or not, which can improve the rationality of the control of the unmanned vehicle and thereby improve the mission execution efficiency of the unmanned vehicle.

[0028] According to a second aspect of the implementation of the present application, there is provided an unmanned vehicle control device, including a task decomposition module, a target matching module and a task execution module;

[0029] The task decomposition module is used to decompose the first task instruction of the unmanned vehicle to be operated according to the first large language model to obtain the second task instruction; wherein the first large language model is obtained based on the initial large language model and combined with the training of the bottom control interface of the unmanned vehicle; the first task instruction is a natural language instruction generated in response to user input; the second task instruction includes a task goal and a task action;

[0030] The target matching module is used to match the task target according to the semantic navigation map if the task target is not empty, obtain the corresponding navigation target, and generate the task path planning according to the coordinates of the unmanned vehicle and the navigation target in the semantic navigation map; wherein the semantic navigation map is generated by the unmanned vehicle scanning the current task scene in advance;

[0031] The task execution module is used to control the unmanned vehicle to reach the navigation target and complete the task action according to the task path planning.

[0032] In certain embodiments of the present application, the first large language model is obtained based on the initial large language model and combined with the training of the underlying control interface of the unmanned vehicle, and specifically includes:

[0033] Obtaining historical mission data and underlying control interface of the unmanned vehicle;

[0034] According to the historical task data and the underlying control interface, an initial large language model with preset initial parameters is iteratively trained, and the initial parameters are updated during iteration until the initial parameters meet the preset values, and the training is completed to obtain a first large language model.

[0035] In certain embodiments of the present application, the target matching module includes a feature encoding unit, a similarity matching unit, and a path planning unit;

[0036] The feature encoding unit is used to perform semantic feature encoding on the task target to obtain a task feature vector;

[0037] The similarity matching unit is used to match each semantic feature vector of the semantic navigation map according to the task feature vector based on a preset similarity algorithm to obtain a navigation target;

[0038] The path planning unit is used to generate a task path plan based on a preset path-finding algorithm and according to the coordinates of the unmanned vehicle and the navigation target in the semantic navigation map.

[0039] In certain embodiments of the present application, the semantic navigation map is generated by the unmanned vehicle by pre-scanning the current mission scene, specifically including:

[0040] Obtain observation data of the current mission scene based on binocular camera and inertial navigation acquisition; wherein the observation data includes camera pose, scene depth and scene image;

[0041] Corresponding to the camera pose and scene depth of the same frame, the unmanned vehicle is used as the origin of the semantic navigation map to construct global point cloud data;

[0042] Performing pixel-level visual segmentation encoding on the scene image to obtain pixel semantic features corresponding to each pixel point;

[0043] The global point cloud data is aligned with the pixel semantic features corresponding to each pixel point, and a ground projection is performed corresponding to the origin to obtain a semantic navigation map.

[0044] In some embodiments of the present application, an action planning module is also included; the action planning module includes an action planning unit and an action execution unit;

[0045] The action planning unit is used to generate a task action plan by combining the semantic navigation map, the task action and the current position of the unmanned vehicle if the task target is empty;

[0046] The action execution unit is used to control the unmanned vehicle according to the task action plan so that the unmanned vehicle completes the task action.

[0047] The present application first decomposes the first task instruction in natural language format according to the first large language model obtained by training based on the underlying control interface of the unmanned vehicle, and obtains the second task instruction including the task target and the task action. The large language model is trained based on the underlying control interface of the unmanned vehicle, which can improve the interpretability of the model for the task instructions of the unmanned vehicle, thereby reducing the deviation of the control of the unmanned vehicle and improving the rationality of the control of the unmanned vehicle; then, when the task target is not empty, the task target is matched according to the semantic navigation map generated by pre-scanning the current task scene to obtain the navigation target, and then the task path planning is generated according to the coordinates of the unmanned vehicle and the navigation target, which can improve the overall perception of the unmanned vehicle to the current task scene. Compared with the prior art, the unmanned vehicle only needs to perform task path planning to reach the navigation target, and does not need to perform multiple perception modeling as in the prior art. It can more reasonably control the unmanned vehicle to reach the navigation target and complete the task action, thereby improving the task execution efficiency of the unmanned vehicle.

[0048] According to a third aspect of an implementation of the present application, an unmanned vehicle is provided, the unmanned vehicle comprising an unmanned vehicle control device as described in the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 : A flowchart of a method for controlling an unmanned vehicle shown in certain embodiments of the present application;

[0050] Figure 2 : A module structure diagram of an unmanned vehicle control device shown in certain embodiments of the present application. DETAILED DESCRIPTION

[0051] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions from beginning to end. The embodiments described below in conjunction with the accompanying drawings are exemplary and are only used to explain some embodiments of the present application, and should not be construed as limitations on the embodiments of the present application. Based on the embodiments shown in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0052] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, unless otherwise clearly and specifically defined, "multiple" and "several" mean two or more.

[0053] When an unmanned vehicle performs a task, it needs to plan the task path. However, the current target navigation algorithm relies on complex mathematical models and rules, has low adaptability to complex environments, and can only perceive local scenes. It takes multiple perception modeling to reach the target point to perform the task, resulting in low efficiency in task execution. At the same time, users may give tasks to unmanned vehicles through natural language instructions. The existing technology mainly uses large models to convert natural language instructions into task instructions. However, due to the opacity of the model and the poor compatibility between the model and the underlying control actions of the unmanned vehicle, the interpretability of the converted task instructions is not high, and the unmanned vehicle is difficult to identify the task instructions, which reduces the efficiency of task execution. Therefore, how to reasonably control the unmanned vehicle to improve the efficiency of its task execution is an urgent problem to be solved.

[0054] Based on the above technical background, please refer to Figure 1 The embodiment of the present application provides a method for controlling an unmanned vehicle, including steps S101 to S103, each of which is as follows:

[0055] Step S101: According to the first large language model, the first task instruction of the unmanned vehicle to be operated is decomposed to obtain the second task instruction; wherein the first large language model is based on the initial large language model and is obtained by combining the training of the underlying control interface of the unmanned vehicle; the first task instruction is a natural language instruction generated in response to user input; the second task instruction includes a task goal and a task action.

[0056] In certain embodiments of the present application, the first large language model is obtained based on the initial large language model and combined with the training of the underlying control interface of the unmanned vehicle, and specifically includes:

[0057] Obtaining historical mission data and underlying control interface of the unmanned vehicle;

[0058] According to the historical task data and the underlying control interface, an initial large language model with preset initial parameters is iteratively trained, and the initial parameters are updated during iteration until the initial parameters meet the preset values, and the training is completed to obtain a first large language model.

[0059] In certain optional embodiments of the present application, the preferred scheme of the initial large language model includes but is not limited to GPT, Wenxin Yiyan, iFlytek Spark, Tongyi Qianwen or an open source model fine-tuned by Lora.

[0060] Consider training the large language model through the underlying control interface, so that the large language model can learn the characteristics of the underlying control interface of the unmanned vehicle. Then, when the user issues the first task instruction through natural language instructions, the large language model can convert the natural language instruction into a second task instruction including a task goal and a task action. The second task instruction is closer to or equivalent to the machine control instruction, and the unmanned vehicle can recognize and understand it more easily, which improves the interpretability of the second task instruction. Therefore, the unmanned vehicle can be controlled more reasonably, thereby improving the task execution efficiency of the unmanned vehicle.

[0061] This application first obtains the historical mission data and underlying control interface of the unmanned vehicle, and then iteratively trains the initial large language model based on the historical mission data and the underlying control interface to obtain the first large language model, which can improve the compatibility of the model with the underlying control interface of the unmanned vehicle, thereby improving the model's interpretability of the unmanned vehicle's mission instructions, reducing the deviation of the unmanned vehicle's control, and improving the rationality of the unmanned vehicle's control, thereby improving the unmanned vehicle's task execution efficiency.

[0062] Step S102: If the task target is not empty, the task target is matched according to the semantic navigation map to obtain the corresponding navigation target, and the task path planning is generated according to the coordinates of the unmanned vehicle and the navigation target in the semantic navigation map; wherein the semantic navigation map is generated by the unmanned vehicle by pre-scanning the current task scene.

[0063] In certain embodiments of the present application, matching the mission target according to the semantic navigation map to obtain the corresponding navigation target, and generating the mission path planning according to the coordinates of the unmanned vehicle and the navigation target in the semantic navigation map, specifically includes:

[0064] Encoding the semantic features of the task target to obtain a task feature vector;

[0065] Based on a preset similarity algorithm, matching each semantic feature vector of the semantic navigation map according to the task feature vector to obtain a navigation target;

[0066] Based on a preset path-finding algorithm, a task path planning is generated according to the coordinates of the unmanned vehicle and the navigation target in the semantic navigation map.

[0067] In certain embodiments of the present application, the semantic feature encoding of the task target to obtain the task feature vector specifically includes:

[0068] Based on the text encoding module of the preset semantic segmentation model, the semantic features of the task target are encoded to obtain a task feature vector.

[0069] In certain optional embodiments of the present application, the preferred scheme of the preset semantic segmentation model includes but is not limited to the LSeg model, the CAT-Seg model or the ZegCL IP model.

[0070] In some optional implementations of the present application, preferred solutions of the preset similarity algorithm include but are not limited to a cosine similarity algorithm, a Pearson correlation coefficient algorithm, and a Jaccard similarity coefficient algorithm.

[0071] In certain embodiments of the present application, the task path planning is generated based on the preset pathfinding algorithm according to the coordinates of the unmanned vehicle and the navigation target in the semantic navigation map, specifically including:

[0072] Obtaining a first coordinate of the unmanned vehicle and a second coordinate of the navigation target according to the semantic navigation map;

[0073] Determining whether the navigation target is reachable according to the semantic navigation map, the preset first morphological parameter and the second coordinate of the navigation target, and the preset second morphological parameter and the first coordinate of the unmanned vehicle;

[0074] If the navigation target is reachable, generating a task path plan according to a preset path-finding algorithm, the first coordinate of the unmanned vehicle and the second coordinate of the navigation target;

[0075] If the navigation target is unreachable, the preset path-finding algorithm is used to find the reachable target point closest to the navigation target, and the reachable target point is used as the second navigation target. The task path planning is generated based on the first coordinates of the unmanned vehicle and the third coordinates of the second navigation target.

[0076] In certain optional embodiments of the present application, preferred solutions of the preset path-finding algorithm include but are not limited to Dijkstra algorithm, A* algorithm and simulated annealing algorithm.

[0077] This application first performs semantic feature encoding on the task target to obtain a task feature vector, which can convert the task target into a task feature vector that is easier to match, thereby improving the matching efficiency. Then, based on a similarity algorithm, the task feature vector is matched with each semantic feature vector of the semantic navigation map to obtain a navigation target, thereby improving the accuracy of finding the navigation target. Then, based on a preset path-finding algorithm, the task path planning is generated, which can improve the rationality of the task path planning, and then improve the rationality of the control of the unmanned vehicle, thereby improving the task execution efficiency of the unmanned vehicle.

[0078] In certain embodiments of the present application, the semantic navigation map is generated by the unmanned vehicle by pre-scanning the current mission scene, specifically including:

[0079] Obtain observation data of the current mission scene based on binocular camera and inertial navigation acquisition; wherein the observation data includes camera pose, scene depth and scene image;

[0080] Corresponding to the camera pose and scene depth of the same frame, the unmanned vehicle is used as the origin of the semantic navigation map to construct global point cloud data;

[0081] Performing pixel-level visual segmentation encoding on the scene image to obtain pixel semantic features corresponding to each pixel point;

[0082] The global point cloud data is aligned with the pixel semantic features corresponding to each pixel point, and a ground projection is performed corresponding to the origin to obtain a semantic navigation map.

[0083] In certain embodiments of the present application, the camera pose and scene depth corresponding to the same frame are constructed to obtain global point cloud data with the unmanned vehicle as the origin of the semantic navigation map, specifically:

[0084] Based on the SLAM technology, corresponding to the camera pose and scene depth of the same frame, the global point cloud data is constructed with the unmanned vehicle as the origin of the semantic navigation map.

[0085] In certain embodiments of the present application, the pixel-level visual segmentation encoding of the scene image to obtain the pixel semantic features corresponding to each pixel point specifically includes:

[0086] Based on the visual encoding module of the preset semantic segmentation model, the scene image is subjected to pixel-level visual segmentation encoding to obtain pixel semantic features corresponding to each pixel point.

[0087] In certain optional embodiments of the present application, the preferred scheme of the preset semantic segmentation model includes but is not limited to the LSeg model, the CAT-Seg model or the ZegCL IP model.

[0088] This application first obtains observation data of the current mission scene based on binocular cameras and inertial navigation, and then constructs global point cloud data according to the camera pose and scene depth. It can pre-construct the global scene and improve the unmanned vehicle's perception of the current mission scene. It then performs pixel-level visual segmentation encoding on the scene image to obtain the pixel semantic features corresponding to each pixel point, and aligns them with the global point cloud data. At the same time, it performs corresponding ground projection to obtain a semantic navigation map, which can integrate the pixel semantic features into the global point cloud data to enhance the unmanned vehicle's overall perception of the current mission scene. Compared with the existing technology, the unmanned vehicle does not need multiple perception modeling to reach the navigation target, so that the unmanned vehicle can be more reasonably controlled to reach the navigation target and complete the mission action, thereby improving the unmanned vehicle's mission execution efficiency.

[0089] Step S103: Control the unmanned vehicle to reach the navigation target and complete the task action according to the task path planning.

[0090] In certain embodiments of the present application, it further includes:

[0091] If the task target is empty, combining the semantic navigation map, the task action and the current position of the unmanned vehicle to generate a task action plan;

[0092] The unmanned vehicle is controlled according to the mission action plan so that the unmanned vehicle completes the mission action.

[0093] When the current mission target is empty, this application first generates a mission action plan by combining the semantic navigation map, mission action and the current position of the unmanned vehicle, and controls the unmanned vehicle according to the mission action plan to enable the unmanned vehicle to complete the mission action. Different processing methods are set according to whether the mission target is empty or not, which can improve the rationality of the control of the unmanned vehicle and thereby improve the mission execution efficiency of the unmanned vehicle.

[0094] In certain embodiments of the present application, another implementation of the unmanned vehicle control method as a whole relies on a large language model architecture, and certain preferred implementations of the large language model architecture are a combination of multiple large language models, specifically:

[0095] In step S101, the initial large language model is provided by a large language model architecture, and the preferred scheme of the initial large language model includes but is not limited to an open source model fine-tuned by GPT, Wenxin Yiyan, iFlytek Spark, Tongyi Qianwen or Lora, and fine-tuning training is performed in combination with the underlying control interface of the unmanned vehicle to obtain a first large language model; according to the first large language model, the first task instruction is decomposed by a prompt word to obtain a second task instruction;

[0096] In step S102, when the semantic feature encoding is performed on the task target to obtain the task feature vector, the text encoding module of the preset semantic segmentation model required is provided by the large language model architecture;

[0097] In step S102, when constructing a semantic navigation map, the scene image is subjected to pixel-level visual segmentation encoding to obtain pixel semantic features corresponding to each pixel point, and the required visual encoding module of the preset semantic segmentation model is provided by a large language model architecture; wherein, the preferred scheme of the preset semantic segmentation model includes but is not limited to an LSeg model, a CAT-Seg model or a ZegCL IP model.

[0098] The present application first decomposes the first task instruction in natural language format according to the first large language model obtained by training based on the underlying control interface of the unmanned vehicle, and obtains the second task instruction including the task target and the task action. The large language model is trained based on the underlying control interface of the unmanned vehicle, which can improve the interpretability of the model for the task instructions of the unmanned vehicle, thereby reducing the deviation of the control of the unmanned vehicle and improving the rationality of the control of the unmanned vehicle; then, when the task target is not empty, the task target is matched according to the semantic navigation map generated by pre-scanning the current task scene to obtain the navigation target, and then the task path planning is generated according to the coordinates of the unmanned vehicle and the navigation target, which can improve the overall perception of the unmanned vehicle to the current task scene. Compared with the prior art, the unmanned vehicle only needs to perform task path planning to reach the navigation target, and does not need to perform multiple perception modeling as in the prior art. It can more reasonably control the unmanned vehicle to reach the navigation target and complete the task action, thereby improving the task execution efficiency of the unmanned vehicle.

[0099] Corresponding to the previous method, see Figure 2 , the embodiment of the present application provides an unmanned vehicle control device, including a task decomposition module 210, a target matching module 220 and a task execution module 230;

[0100] The task decomposition module 210 is used to decompose the first task instruction of the unmanned vehicle to be operated according to the first large language model to obtain a second task instruction; wherein the first large language model is obtained based on the initial large language model and combined with the training of the bottom control interface of the unmanned vehicle; the first task instruction is a natural language instruction generated in response to user input; the second task instruction includes a task goal and a task action;

[0101] The target matching module 220 is used to match the task target according to the semantic navigation map to obtain the corresponding navigation target if the task target is not empty, and generate the task path planning according to the coordinates of the unmanned vehicle and the navigation target in the semantic navigation map; wherein the semantic navigation map is generated by the unmanned vehicle scanning the current task scene in advance;

[0102] The task execution module 230 is used to control the unmanned vehicle to reach the navigation target and complete the task action according to the task path planning.

[0103] In certain embodiments of the present application, the first large language model is obtained based on the initial large language model and combined with the training of the underlying control interface of the unmanned vehicle, and specifically includes:

[0104] Obtaining historical mission data and underlying control interface of the unmanned vehicle;

[0105] According to the historical task data and the underlying control interface, an initial large language model with preset initial parameters is iteratively trained, and the initial parameters are updated during iteration until the initial parameters meet the preset values, and the training is completed to obtain a first large language model.

[0106] In certain embodiments of the present application, the target matching module 220 includes a feature encoding unit 221, a similarity matching unit 222, and a path planning unit 223;

[0107] The feature encoding unit 221 is used to perform semantic feature encoding on the task target to obtain a task feature vector;

[0108] The similarity matching unit 222 is used to match each semantic feature vector of the semantic navigation map according to the task feature vector based on a preset similarity algorithm to obtain a navigation target;

[0109] The path planning unit 223 is used to generate a task path plan based on a preset path-finding algorithm and according to the coordinates of the unmanned vehicle and the navigation target in the semantic navigation map.

[0110] In certain embodiments of the present application, the semantic navigation map is generated by the unmanned vehicle by pre-scanning the current mission scene, specifically including:

[0111] Obtain observation data of the current mission scene based on binocular camera and inertial navigation acquisition; wherein the observation data includes camera pose, scene depth and scene image;

[0112] Corresponding to the camera pose and scene depth of the same frame, the unmanned vehicle is used as the origin of the semantic navigation map to construct global point cloud data;

[0113] Performing pixel-level visual segmentation encoding on the scene image to obtain pixel semantic features corresponding to each pixel point;

[0114] The global point cloud data is aligned with the pixel semantic features corresponding to each pixel point, and a ground projection is performed corresponding to the origin to obtain a semantic navigation map.

[0115] In some embodiments of the present application, an action planning module 240 is further included; the action planning module 240 includes an action planning unit 241 and an action execution unit 242;

[0116] The action planning unit 241 is used to generate a task action plan by combining the semantic navigation map, the task action and the current position of the unmanned vehicle if the task target is empty;

[0117] The action execution unit 242 is used to control the unmanned vehicle according to the task action plan so that the unmanned vehicle completes the task action.

[0118] The present application first decomposes the first task instruction in natural language format according to the first large language model obtained by training based on the underlying control interface of the unmanned vehicle, and obtains the second task instruction including the task target and the task action. The large language model is trained based on the underlying control interface of the unmanned vehicle, which can improve the interpretability of the model for the task instructions of the unmanned vehicle, thereby reducing the deviation of the control of the unmanned vehicle and improving the rationality of the control of the unmanned vehicle; then, when the task target is not empty, the task target is matched according to the semantic navigation map generated by pre-scanning the current task scene to obtain the navigation target, and then the task path planning is generated according to the coordinates of the unmanned vehicle and the navigation target, which can improve the overall perception of the unmanned vehicle to the current task scene. Compared with the prior art, the unmanned vehicle only needs to perform task path planning to reach the navigation target, and does not need to perform multiple perception modeling as in the prior art. It can more reasonably control the unmanned vehicle to reach the navigation target and complete the task action, thereby improving the task execution efficiency of the unmanned vehicle.

[0119] Corresponding to the aforementioned device, an embodiment of the present application provides an unmanned vehicle, which includes an unmanned vehicle control device as described in the present application.

[0120] Adaptively, the embodiments of the present application also provide a computer device and a computer-readable storage medium.

[0121] The computer device comprises: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor;

[0122] Among them, when the processor executes the computer program, an unmanned vehicle control method of the present application is implemented.

[0123] The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for loading by a processor to execute an unmanned vehicle control method of the present application.

[0124] The above are some implementation methods of the present application, and the purpose, technical scheme and beneficial effects of the present application are further described in detail. It should be clear that the above-mentioned partial implementation methods of the present application cannot be understood as limiting the present application. It is particularly pointed out that for those skilled in the art, any changes, modifications, equivalent substitutions and variations made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for controlling an unmanned vehicle, characterized in that: include: According to the first large language model, the first task instruction of the unmanned vehicle to be operated is decomposed to obtain a second task instruction; wherein the first large language model is obtained based on the initial large language model and combined with the training of the bottom control interface of the unmanned vehicle; the first task instruction is a natural language instruction generated in response to user input; the second task instruction includes a task goal and a task action; If the task target is not empty, the task target is matched according to the semantic navigation map to obtain the corresponding navigation target, and the task path planning is generated according to the coordinates of the unmanned vehicle and the navigation target in the semantic navigation map; wherein the semantic navigation map is generated by the unmanned vehicle by pre-scanning the current task scene; According to the mission path planning, the unmanned vehicle is controlled to reach the navigation target and complete the mission action.

2. The unmanned vehicle control method according to claim 1, characterized in that: The first large language model is obtained based on the initial large language model and combined with the training of the bottom control interface of the unmanned vehicle, and specifically includes: Obtaining historical mission data and underlying control interface of the unmanned vehicle; According to the historical task data and the underlying control interface, an initial large language model with preset initial parameters is iteratively trained, and the initial parameters are updated during iteration until the initial parameters meet the preset values, and the training is completed to obtain a first large language model.

3. The unmanned vehicle control method according to claim 1, characterized in that: The matching of the task target according to the semantic navigation map to obtain the corresponding navigation target, and generating the task path planning according to the coordinates of the unmanned vehicle and the navigation target in the semantic navigation map, specifically includes: Encoding the semantic features of the task target to obtain a task feature vector; Based on a preset similarity algorithm, matching each semantic feature vector of the semantic navigation map according to the task feature vector to obtain a navigation target; Based on a preset path-finding algorithm, a task path planning is generated according to the coordinates of the unmanned vehicle and the navigation target in the semantic navigation map.

4. The unmanned vehicle control method according to claim 1, characterized in that: The semantic navigation map is generated by the unmanned vehicle through pre-scanning of the current mission scene, specifically including: Obtain observation data of the current mission scene based on binocular camera and inertial navigation acquisition; wherein the observation data includes camera pose, scene depth and scene image; Corresponding to the camera pose and scene depth of the same frame, the unmanned vehicle is used as the origin of the semantic navigation map to construct global point cloud data; Performing pixel-level visual segmentation encoding on the scene image to obtain pixel semantic features corresponding to each pixel point; The global point cloud data is aligned with the pixel semantic features corresponding to each pixel point, and ground projection is performed corresponding to the origin to obtain a semantic navigation map.

5. The unmanned vehicle control method according to claim 1, characterized in that: Also includes: If the task target is empty, combining the semantic navigation map, the task action and the current position of the unmanned vehicle to generate a task action plan; The unmanned vehicle is controlled according to the mission action plan so that the unmanned vehicle completes the mission action.

6. An unmanned vehicle control device, characterized in that: It includes task decomposition module, target matching module and task execution module; The task decomposition module is used to decompose the first task instruction of the unmanned vehicle to be operated according to the first large language model to obtain the second task instruction; wherein the first large language model is obtained based on the initial large language model and combined with the training of the bottom control interface of the unmanned vehicle; the first task instruction is a natural language instruction generated in response to user input; the second task instruction includes a task goal and a task action; The target matching module is used to match the task target according to the semantic navigation map if the task target is not empty, obtain the corresponding navigation target, and generate the task path planning according to the coordinates of the unmanned vehicle and the navigation target in the semantic navigation map; wherein the semantic navigation map is generated by the unmanned vehicle scanning the current task scene in advance; The task execution module is used to control the unmanned vehicle to reach the navigation target and complete the task action according to the task path planning.

7. The unmanned vehicle control device according to claim 6, characterized in that: The first large language model is obtained based on the initial large language model and combined with the training of the bottom control interface of the unmanned vehicle, and specifically includes: Obtaining historical mission data and underlying control interface of the unmanned vehicle; According to the historical task data and the underlying control interface, an initial large language model with preset initial parameters is iteratively trained, and the initial parameters are updated during iteration until the initial parameters meet the preset values, and the training is completed to obtain a first large language model.

8. The unmanned vehicle control device according to claim 6, characterized in that: The target matching module includes a feature encoding unit, a similarity matching unit and a path planning unit; The feature encoding unit is used to encode the semantic features of the task target to obtain a task feature vector; The similarity matching unit is used to match each semantic feature vector of the semantic navigation map according to the task feature vector based on a preset similarity algorithm to obtain a navigation target; The path planning unit is used to generate a task path plan based on a preset path-finding algorithm and according to the coordinates of the unmanned vehicle and the navigation target in the semantic navigation map.

9. The unmanned vehicle control device according to claim 6, characterized in that: The semantic navigation map is generated by the unmanned vehicle through pre-scanning of the current mission scene, specifically including: Obtain observation data of the current mission scene based on binocular camera and inertial navigation acquisition; wherein the observation data includes camera pose, scene depth and scene image; Corresponding to the camera pose and scene depth of the same frame, the unmanned vehicle is used as the origin of the semantic navigation map to construct global point cloud data; Performing pixel-level visual segmentation encoding on the scene image to obtain pixel semantic features corresponding to each pixel point; The global point cloud data is aligned with the pixel semantic features corresponding to each pixel point, and ground projection is performed corresponding to the origin to obtain a semantic navigation map.

10. The unmanned vehicle control device according to claim 6, characterized in that: It also includes an action planning module; the action planning module includes an action planning unit and an action execution unit; The action planning unit is used to generate a task action plan by combining the semantic navigation map, the task action and the current position of the unmanned vehicle if the task target is empty; The action execution unit is used to control the unmanned vehicle according to the task action plan so that the unmanned vehicle completes the task action.

11. An unmanned vehicle, characterized in that: The unmanned vehicle includes an unmanned vehicle control device as described in any one of claims 6 to 10.

Images