An indoor autonomous planning method for land-air amphibious unmanned aerial vehicles based on a large language model

Abstract

The application provides an indoor autonomous planning method for an amphibious unmanned aerial vehicle based on a large language model, comprising: acquiring indoor environment perception data and IMU data collected by the amphibious unmanned aerial vehicle at the current time; the indoor environment perception data comprises RGB images and point cloud data; the RGB images are processed to obtain an obstacle detection result of the indoor environment; a large language model is used to process task instructions and indoor environment data to obtain task semantics; based on key point types, the RGB images and the point cloud data at the current time are processed to obtain a plurality of key points and their positions; the large language model is used to process the task semantics, the indoor environment perception data, the IMU data, the obstacle detection result and the plurality of key points and their positions to generate mode switching and path planning of the amphibious unmanned aerial vehicle at the next time. The application can improve the adaptability and autonomy of the amphibious unmanned aerial vehicle in the indoor environment, and is especially suitable for indoor disaster relief and the like.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of amphibious unmanned aerial vehicles, and in particular to an indoor autonomous planning method for an amphibious unmanned aerial vehicle based on a large language model. BACKGROUND

[0002] At present, amphibious unmanned aerial vehicles have great potential in indoor search and rescue missions. After disasters such as earthquakes and fires, the interior of buildings is often full of uncertainties, such as structural collapse and obstacle accumulation, and traditional rescue methods are difficult to quickly enter and locate survivors. Amphibious unmanned aerial vehicles, with their dual capabilities of land travel and air flight, can flexibly switch modes in complex environments and provide new solutions for rescue missions.

[0003] However, the complexity and dynamics of indoor environments pose serious challenges to the autonomous operation of amphibious unmanned aerial vehicles. First, indoor spaces are usually complex in layout and irregular in obstacle distribution, and traditional path planning methods are difficult to adapt to environmental changes in real time, causing the unmanned aerial vehicle to easily get into trouble during travel. Second, existing amphibious unmanned aerial vehicles have obvious shortcomings in intelligent decision-making and human-machine interaction, and are difficult to quickly adjust task strategies in the face of unexpected situations such as path blockage or discovery of new signs of life. In addition, the efficiency of command transmission between operators and unmanned aerial vehicles is low, making it difficult to achieve real-time adjustment of complex tasks, limiting the effectiveness of unmanned aerial vehicles in rescue missions. SUMMARY

[0004] Therefore, the application provides an indoor autonomous planning method for an amphibious unmanned aerial vehicle based on a large language model to solve the above technical problems.

[0005] In a first aspect, the embodiments of the application provide an indoor autonomous planning method for an amphibious unmanned aerial vehicle based on a large language model, comprising:

[0006] Obtain indoor environment perception data and IMU data collected by the amphibious unmanned aerial vehicle at the current time; the indoor environment perception data includes RGB images and point cloud data;

[0007] Process the RGB images at the current time to obtain obstacle detection results of the indoor environment;

[0008] Obtain a task instruction input by a user;

[0009] Process the task instruction and the indoor environment data using a large language model to obtain a task semantic, the task semantic including: mode priority, task target, key point type and constraint condition;

[0010] Based on the key point type, the RGB image and the point cloud data at the current time are processed to obtain a plurality of key points and their positions, each key point representing a semantic key point in the indoor environment.

[0011] The task semantics, indoor environment perception data, IMU data, obstacle detection results, and a plurality of key points and their positions are processed by using a large language model to generate mode switching and path planning of the amphibious unmanned aerial vehicle at the next time, wherein the path of the amphibious unmanned aerial vehicle passes through the key points as much as possible.

[0012] In one possible implementation, the RGB image at the current time is processed to obtain obstacle detection results of the indoor environment, including:

[0013] The RGB image at the current time is processed by using a YOLOv8 network to obtain obstacle detection results of the indoor environment, and the obstacle detection results include the category, size, and two-dimensional position of the obstacle.

[0014] In one possible implementation, based on the key point type, the RGB image and the point cloud data at the current time are processed to obtain a plurality of key points and their positions, including:

[0015] The RGB image at the current time is semantically segmented by using a SAM model to obtain a segmentation mask, and the segmentation mask includes the outline and type of each object.

[0016] The RGB image at the current time is high-dimensional feature extracted by using a DINOv2 model to obtain a high-dimensional feature map, wherein each feature vector of the high-dimensional feature map corresponds to a region in the image and contains semantic description information of the region.

[0017] The segmentation mask and the high-dimensional feature map are fused to obtain a feature vector set V marked as foreground by the segmentation mask.

[0018] Based on the key point type, the feature vector set V is clustered by using a k-means clustering algorithm to obtain key points in the indoor environment.

[0019] The key points are mapped to a three-dimensional world coordinate system by using the point cloud data to obtain the positions of the key points.

[0020] In one possible implementation, the task semantics, indoor environment perception data, IMU data, obstacle detection results, and a plurality of key points and their positions are processed by using a large language model to generate mode switching and path planning of the amphibious unmanned aerial vehicle at the next time, including:

[0021] determine the action mode of the amphibious unmanned aerial vehicle at the next moment based on the obstacle detection result at the current moment, the indoor environment perception data, and the task semantics; the action mode includes: a land mode and a flight mode;

[0022] determine the motion state of the amphibious unmanned aerial vehicle at the current moment according to the IMU data at the current moment;

[0023] determine whether to perform mode switching according to the motion state of the amphibious unmanned aerial vehicle at the current moment and the action mode of the unmanned aerial vehicle at the next moment;

[0024] generate a key point constraint according to the task semantics and the position of the key point;

[0025] perform path planning according to the action mode of the amphibious unmanned aerial vehicle at the next moment, the key point constraint, and the constraint condition, and the position of the amphibious unmanned aerial vehicle at the current moment and the position of the key point.

[0026] In one possible implementation, determining the action mode of the amphibious unmanned aerial vehicle at the next moment based on the obstacle detection result at the current moment, the indoor environment perception data, and the task semantics includes:

[0027] when the height of the obstacle is greater than a first threshold value and the width of the obstacle is greater than a second threshold value, determining that the first action mode of the amphibious unmanned aerial vehicle at the next moment is the flight mode, and the first action mode value is 1; otherwise, determining that the first action mode of the amphibious unmanned aerial vehicle at the next moment is the land mode, and the first action mode value is 0;

[0028] processing the point cloud data to obtain a slope value and a flatness value of the front region, when the slope value is greater than 15° and the flatness value is greater than 0.1 m, determining that the second action mode of the amphibious unmanned aerial vehicle at the next moment is the flight mode, and the second action mode value is 1; otherwise, determining that the second action mode of the amphibious unmanned aerial vehicle at the next moment is the land mode, and the second action mode value is 0;

[0029] determining that the third action mode of the amphibious unmanned aerial vehicle at the next moment is the flight mode according to the mode priority in the task semantics, and the third action mode value is 1; otherwise, determining that the third action mode of the amphibious unmanned aerial vehicle at the next moment is the land mode, and the third action mode value is 0;

[0030] calculating the action mode value of the amphibious unmanned aerial vehicle at the next moment:

[0031] Mode=w1·Mode1+w2·Mode2+w3·Mode3

[0032] wherein Mode1 is a first action mode value, w1 is a first weight, Mode2 is a second action mode value, w2 is a second weight, Mode3 is a third action mode value, and w3 is a third weight;

[0033] When Mode is not less than 0.5, it is determined that the final action mode of the amphibious unmanned vehicle at the next moment is the flight mode, otherwise, it is determined that the final action mode of the amphibious unmanned vehicle at the next moment is the land mode.

[0034] In a possible implementation, according to the task semantics and the positions of the key points, a key point constraint is generated, including:

[0035] According to the task semantics, a task key point related to a task target is selected from the key points;

[0036] The key point constraint includes a distance constraint and a direction constraint.

[0037] The distance constraint c1 is:

[0038] c1 = ||k1-k2|| ≤ dsafe

[0039] wherein dsafe is a minimum safe distance of the amphibious unmanned vehicle from an obstacle, k1 is a three-dimensional position of a task key point closest to the position of the amphibious unmanned vehicle at the current moment, and k2 is a three-dimensional position of a task key point at the next moment.

[0040] The direction constraint c2 is to ensure that a line connecting the task key point closest to the position of the amphibious unmanned vehicle at the current moment and the task key point at the next moment is as small as possible in the flight direction of the amphibious unmanned vehicle.

[0041] In a possible implementation, according to the action mode of the amphibious unmanned vehicle at the next moment, the key point constraint, and the constraint condition, a path is planned by using the position of the amphibious unmanned vehicle at the current moment and the positions of the key points, including:

[0042] When the action mode of the amphibious unmanned vehicle at the next moment is the land mode, the position of the amphibious unmanned vehicle at the next moment is determined by using a 2D A* algorithm according to the constraint condition, wherein the position of the amphibious unmanned vehicle at the next moment is as close as possible to the task key point satisfying the key point constraint.

[0043] When the action mode of the amphibious unmanned vehicle at the next moment is the flight mode, the position of the amphibious unmanned vehicle at the next moment is determined by using a 3D A* algorithm according to the constraint condition, wherein the position of the amphibious unmanned vehicle at the next moment is as close as possible to the task key point satisfying the key point constraint.

[0044] In a second aspect, an indoor autonomous planning device for an amphibious unmanned aerial vehicle based on a large language model is provided, and the device comprises:

[0045] A first obtaining unit is configured to obtain indoor environment perception data and IMU data collected by the amphibious unmanned aerial vehicle at a current time, wherein the indoor environment perception data comprises RGB images and point cloud data.

[0046] A first processing unit is configured to process the RGB images at the current time to obtain an obstacle detection result of the indoor environment.

[0047] A second obtaining unit is configured to obtain a task instruction input by a user.

[0048] A second processing unit is configured to process the task instruction and the indoor environment data by using a large language model to obtain a task semantic, wherein the task semantic comprises a mode priority, a task target, a key point type, and a constraint condition.

[0049] A third processing unit is configured to process the RGB images and the point cloud data at the current time based on the key point type to obtain a plurality of key points and their positions, wherein each key point represents a semantic key point in the indoor environment.

[0050] An autonomous planning unit is configured to process the task semantic, the indoor environment perception data, the IMU data, the obstacle detection result, and the plurality of key points and their positions by using a large language model to generate a mode switching and a path planning of the amphibious unmanned aerial vehicle at a next time, wherein the path of the amphibious unmanned aerial vehicle passes through the key points as much as possible.

[0051] In a third aspect, an electronic device is provided, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method of the embodiments of the present application when executing the computer program.

[0052] In a fourth aspect, a computer readable storage medium is provided, which stores computer instructions, and the computer instructions are executed by a processor to implement the method of the embodiments of the present application.

[0053] The present application can realize efficient and intelligent reconnaissance task execution of the amphibious unmanned aerial vehicle in a complex indoor scene, enable the unmanned aerial vehicle to switch the path planning of the land-flight mode according to the real scene, and improve the adaptability and autonomy of the unmanned aerial vehicle in the indoor environment, and is especially suitable for indoor disaster relief scenes. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the specific embodiments or prior art of the present application, the drawings needed to be used in the description of the specific embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0055] Figure 1 The flowchart of the indoor autonomous planning method of the land-air amphibious unmanned aerial vehicle based on the large language model provided by the embodiments of the present application;

[0056] Figure 2 The functional structure diagram of the indoor autonomous planning device of the land-air amphibious unmanned aerial vehicle based on the large language model provided by the embodiments of the present application;

[0057] Figure 3 The functional structure diagram of the electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0059] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by those skilled in the art without creative labor based on the embodiments in the present application are within the scope of protection of the present application.

[0060] First, the design idea of the embodiments of the present application will be briefly introduced.

[0061] In recent years, the breakthrough of large language models in natural language processing and intelligent decision-making fields provides a new solution to the above problems. Large language models can efficiently analyze complex task instructions and combine multi-modal sensor data for environment perception and decision planning. However, how to deeply integrate large language models with hardware systems of land-air amphibious unmanned aerial vehicles, especially in complex indoor environments to achieve accurate task planning and execution, is still a technical problem to be solved. Key point constraint technology has made significant progress in path planning and autonomous navigation, providing a new solution to the above problems. By identifying key points and combining multi-modal sensor data, unmanned aerial vehicles can more accurately perceive the environment and make decision planning.

[0062] The application can realize efficient and intelligent reconnaissance task execution of an amphibious unmanned aerial vehicle in a complex indoor scene, enable the unmanned aerial vehicle to switch path planning between land and flight modes according to a real scene, and improve adaptability and autonomy of the unmanned aerial vehicle in an indoor environment, and is suitable for indoor disaster relief scenes.

[0063] After introducing the application scenarios and design ideas of the embodiments of the application, the technical solutions provided by the embodiments of the application are described below.

[0064] As shown in Figure 1 The application provides an indoor autonomous planning method for an amphibious unmanned aerial vehicle based on a large language model, which comprises the following steps:

[0065] Step 101: acquiring indoor environment perception data and IMU (Inertial Measurement Unit) data collected by an amphibious unmanned aerial vehicle at a current time; the indoor environment perception data comprises RGB images and point cloud data;

[0066] Step 102: processing the RGB images at the current time to obtain obstacle detection results of the indoor environment;

[0067] For example, the indoor environment comprises an office, a corridor and a staircase, and the obstacles comprise collapsed walls, rubbish piles and furniture.

[0068] Step 103: acquiring a task instruction input by a user;

[0069] Step 104: processing the task instruction and the indoor environment data by using a large language model to obtain task semantics, wherein the task semantics comprise a mode priority, a task target, a key point type and a constraint condition;

[0070] For example, the task instruction is to search for trapped personnel on the second floor; the task target is to search for trapped personnel on the second floor, the key point type is "corridor starting point", "obstacle front" and "obstacle rear", and the constraint condition is "land mode priority" and "flight mode obstacle avoidance".

[0071] Step 105: processing the RGB images and the point cloud data at the current time based on the key point type to obtain a plurality of key points and positions thereof, wherein each key point represents a semantic key point in the indoor environment;

[0072] Specifically, the key points are "obstacle front", "target point" and the like.

[0073] Step 106: Process the task semantics, indoor environment perception data, IMU data, obstacle detection results, and several key points and their positions using a large language model to generate the mode switching and path planning of the amphibious unmanned aerial vehicle at the next time, where the path of the amphibious unmanned aerial vehicle passes through the key points as much as possible.

[0074] In some embodiments, a high-definition camera and a laser radar are equipped on the amphibious unmanned aerial vehicle to support indoor reconnaissance and personnel search and rescue tasks. The high-definition camera serves as the main visual information acquisition sensor, providing rich image data. In land and air modes, the high-definition camera adjusts the shooting angle and range according to different task stages. In land mode, the camera mainly focuses on the terrain and obstacles in front and around to obtain clear images containing personnel, potential targets, and obstacles; in air mode, the camera shoots downward to provide wide-angle image information for target search and environment monitoring during flight. These images will be used as input for YOLOv8 target detection to identify different types of target objects such as trapped personnel, rescue equipment, or dangerous objects.

[0075] The laser radar is mainly responsible for accurate ranging and terrain mapping. In land and air modes, the laser radar plays an important role. On land, it can accurately scan the terrain profile and the distance of obstacles, providing distance information for the amphibious unmanned aerial vehicle's land travel; in the air, it can detect obstacles on the flight path to ensure flight safety. Its data will be used for subsequent environment perception and data fusion to assist YOLOv8 in judging the spatial position of targets and improving environmental information.

[0076] Specifically, the current time's RGB image is processed to obtain the indoor environment's obstacle detection results, including:

[0077] The current time's RGB image is processed using the YOLOv8 network to obtain the indoor environment's obstacle detection results, including: the category, size, and two-dimensional position of the obstacle.

[0078] Wherein, YOLOv8 is a target detection model used to identify different target objects in images captured by the camera. It is based on a convolutional neural network (CNN) architecture, which includes multiple convolutional layers, pooling layers, and fully connected layers. For input images, YOLOv8 divides them into multiple grids and predicts multiple bounding boxes and corresponding class probabilities for each grid.

[0079] For example, the task instructions and indoor environment data are processed using a large language model to obtain task semantics, including:

[0080] Based on the pre-trained general-purpose large language model (Qwen-7B), natural language instructions such as "avoid left obstacles and land on the red roof" are converted into executable semantic parameters. The user instruction is first processed by the BERT model for word segmentation, and 7 types of special markers such as OBSTACLE and ACTION are embedded to generate a semantic parsing tree containing target position, path constraints, and action instructions. For example, "red roof" in the instruction is located by the visual semantic extraction module and matched with the laser radar point cloud to verify.

[0081] In some embodiments, based on the key point type, the RGB image and the point cloud data at the current time are processed to obtain a plurality of key points and their positions, including:

[0082] The SAM model is used to perform semantic segmentation on the RGB image at the current time to obtain a segmentation mask, which includes the outline and type of each object;

[0083] The SAM model can classify each pixel in the image in a self-supervised learning manner, thereby extracting potential objects (such as walls, tables, chairs, doors, etc.) and outline information in the scene. The segmentation mask M generated in this process is a binary matrix, and each element in the matrix represents whether the corresponding pixel belongs to a certain specific object. Through the segmentation mask M, the accurate boundary and outline information of each object in the image can be obtained, providing basic data support for subsequent feature extraction and key point detection

[0084] The DINOv2 model is used to extract high-dimensional features from the RGB image at the current time to obtain a high-dimensional feature map, wherein each feature vector of the high-dimensional feature map corresponds to a region in the image and contains semantic description information of the region;

[0085] DINOv2 is a visual model based on self-supervised learning, which can extract rich semantic information from images, such as object categories, texture features, and spatial relationships. The feature map F extracted by DINOv2 not only describes the attributes of individual objects, but also reflects the relative positions and spatial relationships between objects, providing more comprehensive semantic support for subsequent key point detection.

[0086] The segmentation mask and the high-dimensional feature map are fused to obtain a set of feature vectors V marked as foreground by the segmentation mask;

[0087] Based on the key point type, the k-means clustering algorithm is used to cluster the feature vector set V to obtain the key points in the indoor environment;

[0088] The key points are mapped to the three-dimensional world coordinate system using the point cloud data to obtain the positions of the key points.

[0089] Among them, the extracted key points represent the semantically significant areas in the scene, such as object edges, corridor starting points, or near obstacles, which provide important references for UAV navigation and path planning. The mapped 3D key points represent the spatial position of a certain semantically significant area in the scene, providing precise spatial information support for UAV navigation. In this way, the UAV can better understand the three-dimensional structure of the scene, thereby optimizing path planning and obstacle avoidance strategies to ensure efficient execution of task instructions.

[0090] In some embodiments, the task semantics, indoor environment perception data, IMU data, obstacle detection results, and a number of key points and their positions are processed using a large language model to generate the mode switching and path planning of the amphibious UAV at the next moment, including:

[0091] Based on the obstacle detection results, indoor environment perception data, and task semantics at the current moment, determine the action mode of the amphibious UAV at the next moment; the action mode includes: land mode and flight mode;

[0092] According to the IMU data at the current moment, determine the motion state of the amphibious UAV at the current moment; wherein the motion state includes: position, speed and action mode;

[0093] According to the motion state of the amphibious UAV at the current moment and the action mode of the UAV at the next moment, determine whether to switch modes;

[0094] According to the task semantics and the positions of the key points, generate key point constraints;

[0095] According to the action mode of the amphibious UAV at the next moment, the key point constraints and the constraint conditions, use the position of the amphibious UAV at the current moment and the position of the key points to plan the path.

[0096] In some embodiments, based on the obstacle detection results, indoor environment perception data, and task semantics at the current moment, the action mode of the amphibious UAV at the next moment is determined, including:

[0097] When the height of the obstacle is greater than a first threshold and the width of the obstacle is greater than a second threshold (for example, the height is greater than 0.5m and the width is greater than 0.8m), determine the first action mode of the amphibious UAV at the next moment to be flight mode, and the first action mode value is 1; otherwise, determine the first action mode of the amphibious UAV at the next moment to be land mode, and the first action mode value is 0;

[0098] The point cloud data is processed to obtain a slope value and a flatness value of the front area. When the slope value is greater than 15° and the flatness value is greater than 0.1 m, it is determined that the second action mode of the amphibious unmanned vehicle at the next moment is the flight mode, and the second action mode value is 1; otherwise, it is determined that the second action mode of the amphibious unmanned vehicle at the next moment is the land mode, and the second action mode value is 0.

[0099] According to the mode priority in the task semantics, it is determined that the third action mode of the amphibious unmanned vehicle at the next moment is the flight mode, and the third action mode value is 1; otherwise, it is determined that the third action mode of the amphibious unmanned vehicle at the next moment is the land mode, and the third action mode value is 0.

[0100] For example, if the task semantics requires priority land mode (such as "search for trapped personnel"), the land mode is used as much as possible when there is no mandatory flight requirement. If the task semantics requires priority flight mode (such as "quickly reach the target point"), the flight mode is used as much as possible when there is no mandatory land requirement.

[0101] The action mode value of the amphibious unmanned vehicle at the next moment is calculated as:

[0102] Mode = w1 Mode1 + w2 Mode2 + w3 Mode3

[0103] wherein Mode1 is the first action mode value, w1 is the first weight, Mode2 is the second action mode value, w2 is the second weight, Mode3 is the third action mode value, and w3 is the third weight.

[0104] When Mode is not less than 0.5, it is determined that the final action mode of the amphibious unmanned vehicle at the next moment is the flight mode, otherwise, it is determined that the final action mode of the amphibious unmanned vehicle at the next moment is the land mode.

[0105] In some embodiments, according to the task semantics and the positions of the key points, a key point constraint is generated, including:

[0106] According to the task semantics, a task key point related to the task target is selected from the key points;

[0107] The key point constraint represents the spatial relationship between the key points, including a distance constraint and a direction constraint.

[0108] The distance constraint c1 is:

[0109] c1 = ||k1-k2|| ≤ dsafe

[0110] Where dsafe is the minimum safe distance between the amphibious UAV and obstacles; k1 is the three-dimensional position of the mission key point closest to the current position of the amphibious UAV; and k2 is the three-dimensional position of the mission key point at the next moment.

[0111] The directional constraint c2 is: to ensure that the line connecting the nearest mission critical point to the next mission critical point at the current moment is as close as possible to the flight direction of the amphibious UAV.

[0112] In some embodiments, path planning is performed using the current position of the amphibious drone and the positions of key points, based on the drone's motion mode, key point constraints, and constraint conditions at the next moment. This includes:

[0113] When the amphibious drone's action mode is land mode at the next moment, the position of the amphibious drone at the next moment is determined by the 2D A* algorithm according to the constraints. The position of the amphibious drone at the next moment should be as close as possible to the mission key point that satisfies the key point constraint.

[0114] When the amphibious drone's action mode is flight mode at the next moment, the 3DA* algorithm is used to determine the drone's position at the next moment based on the constraints. The drone's position at the next moment should be as close as possible to the mission's key points that satisfy the key point constraints.

[0115] In addition, the method also includes the step of training a large language model.

[0116] Based on the above embodiments, this application provides an indoor autonomous planning device for amphibious unmanned aerial vehicles (UAVs) based on a large language model. (See also...) Figure 2 As shown, the indoor autonomous planning device 200 for amphibious unmanned aerial vehicles based on a large language model provided in this application embodiment includes at least:

[0117] The first acquisition unit 201 is used to acquire indoor environmental perception data and IMU data collected by the amphibious UAV at the current moment; the indoor environmental perception data includes RGB images and point cloud data.

[0118] The first processing unit 202 is used to process the RGB image at the current moment to obtain the obstacle detection results of the indoor environment;

[0119] The second acquisition unit 203 is used to acquire the task instructions input by the user;

[0120] The second processing unit 204 is used to process the task instructions and indoor environment data using a large language model to obtain task semantics, which includes: pattern priority, task objective, key point type and constraint conditions.

[0121] The third processing unit 205 is used to process the RGB image and point cloud data at the current time based on the key point type to obtain several key points and their positions, each key point representing a semantic key point in the indoor environment.

[0122] The autonomous planning unit 206 is used to process task semantics, indoor environment perception data, IMU data, obstacle detection results and several key points and their locations using a large language model to generate the mode switching and path planning of the amphibious UAV in the next moment. The path of the amphibious UAV passes through all key points as much as possible.

[0123] It should be noted that the principle of the indoor autonomous planning device 200 for amphibious unmanned aerial vehicles based on a large language model provided in this application embodiment to solve the technical problem is similar to the method provided in this application embodiment. Therefore, the implementation of the indoor autonomous planning device 200 for amphibious unmanned aerial vehicles based on a large language model provided in this application embodiment can refer to the implementation of the method provided in this application embodiment, and the repeated parts will not be described again.

[0124] Based on the above embodiments, this application also provides an electronic device, see below. Figure 3 As shown, the electronic device 300 provided in this application embodiment includes at least: a processor 301, a memory 302, and a computer program stored in the memory 302 and capable of running on the processor 301. When the processor 301 executes the computer program, it implements the indoor autonomous planning method for amphibious unmanned aerial vehicles based on a large language model provided in this application embodiment.

[0125] The electronic device 300 provided in this application embodiment may further include a bus 303 connecting different components (including processor 301 and memory 302). The bus 303 represents one or more types of bus structures, including memory bus, peripheral bus, local area bus, etc.

[0126] The memory 302 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 3021 and / or cache memory 3022, and may further include read-only memory (ROM) 3023.

[0127] The memory 302 may also include a program tool 3025 having a set (at least one) of program modules 3024, including but not limited to: an operating subsystem, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0128] Electronic device 300 can also communicate with one or more external devices 304 (e.g., keyboard, remote control, etc.), and with one or more devices that enable a user to interact with electronic device 300 (e.g., mobile phone, computer, etc.), and / or with any device that enables electronic device 300 to communicate with one or more other electronic devices 300 (e.g., router, modem, etc.). This communication can be performed through input / output (I / O) interface 305. Furthermore, electronic device 300 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 306. Figure 3 As shown, network adapter 306 communicates with other modules of electronic device 300 via bus 303. It should be understood that, although... Figure 3 As not shown, other hardware and / or software modules may be used in conjunction with electronic device 300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, Redundant Arrays of Independent Disks (RAID) subsystems, tape drives, and data backup storage subsystems.

[0129] It should be noted that, Figure 3 The electronic device 300 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0130] This application also provides a computer-readable storage medium storing computer instructions. When executed by a processor, these instructions implement the indoor autonomous planning method for amphibious unmanned aerial vehicles (UAVs) based on a large language model provided in this application. Specifically, the executable program can be built into or installed in an electronic device 300, allowing the electronic device 300 to implement the indoor autonomous planning method for amphibious unmanned aerial vehicles (UAVs) based on a large language model provided in this application by executing the built-in or installed executable program.

[0131] The indoor autonomous planning method for amphibious unmanned aerial vehicles based on a large language model provided in this application embodiment can also be implemented as a program product. The program product includes program code. When the program product can run on the electronic device 300, the program code is used to enable the electronic device 300 to execute the indoor autonomous planning method for amphibious unmanned aerial vehicles based on a large language model provided in this application embodiment.

[0132] The program product provided in this application embodiment can be any combination of one or more readable media, wherein the readable media can be a readable signal medium or a readable storage medium, and the readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. Specifically, more specific examples of readable storage media (a non-exhaustive list) include: electrical connections with one or more wires, portable disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0133] The program product provided in this application embodiment can be a CD-ROM and include program code, and can also run on a computing device. However, the program product provided in this application embodiment is not limited thereto. In this application embodiment, the readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0134] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0135] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of this application do not depart from the spirit and scope of the technical solutions of this application, and should all be covered within the scope of the claims of this application.

Claims

1. A method for indoor autonomous planning of amphibious unmanned aerial vehicles (UAVs) based on a large language model, characterized in that, include: Acquire indoor environmental perception data and IMU data collected by the amphibious drone at the current moment; The indoor environment perception data includes RGB images and point cloud data; The RGB image at the current moment is processed to obtain the obstacle detection results of the indoor environment; Get the task instructions input by the user; The task instructions and indoor environment data are processed using a large language model to obtain task semantics, which include: pattern priority, task objective, key point type and constraint conditions. Based on the key point type, the RGB image and point cloud data at the current moment are processed to obtain several key points and their locations. Each key point represents a semantic key point in the indoor environment. By using a large language model to process task semantics, indoor environment perception data, IMU data, obstacle detection results, and several key points and their locations, the mode switching and path planning of the amphibious UAV in the next moment are generated. The path of the amphibious UAV passes through key points as much as possible.

2. The method according to claim 1, characterized in that, The RGB image at the current moment is processed to obtain obstacle detection results for the indoor environment, including: The RGB image at the current moment is processed using the YOLOv8 network to obtain obstacle detection results for the indoor environment. The obstacle detection results include the type, size, and two-dimensional position of the obstacle.

3. The method according to claim 1, characterized in that, Based on the keypoint type, the RGB image and point cloud data at the current moment are processed to obtain several keypoints and their locations, including: The SAM model is used to perform semantic segmentation on the RGB image at the current time to obtain a segmentation mask, which includes the outline and type of each object; The DINOv2 model is used to extract high-dimensional features from the RGB image at the current time to obtain a high-dimensional feature map. Each feature vector of the high-dimensional feature map corresponds to a certain region in the image and contains semantic description information of that region. The segmentation mask and the high-dimensional feature map are fused to obtain the feature vector set V of all segments labeled as foreground by the segmentation mask; Based on the key point type, the k-means clustering algorithm is used to cluster the feature vector set V to obtain the key points in the indoor environment; By using point cloud data, key points are mapped to a 3D world coordinate system to obtain the location of the key points.

4. The method according to claim 3, characterized in that, The system utilizes a large language model to process task semantics, indoor environmental perception data, IMU data, obstacle detection results, and several key points and their locations to generate the mode switching and path planning for the amphibious UAV in the next time step, including: Based on the obstacle detection results, indoor environment perception data, and task semantics at the current moment, the action mode of the amphibious UAV at the next moment is determined; the action mode includes: land mode and flight mode; Based on the IMU data at the current moment, determine the current motion status of the amphibious drone. Based on the current motion status of the amphibious drone and the drone's action mode in the next moment, determine whether to switch modes. Generate key point constraints based on task semantics and key point locations; Based on the amphibious drone's movement mode, key point constraints, and constraint conditions at the next moment, path planning is performed using the current position of the amphibious drone and the positions of key points.

5. The method according to claim 4, characterized in that, Based on the obstacle detection results, indoor environment perception data, and task semantics at the current moment, determine the action mode of the amphibious UAV at the next moment, including: If the height of the obstacle is greater than the first threshold and the width of the obstacle is greater than the second threshold, the first action mode of the amphibious drone at the next moment is determined to be flight mode, and the value of the first action mode is 1; otherwise, the first action mode of the amphibious drone at the next moment is determined to be land mode, and the value of the first action mode is 0. The point cloud data is processed to obtain the slope and flatness values ​​of the area in front. If the slope value is greater than 15° and the flatness value is greater than 0.1m, the second action mode of the amphibious drone at the next moment is determined to be the flight mode, and the second action mode value is 1; otherwise, the second action mode of the amphibious drone at the next moment is determined to be the land mode, and the second action mode value is 0. Based on the mode priority in the task semantics, determine the third action mode of the amphibious UAV at the next moment as flight mode, with a value of 1; otherwise, determine the third action mode of the amphibious UAV at the next moment as land mode, with a value of 0. Calculate the motion mode value of the amphibious drone at the next moment: Mode=w1·Mode1+w2·Mode2+w3·Mode3 Wherein, Mode1 is the first action mode value, w1 is the first weight, Mode2 is the second action mode value, w2 is the second weight, Mode3 is the third action mode value, and w3 is the third weight; If Mode is not less than 0.5, the final action mode of the amphibious drone at the next moment is determined to be flight mode; otherwise, the final action mode of the amphibious drone at the next moment is determined to be land mode.

6. The method according to claim 4, characterized in that, Based on the task semantics and the location of key points, key point constraints are generated, including: Based on the task semantics, select the task key points that are related to the task objective from the key points; The key point constraints include: distance constraints and direction constraints; Wherein, the distance constraint c1 is: c1=||k1-k2||≤dsafe Where dsafe is the minimum safe distance between the amphibious UAV and obstacles; k1 is the three-dimensional position of the mission key point closest to the current position of the amphibious UAV; and k2 is the three-dimensional position of the mission key point at the next moment. The directional constraint c2 is: to ensure that the line connecting the nearest mission critical point to the next mission critical point at the current moment is as close as possible to the flight direction of the amphibious UAV.

7. The method according to claim 6, characterized in that, Based on the amphibious drone's movement mode, key point constraints, and conditions at the next moment, path planning is performed using the drone's current position and key point locations, including: When the amphibious drone's action mode is land mode at the next moment, the position of the amphibious drone at the next moment is determined by the 2D A* algorithm according to the constraints. The position of the amphibious drone at the next moment should be as close as possible to the mission key point that satisfies the key point constraint. When the amphibious drone's action mode is flight mode at the next moment, the 3DA* algorithm is used to determine the drone's position at the next moment based on the constraints. The drone's position at the next moment should be as close as possible to the mission's key points that satisfy the key point constraints.

8. An indoor autonomous planning device for amphibious unmanned aerial vehicles based on a large language model, characterized in that, include: The first acquisition unit is used to acquire indoor environmental perception data and IMU data collected by the amphibious UAV at the current moment; The indoor environment perception data includes RGB images and point cloud data; The first processing unit is used to process the RGB image at the current moment to obtain the obstacle detection results of the indoor environment; The second acquisition unit is used to acquire the task instructions input by the user; The second processing unit is used to process task instructions and indoor environment data using a large language model to obtain task semantics, which includes: pattern priority, task objective, key point type and constraint conditions. The third processing unit is used to process the RGB image and point cloud data at the current moment based on the key point type to obtain several key points and their locations. Each key point represents a semantic key point in the indoor environment. The autonomous planning unit is used to process task semantics, indoor environment perception data, IMU data, obstacle detection results, and several key points and their locations using a large language model to generate the mode switching and path planning of the amphibious UAV for the next moment. The path of the amphibious UAV passes through key points as much as possible.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-7.

Images