A flight control method, device, equipment and medium of a UAV

Abstract

The application discloses a flight control method, device and equipment of a UAV, and a medium. The method comprises: acquiring video stream data collected by a monocular RGB camera equipped on the UAV; performing scene understanding on the video stream data to acquire multi-vision attribute information comprising target categories, position information, color information and depth information; generating a prompt word based on the multi-vision attribute information, and generating a flight plan by a large model based on the prompt word; and performing flight control on the UAV based on the flight plan. According to the application, the video stream data is collected by the monocular RGB camera equipped on the UAV, and scene understanding is performed on the video stream data to acquire multi-vision attribute information comprising target categories, position information, color information and depth information. By setting the color information and the depth information in the vision attribute information, the large model can more accurately locate the predetermined target, which is conducive to the understanding of the scene by the large model and improves the accuracy of the flight plan output by the large model. Meanwhile, the depth information can also assist the UAV in obstacle avoidance and speed regulation, thereby ensuring the accuracy and safety of the UAV control.

Claims

1. A flight control method for an unmanned aerial vehicle (UAV), characterized in that, The flight control method for the aforementioned UAV specifically includes: Acquire video stream data captured by a monocular RGB camera mounted on a drone; Scene understanding is performed on the video stream data to obtain multi-visual attribute information, wherein the multi-visual attribute information includes target category, location information, color information, and depth information; Based on the multi-visual attribute information, prompt words are generated, and a flight plan is generated based on the prompt words using a large model. The UAV is controlled in flight based on the flight plan.

2. The flight control method for an unmanned aerial vehicle according to claim 1, characterized in that, The process of performing scene understanding on the video stream data to obtain multi-visual attribute information specifically includes: The instance segmentation model is used to determine the target category, location information of the main target, and segmentation mask of the video frame in the video stream data. The depth map of the video frame is determined by a monocular depth estimation model, and the depth information of the main target is determined based on the depth map and the position information of the main target. The color information of the main target is determined based on the segmentation mask and the location information of the main target. Based on the target category, location information, color information, and depth information of the main target, determine the multi-visual attribute information of the main target to obtain multi-visual attribute information.

3. The flight control method for an unmanned aerial vehicle according to claim 2, characterized in that, The step of determining the color information of the main target based on the segmentation mask and the location information of the main target specifically includes: Based on the location information of the main target, a mask region corresponding to the main target is selected in the segmentation mask, and RGB value clustering is performed on the pixels included in the mask region corresponding to the main target to determine the RGB value of the main target; The RGB values ​​of the main target are converted into natural language descriptions of color to obtain the color information of the main target.

4. The flight control method for an unmanned aerial vehicle according to claim 3, characterized in that, The step of clustering the RGB values ​​of the pixels included in the mask region corresponding to the main target to determine the RGB value of the main target specifically includes: The pixels included in the target mask region corresponding to the main target are clustered by RGB values ​​to obtain several RGB clusters; Select the target RGB cluster containing the most pixels from several RGB clusters, and use the RGB value of the cluster center of the target RGB cluster as the RGB value of the main target.

5. The flight control method for an unmanned aerial vehicle according to claim 3, characterized in that, The method further includes: The passability map corresponding to the video frame in the video stream data is extracted using the gradient information of the depth map; The object state information is calculated using the depth map, wherein the object state information includes a static state or a moving state; The accessibility map and the object state information are added as visual attributes to the multi-visual attribute information.

6. The flight control method for an unmanned aerial vehicle according to claim 1, characterized in that, The specific steps of generating prompt words based on the multi-visual attribute information include: Receive task description input from the user; Acquire system knowledge and add syntax checking information and multiple visual attribute examples to the system knowledge. The syntax checking information includes several syntax checking items and several syntax error items. The syntax error items include syntax error examples and their corresponding correction methods. The multiple visual attribute examples include color application examples and / or depth application examples. Prompt words are constructed based on the task description, the system knowledge, and the multi-visual attribute information.

7. The flight control method for an unmanned aerial vehicle according to claim 1, characterized in that, The onboard computing platform of the drone is equipped with a large model, and the process of generating a flight plan based on the prompt words using the large model is as follows: A flight plan is generated based on the prompt words by combining a large model carried on the drone's onboard computing platform and / or a large model carried on the corresponding ground-based computing device of the drone.

8. A flight control device for an unmanned aerial vehicle (UAV), characterized in that, The flight control device for the aforementioned UAV specifically includes: The visual encoding module is used to acquire video stream data collected by a monocular RGB camera equipped on a drone, and to perform scene understanding on the video stream data to obtain multi-visual attribute information, wherein the multi-visual attribute information includes target category, position information, color information and depth information; The prompt word generation module is used to generate prompt words based on the multi-visual attribute information; The large model module is used to generate flight plans based on the prompt words using a large model; The drone control module is used to control the drone's flight based on the flight plan.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps in the flight control method for the unmanned aerial vehicle as described in any one of claims 1-7.

10. A terminal device, characterized in that, include: Processor and memory; The memory stores a computer-readable program that can be executed by the processor; When the processor executes the computer-readable program, it implements the steps in the flight control method for an unmanned aerial vehicle as described in any one of claims 1-7.

Images