Unmanned aerial vehicle lighting method, system and device

Abstract

The invention discloses an unmanned aerial vehicle lighting method, system and device. The invention aims to realize intelligent illumination of a target object through advanced image processing and deep learning technologies. The system ensures that the unmanned aerial vehicle can accurately follow and illuminate a specified target through the steps of user input, image acquisition and analysis, target identification, attitude tracking, position adjustment and the like, and dynamically adjusts the flight position and the illumination angle of the unmanned aerial vehicle to adapt to the movement of the target.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicles, and relates to a method, a system and a device for illuminating an unmanned aerial vehicle. Background Art

[0002] With the development of unmanned aerial vehicle technology, its application scope is becoming wider and wider. However, most of the existing unmanned aerial vehicles do not have the function of illumination, and the pictures taken at night or under poor light conditions are relatively blurred, which brings great trouble to aerial photography.

[0003] In a complex and dynamic environment, it is difficult for traditional methods to accurately identify the main target specified by the user and continuously track its position. When the main target is in a moving state, the existing illumination methods of unmanned aerial vehicles cannot adjust their own positions and illumination angles in a timely manner according to the movement information of the main target. Summary of the Invention

[0004] In order to solve the problems in the background art, the present invention provides a method, a system and a device for illuminating an unmanned aerial vehicle.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows: On the one hand, the present invention provides a method for illuminating an unmanned aerial vehicle, including: The user inputs the target information to be illuminated, and defines the target to be illuminated as the main target.

[0006] Continuously collect multiple environmental images through an image acquisition device, and identify the main target to be illuminated and secondary targets other than the main target in the first environmental image through image segmentation and a Transformer model.

[0007] Mark the main target in the multiple continuously collected environmental images.

[0008] Determine the position of the main target in the first environmental image through a capsule neural network, add the movement information of the unmanned aerial vehicle, mark the movement information on multiple environmental images, confirm the movement information of the main target through the capsule neural network, and after the confirmation is completed, obtain the relative position of the main target with respect to the static secondary targets in multiple environmental images.

[0009] Based on the change in the relative position of the main target with respect to the static secondary targets, adjust the flight position and illumination angle of the unmanned aerial vehicle.

[0010] Further, the specific method for identifying the main target to be illuminated in the first environmental image through image segmentation and a Transformer model is: Filter the first environmental image to remove the noise in the first environmental image; Perform object detection on the first environmental image based on a convolutional neural network, use a region proposal network to generate candidate boxes in the first environmental image, and then refine the object boundaries through classification and regression; Extract the exact shape and boundaries of each object in the first environmental image through image segmentation, analyze the main object through a transformer model, and determine whether it is the main object based on the characteristics of the objects in the first environmental image; Identify and label the main object and secondary objects in the first environmental image.

[0011] Further, the specific method for labeling the main object in multiple continuously acquired environmental images is as follows: The capsule neural network distinguishes the different features of the main object and secondary objects in the first environmental image through a dynamic routing mechanism; Mark the corresponding main object features in multiple environmental images, extract the pose information of the corresponding main object features from multiple environmental images, and screen out the objects with pose information similar to the main object in the first environmental image as the main object through the k-nearest neighbor method; The capsule neural network distinguishes the different poses of the main object and secondary objects through a dynamic routing mechanism, extracts the objects with pose information and feature information consistent with the main object in multiple environmental images as the main object, and completes the positioning of the main object position in multiple environmental images.

[0012] Further, the specific method for confirming the movement information of the main object through the capsule neural network is as follows: Based on the movement information marking, extract the static secondary objects in the environmental image obtained at the previous moment, and judge the moving vector of the static secondary objects in the image based on the drone movement judgment and the objects corresponding to the drone movement; When the static secondary objects in the image move, use the moving vector of the static secondary objects as the reference value, observe the vector displacement of the static secondary objects in consecutive frames through the capsule neural network, and superimpose the moving vector of the static secondary objects on the moving vector of the main object to calculate the moving vector of the main object in the current environmental image.

[0013] Further, the specific method for obtaining the relative position of the main object with respect to the static secondary objects is as follows: Extract the main object and secondary objects from the first environmental image through the capsule neural network, and define the main object as a vector and define multiple secondary objects as a set ; First, based on the positions of the secondary objects in the first environmental image and the positions of the secondary objects in multiple environmental images, determine the static secondary objects and generate a set of static secondary objects ; Based on the positional gap between the main target vector and the static secondary target vector, the positional relationship of the main target with respect to the static secondary target is calculated. The specific formula is: ; where is the relative position of the main target with respect to the th static secondary target, is the vector corresponding to the th static secondary target is a function for calculating the absolute position of the main target vector in the first environmental image.

[0014] On the other hand, the present invention provides a drone lighting system that executes the above-mentioned drone lighting method, including: A user input module, where the user inputs the target information to be illuminated and defines the target to be illuminated as the main target; A data acquisition module, which continuously acquires multiple environmental images through an image acquisition device; An image recognition module, which identifies the main target to be illuminated and secondary targets other than the main target in the first environmental image through image segmentation and a Transformer model; A data processing module, which determines the position of the main target in the first environmental image through a capsule neural network, adds the drone movement information, marks the movement information on multiple environmental images, and confirms the movement information of the main target through the capsule neural network. After confirmation, the relative position of the main target with respect to the static secondary targets is obtained in multiple environmental images; A drone control module, which adjusts the flight position and lighting angle of the drone based on the change in the relative position of the main target with respect to the static secondary targets.

[0015] On the other hand, the present invention provides a drone lighting device, including a memory and a processor. The above-mentioned drone lighting system is stored in the memory, and the processor calls the drone lighting system in the memory during operation.

[0016] Compared with the prior art, the present invention has the following beneficial effects: By using image segmentation, a Transformer model, and a capsule neural network, the present invention can more accurately distinguish the main target from the environment and stably track the position change of the main target in multiple continuously acquired environmental images.

[0017] By introducing drone movement information marking and relative position analysis of static secondary targets, the present invention can effectively calculate the movement vector of the main target, enabling the drone to make corresponding adjustments to maintain the lighting effect.

[0018] Based on the position change of the main target relative to the static secondary target, the flight position and lighting angle of the drone are automatically adjusted to ensure that the lighting always focuses on the main target, enabling precise lighting even in a complex three-dimensional space. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is the flowchart of the method of the present invention; Figure 2 is the structural diagram of the system of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0021] As Figure 1 shown, the technical solution adopted by the present invention is as follows: A drone lighting method, including: The user inputs the target information to be illuminated, and defines the target to be illuminated as the main target.

[0022] The image acquisition device continuously acquires multiple environmental images, and through image segmentation and the transformer model, the main target to be illuminated and the secondary targets other than the main target in the first environmental image are identified.

[0023] The main target is marked in the multiple continuously acquired environmental images.

[0024] The position of the main target in the first environmental image is determined through the capsule neural network, the drone movement information is added, the movement information is marked on the multiple environmental images, and the movement information of the main target is confirmed through the capsule neural network. After the confirmation is completed, the relative position of the main target to the static secondary target is obtained in the multiple environmental images.

[0025] Based on the relative position change of the main target to the static secondary target, the flight position and lighting angle of the drone are adjusted.

[0026] The user inputs the target information to be illuminated, and defines the target to be illuminated as the main target. Guided by the user's needs, the object to be illuminated is clarified, so that the lighting action of the drone is targeted, avoiding blind lighting and improving the utilization efficiency of lighting resources.

[0027] Continuously collect multiple environmental images through an image acquisition device, and identify the main target that needs to be illuminated and secondary targets other than the main target in the first environmental image through image segmentation and a Transformer model. Continuously collecting images lays the foundation for subsequent dynamic tracking. The combination of image segmentation and the Transformer model can finely analyze the image content and accurately distinguish the main and secondary targets. In complex scenarios, such as construction sites and wild rescue sites, it can also quickly locate key and auxiliary lighting objects, providing a basis for precise lighting planning.

[0028] Filter the first environmental image to remove the noise in it. Purify the image data, reduce noise interference, greatly improve the accuracy of subsequent target detection and recognition, avoid misjudging targets due to noise points, and ensure that the lighting planning is based on clear and accurate visual information.

[0029] Perform target detection on the first environmental image based on a convolutional neural network. Use the region proposal network to generate candidate boxes in the first environmental image, and then refine the target boundaries through classification and regression. The convolutional neural network efficiently scans the image, the region proposal network initially frames the target range, and classification and regression further analyze the target contours in the image, narrowing the range for subsequent accurate identification of the main target and accelerating the processing flow.

[0030] Extract the precise shape and boundary of each target in the first environmental image through image segmentation, analyze the main target through the Transformer model, and judge whether it is the main target according to the characteristics of the targets in the first environmental image; identify and label the main target and secondary targets in the first environmental image. Image segmentation presents the target details, and the Transformer model analyzes the corresponding characteristics of the main target based on semantics. The two work together to accurately distinguish the main target, which is convenient for subsequent retrieval of target information at any time. Whether it is planning the lighting angle or tracking the target movement, there is an exact reference.

[0031] Label the main target in the continuously collected multiple environmental images. Continuously track the main target to provide continuous reference for dynamically adjusting the lighting direction and position.

[0032] The capsule neural network distinguishes the different features of the main target and the secondary target in the first environmental image through a dynamic routing mechanism; marks the corresponding main target features in multiple environmental images, extracts the pose information of the corresponding main target features from multiple environmental images, and screens out the targets with pose information similar to the main target in the first environmental image as the main target through the k-nearest neighbor method; the capsule neural network distinguishes the different poses of the main target and the secondary target through a dynamic routing mechanism, extracts the targets with pose information and feature information consistent with the main target in multiple environmental images as the main target, and completes the positioning of the main target position in multiple environmental images. The capsule neural network uses the dynamic routing to accurately identify the target features and poses, and the k-nearest neighbor method assists in quick matching to achieve the accurate positioning of the main target in continuous images, enabling the drone to keep track of the main target in real time. Even when the target is in the process of displacement, the moving direction and moving result of the target can be calculated.

[0033] Determine the position of the main target in the first environmental image through the capsule neural network, add the drone movement information, mark the movement information on multiple environmental images, and confirm the movement information of the main target through the capsule neural network. Combine the drone's own movement data with the perception of the main target by the capsule neural network to comprehensively track the dynamics of the main target, mark the movement information on the image, clearly present the target trajectory, and provide key dynamic data for subsequent accurate calculation of the relative position and adjustment of lighting parameters.

[0034] CNN focuses on detecting important features in image pixels. The high-level features are the weighted sum of the combinations of low-level features. The activation of the previous layer is multiplied and added to the weights of the neurons in the next layer, and then activated through a non-linear activation function. In such an architecture, the positional relationship between high-level features and low-level features becomes blurred. The capsule neural network encodes the probability of feature detection as the length of its output vector. The state of the detected feature is encoded as the direction pointed by the vector. So, when the detected feature moves in the image or its state changes somehow, the probability remains unchanged, but its direction changes. Using the capsule neural network can analyze the movement of the main target by combining the positional relationship between the main target and the secondary target.

[0035] Based on mobile information markers, extract static secondary targets in the environmental image obtained at the previous moment. Based on the movement of the drone and the corresponding targets during the drone's movement, determine the movement vector of the static secondary targets in the image. When the static secondary targets in the image move, use the movement vector of the static secondary targets as the reference value, observe the vector displacement of the static secondary targets within consecutive frames through a capsule neural network, and superimpose the movement vector of the static secondary targets on the movement vector of the main target to calculate the movement vector of the main target in the current environmental image. By using the movement reference of the static secondary targets and combining the measurement of the capsule neural network, the movement vector of the main target can be obtained, enabling real-time analysis of the movement trend of the main target and timely adjustment of the flight attitude and lighting direction to adapt to complex and changing lighting scenarios.

[0036] Obtain the relative position of the main target with respect to the static secondary targets. Extract the main target and secondary targets from the first environmental image through a capsule neural network, define the main target as a vector, and define multiple secondary targets as a set. First, based on the positions of the secondary targets in the first environmental image and their positions in multiple environmental images, determine the static secondary targets and generate a set of static secondary targets; calculate the positional relationship of the main target with respect to the static secondary targets based on the positional differences between the main target vector and the static secondary target vectors. The specific formula is: ; where is the relative position of the main target with respect to the th static secondary target, is the vector corresponding to the th static secondary target, and is a function for calculating the absolute position of the main target vector in the first environmental image. Quantify the relative position of the main target and the static secondary targets to provide a specific calculation basis for adjusting the flight position and lighting angle of the drone, and achieve precise lighting control.

[0037] Based on the change in the relative position of the main target with respect to the static secondary targets, adjust the flight position and lighting angle of the drone. Respond in real time to the change in relative position, dynamically optimize the drone's attitude and lighting direction to ensure that the main target is always in the best lighting area.

[0038] On the other hand, as Figure 2 shown, the present invention provides a drone lighting system that executes the above-mentioned drone lighting method, including: A user input module, where the user inputs the target information to be illuminated and defines the target to be illuminated as the main target; A data acquisition module, which continuously acquires multiple environmental images through an image acquisition device; The image recognition module identifies the main target to be illuminated and secondary targets other than the main target in the first environmental image through image segmentation and the Transformer model; The data processing module determines the position of the main target in the first environmental image through the capsule neural network, adds the UAV movement information, marks the movement information on multiple environmental images, and confirms the movement information of the main target through the capsule neural network. After the confirmation is completed, the relative position of the main target to the static secondary targets is obtained in multiple environmental images; The UAV control module adjusts the flight position and lighting angle of the UAV based on the change in the relative position of the main target to the static secondary targets.

[0039] On the other hand, the present invention provides a UAV lighting device, including a memory and a processor. The above-mentioned UAV lighting system is stored in the memory, and the processor calls the UAV lighting system in the memory during operation.

[0040] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for lighting a drone, characterized in that: Included are: The user inputs the target information that needs to be illuminated, and defines the target that needs to be illuminated as the main target; Continuously collecting multiple environmental images through an image acquisition device, and identifying a main target that needs to be illuminated and secondary targets other than the main target in the first environmental image through image segmentation and a transformer model; Mark the main target in multiple environment images collected continuously; The position of the main target in the first environment image is determined by the capsule neural network, the movement information of the drone is added, the movement information is marked on multiple environment images, and the movement information of the main target is confirmed by the capsule neural network. After the confirmation is completed, the relative position of the main target to the static secondary target is obtained in multiple environment images; Adjust the UAV flight position and lighting angle based on the relative position change of the primary target to the static secondary target.

2. The method for lighting a drone according to claim 1, characterized in that: The specific method of identifying the main target that needs to be illuminated in the first environment image through image segmentation and transformer model is: Filtering the first environment image to remove noise in the first environment image; Perform target detection on the first environment image based on a convolutional neural network, generate candidate boxes in the first environment image using a region proposal network, and then refine the target boundary through classification and regression; The precise shape and boundary of each target in the first environment image are extracted through image segmentation, and the main target is analyzed through the transformer model. The characteristics of the target in the first environment image are used to determine whether it is the main target. The primary and secondary targets are identified and annotated in the first environment image.

3. The method for lighting a drone according to claim 1, characterized in that: The specific method of marking the main target in the continuously collected multiple environment images is: The capsule neural network distinguishes the different features of the primary and secondary targets in the first environment image through a dynamic routing mechanism; By marking the corresponding main target features in multiple environmental images, the posture information of the corresponding main target features is extracted from the multiple environmental images, and the target with posture information similar to the main target in the first environmental image is screened out from the multiple environmental images by the k-nearest neighbor method and marked as the main target; The capsule neural network distinguishes the different postures of the main target and the secondary target through a dynamic routing mechanism, extracts the target whose posture information and feature information are consistent with the main target from multiple environmental images as the main target, and completes the positioning of the main target in multiple environmental images.

4. The method for lighting a drone according to claim 3, characterized in that: The specific method of confirming the movement information of the main target through the capsule neural network is: Based on the movement information mark, the static secondary target in the environment image acquired at the last moment is extracted, and based on the UAV movement judgment and the target corresponding to the UAV movement, the movement vector of the static secondary target in the image is determined; The static secondary target in the image moves. The motion vector of the static secondary target is used as the reference value. The vector displacement of the static secondary target in continuous frames is observed through the capsule neural network. The motion vector of the static secondary target is superimposed on the motion vector of the main target to calculate the motion vector of the main target in the current environment image.

5. The method for lighting a drone according to claim 4, characterized in that: The specific method for obtaining the relative position of the main target to the static secondary target is: The capsule neural network extracts the main target and the secondary target from the first environment image, and defines the main target as the vector , multiple secondary goals are defined as sets ; First, according to the position of the secondary target in the first environment image and the position of the secondary target in multiple environment images, the static secondary target is determined to generate a static secondary target set. ; According to the position difference between the main target vector and the static secondary target vector, the position relationship of the main target with respect to the static secondary target is calculated. The specific formula is: ； in Main goal about The relative position of the static secondary targets, For the Static secondary target corresponding vector To calculate the main target vector Function of the absolute position in the first environment image.

6. A UAV lighting system, executing the UAV lighting method as claimed in claim 1, characterized in that: Included are: A user input module, in which the user inputs information of a target to be illuminated, and defines the target to be illuminated as a main target; A data acquisition module continuously acquires multiple environmental images through an image acquisition device; The image recognition module uses image segmentation and the transformer model to identify the main target that needs to be illuminated and the secondary targets other than the main target in the first environment image; The data processing module determines the position of the main target in the first environmental image through the capsule neural network, adds the movement information of the drone, marks the movement information on multiple environmental images, confirms the movement information of the main target through the capsule neural network, and obtains the relative position of the main target to the static secondary target in multiple environmental images after confirmation; The drone control module adjusts the drone flight position and lighting angle based on the relative position change of the primary target to the static secondary target.

7. A lighting device for a drone, comprising a memory and a processor, characterized in that: The memory stores the UAV lighting system according to claim 6, and the processor calls the UAV lighting system in the memory during operation.

Images