A drone light painting method based on neural networks and artistic programming

Abstract

The present invention discloses a light painting method for an unmanned aerial vehicle (UAV) based on a neural network and artistic programming, comprising the following steps: S1, selecting an original image for light painting as input, and creating a mask training set; S2, building an image segmentation model; S3, obtaining a feature map; S4, drawing a route map; S5, generating and exporting a route map file in a kml format; S6, importing the route map file in the kml format into litchi, and adjusting punctuation positions as required; S7, litchi exporting a csv format file and saving it to an intelligent UAV; S8, completing shooting by superimposing a sequence of long exposures using a moving window, performing segmentation using a UNet, obtaining a feature map suitable for drawing a route, generating a kml file using Google Earth as input to a litchi interface, then outputting a csv file for application to the flight route of the UAV, and adjusting the altitude angle, which can be achieved using a coordinate conversion program, and then inputting the obtained csv file into litchi to complete all preparations, thereby achieving a smoother light painting process by opening up the operation route.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a drone light painting method based on neural networks and artistic programming. Background Art

[0002] From military to commercial use to today's everyday use, drone technology is becoming increasingly integrated into our lives. Today, drones are not only a must-have for geeks, but also a creation for many programming enthusiasts. Furthermore, leading drone technology companies are continuously expanding the scope of drone applications and providing various programmable platforms, giving countless enthusiasts unlimited creative opportunities.

[0003] Currently, drone-based content is plentiful and popular across various video platforms. There are two general approaches to enhancing video quality: first, using hardware to assist with filming, such as adding lights to drones or upgrading cameras to improve quality; second, leveraging programming interfaces to access drones' intelligent features and create new ways to play.

[0004] Light painting utilizes a drone's intelligent flight modes to record the light trails created by the drone's flight path by pre-setting a route and employing time-lapse photography. However, light painting faces three practical challenges, and effective solutions have been lacking: how to obtain a feature map suitable for drawing the trajectory from ordinary photos; how to adjust the height and angle of the route to best suit the needs; and how to streamline the operation of the route for a smoother light painting process. Summary of the Invention

[0005] In response to the above technical problems, the present invention provides a drone light painting method based on neural networks and artistic programming, which can ultimately make the light painting process smoother.

[0006] In order to solve the above technical problems, the technical solution of the present invention is:

[0007] A drone light painting method based on neural networks and artistic programming includes the following steps:

[0008] S1. Select the original image used for light painting as input and create a mask training set;

[0009] S2. Build an image segmentation model;

[0010] S3. Get feature map

[0011] The original image to be light-painted is used as input, and a high-quality feature map suitable for path drawing is obtained through the image segmentation model;

[0012] S4. Draw a roadmap

[0013] S5. Generate and export the route map file in kml format from Google Earth;

[0014] S6. Import the kml format route map file into litchi and adjust the punctuation position as required;

[0015] S7 and litchi export CSV format files and save them to the drone;

[0016] S8. Use the moving window long exposure sequence superposition to complete the shooting.

[0017] Preferably, in step S2

[0018] S2-1. Select the U-Net network as the initial model. The U-Net network includes a convolution layer with a convolution kernel size of 3*3, cropping and copying, a maximum pooling operation with a convolution kernel of 2*2, an upsampling operation with a convolution kernel of 2*2, and a convolution layer with a convolution kernel of 1*1;

[0019] S2-2. Use the existing original images and mask training sets to perform targeted training on the U-Net network.

[0020] Preferably, in the U-Net network, the convolution operation uses a total of 19 convolutions, and the other pooling, upsampling, cropping and copying operations are all used four times.

[0021] Preferably, the U-Net network uses skip connections to directly connect the feature maps between the encoder and the decoder.

[0022] As a preference, the implementation method of step S4 is

[0023] S4-1. Draw on Google Earth based on the feature map. First, add the feature map according to the software function of Google Earth and place it according to the required location while ensuring safety.

[0024] S4-2. Add a route layer to the feature map added in S4-1 based on the software function of Google Earth, draw the route based on the processed feature map, and meet the accuracy and safety requirements.

[0025] Preferably, in step S6, if the drawn pattern does not meet the accuracy and safety requirements, the punctuation points can be directly dragged to make adjustments on the litchi interface.

[0026] Preferably, in step S7, if the height and angle need to be adjusted, then step S7-1 is entered, and after the adjustment is completed, step S8 is entered; if no adjustment is required, step S8 is entered directly.

[0027] In step S7-1, the CSV format file exported by litchi is imported into the coordinate conversion program, and the pattern is rotated and transformed in the air to achieve adjustment of any height and angle.

[0028] The present invention has the following characteristics and beneficial effects:

[0029] The above technical solution mainly uses the UNet neural network to segment the input ordinary image and obtain a feature map suitable for drawing the route. Then, the kml file is generated through Google Earth as the input of the litchi interface. The output csv file can be directly applied to the flight route of the drone. In addition, the altitude angle can be adjusted using the coordinate conversion program to better meet the needs. Then, the obtained csv file is input into litchi to complete all preparations, thereby opening up the operation route and making the light painting process smoother. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0031] Figure 1 , a flow chart of a method according to an embodiment of the present invention.

[0032] Figure 2 , Schematic diagram of the U-Net network structure in an embodiment of the present invention.

[0033] Figure 3 , U-Net network test case in an embodiment of the present invention.

[0034] Figure 4 , a schematic diagram of route drawing in an embodiment of the present invention.

[0035] Figure 5 , schematic diagram of angle conversion in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0037] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0038] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0039] The present invention provides a UAV light painting method based on neural network and artistic programming, such as Figure 1 As shown, the following steps are included:

[0040] S1. Select the original image used for light painting as input and create a mask training set;

[0041] S2. Build an image segmentation model;

[0042] Specifically, in step S2

[0043] S2-1. Select the U-Net network as the initial model. The U-Net network includes a convolution layer with a convolution kernel size of 3*3, cropping and copying, a maximum pooling operation with a convolution kernel of 2*2, an upsampling operation with a convolution kernel of 2*2, and a convolution layer with a convolution kernel of 1*1;

[0044] S2-2. Use the existing original images and mask training sets to perform targeted training on the U-Net network.

[0045] It can be understood that the selected U-Net network is an important network for image segmentation, and its structure is as follows Figure 2As shown in the figure, a U-Net network can be used as input to obtain an image with prominent features of the main characters or objects. The U-Net network mainly consists of an encoder and a decoder, forming an overall U-shape. The arrows in the lower right corner of the figure represent: a convolution layer with a 3*3 kernel size (convolution operation followed by a ReLu activation function), cropping and copying, a maximum pooling operation with a 2*2 kernel size, upsampling (deconvolution) with a 2*2 kernel size, and a convolution layer with a 1*1 kernel size. As can be seen from the figure, convolution is the most common operation in the entire U-Net network, with a total of 19 convolutions. The other operations, pooling, upsampling, cropping, and copying, are used four times each.

[0046] The U-Net network uses skip connections to directly connect the feature maps between the encoder and decoder, resulting in the fusion of semantically dissimilar feature maps. This can better ensure that the required features can be fully extracted.

[0047] In addition, through Figure 3 It can be seen that by performing targeted training in step S2-2, a model with strong task capability, high image segmentation accuracy and high efficiency can be obtained.

[0048] S3. Get feature map

[0049] The original image to be light-painted is used as input, and a high-quality feature map suitable for path drawing is obtained through the image segmentation model;

[0050] S4. Draw a roadmap

[0051] Specifically, such as Figure 4 As shown, the method includes

[0052] S4-1. Draw on Google Earth based on the feature map. First, add the feature map according to the software function of Google Earth and place it according to the required location while ensuring safety.

[0053] S4-2. Add a route layer to the feature map added in S4-1 based on the software function of Google Earth, draw the route based on the processed feature map, and meet the accuracy and safety requirements.

[0054] S5. Generate and export the route map file in kml format from Google Earth;

[0055] S6. Import the kml format route map file into litchi and adjust the punctuation position as required. If the drawn pattern does not meet the accuracy and safety requirements, directly drag the punctuation to adjust it in the litchi interface.

[0056] S7 and litchi export CSV format files and save them to the drone;

[0057] Further, such as Figure 5 As shown, the step S7 also includes step S7-1, importing the CSV format file exported by litchi into a coordinate conversion program, allowing the pattern to rotate and transform in the air to achieve adjustment of any height and angle.

[0058] It can be understood that in step S7, if the height and angle need to be adjusted, then step S7-1 is entered, and after the adjustment is completed, step S8 is entered; if no adjustment is required, step S8 is directly entered.

[0059] It should be noted that the coordinate conversion program is a program for implementing simple conversion, which can be easily implemented by those skilled in the art and can also be implemented by simple substitution. Therefore, it is not described in detail in this embodiment.

[0060] S8. Use the moving window long exposure sequence superposition to complete the shooting.

[0061] Through the above technical solution, the image segmentation model built with the U-Net neural network is used to segment the input ordinary image to obtain a feature map suitable for drawing the route. Then, the kml file is generated through Google Earth as the input of the litchi interface. The output csv file can be directly applied to the flight route of the drone. In addition, the altitude angle can be adjusted using the coordinate conversion program to better meet the needs while improving accuracy and safety. The obtained csv file is then input into litchi to complete all preparations, thereby opening up the operation route to make the light painting process smoother, effectively improving the accuracy of light painting, and making the light-painted image clearer.

[0062] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. It will be apparent to those skilled in the art that various changes, modifications, substitutions, and variations of these embodiments, including components, without departing from the principles and spirit of the present invention are still within the scope of protection of the present invention.

Claims

1. A drone light painting method based on neural networks and artistic programming, characterized in that: The steps include: S1. Select the original image used for light painting as input and create a mask training set; S2. Build an image segmentation model; S2-1. Select a U-Net network as the initial model. The U-Net network includes a convolution layer with a convolution kernel size of 3*3, cropping and copying, a maximum pooling operation with a convolution kernel of 2*2, an upsampling operation with a convolution kernel of 2*2, and a convolution layer with a convolution kernel of 1*1. In the U-Net network, the convolution operation uses 19 convolutions in total, and the other pooling, upsampling, cropping, and copying operations are used four times each. The U-Net network uses skip connections to directly connect the feature maps between the encoder and the decoder. S2-2, use the existing original image and mask training set to perform targeted training on the U-Net network; S3. Get feature map The original image to be light-painted is used as input, and a high-quality feature map suitable for path drawing is obtained through the image segmentation model; S4. Draw a roadmap S4-1. Draw on Google Earth based on the feature map. First, add the feature map according to the software function of Google Earth and place it according to the required location while ensuring safety. S4-2. Add a route layer to the feature map added in S4-1 based on the software function of Google Earth, and draw a route based on the processed feature map, and meet the accuracy and safety requirements; S5. Generate and export the route map file in kml format from Google Earth; S6. Import the kml format route map file into litchi and adjust the punctuation position as required; S7 and litchi export CSV format files and save them to the drone; S8. Use the moving window long exposure sequence superposition to complete the shooting.

2. The UAV light painting method based on neural network and artistic programming according to claim 1 is characterized in that: In step S6, if the drawn pattern does not meet the accuracy and safety requirements, the punctuation points can be directly dragged to adjust it in the litchi interface.

3. The UAV light painting method based on neural network and artistic programming according to claim 1, characterized in that: In step S7, if the height and angle need to be adjusted, then go to step S7-1, and after the adjustment is completed, go to step S8; if no adjustment is required, go directly to step S8. In step S7-1, the CSV format file exported by litchi is imported into the coordinate conversion program, and the pattern is rotated and transformed in the air to achieve adjustment of any height and angle.

Images