Aerial photography control method, device and system and computer readable storage medium

Abstract

The invention relates to the field of aerial photography, in particular to an aerial photography control method, device and system and a computer readable storage medium. The aerial photography control method comprises the following steps: receiving aerial photography task information; the aerial photography task information is input into a pre-trained aerial photography trajectory planning model, a first aerial photography trajectory sequence is obtained, and the first aerial photography trajectory sequence comprises trajectory points arranged according to a time sequence and aerial photography control parameters corresponding to the trajectory points; and controlling the flight equipment to carry out aerial photography based on the trajectory points in the first aerial photography trajectory sequence and the aerial photography control parameters to obtain an aerial photography picture. According to the invention, the operation process of aerial photography control of a user can be simplified, the operation threshold of aerial photography is reduced, and aerial photography pictures meeting user requirements can be shot.

Description

Technical Field

[0001] The present application relates to the field of aerial photography, and specifically to an aerial photography control method, device, system and computer-readable storage medium. Background Art

[0002] Aerial photography refers to the act of taking photos in the air. When a user uses a flying device to take aerial photos, the user needs to determine the object to be photographed, plan the path of the aerial device, and control the camera device in combination with the position of the flying device during the flight of the flying device so that the flying device can capture images or videos that meet the user's requirements.

[0003] However, the above operation process is relatively cumbersome and has high requirements on the user's operation, and it is often difficult for the user to capture a picture that meets his or her expectations. Summary of the invention

[0004] In view of the above, the embodiments of the present application provide an aerial photography control method, device, system and computer-readable storage medium, which aim to simplify the user's aerial photography control operation process, lower the operating threshold of aerial photography, and capture aerial photography images that meet user needs.

[0005] In a first aspect, an embodiment of the present application provides an aerial photography control method for controlling a flying device to perform aerial photography, the aerial photography control method comprising: Receive aerial photography mission information; Inputting the aerial photography task information into a pre-trained aerial photography trajectory planning model to obtain a first aerial photography trajectory sequence, wherein the first aerial photography trajectory sequence includes trajectory points arranged in time sequence and aerial photography control parameters corresponding to the trajectory points; The flying device is controlled to perform aerial photography based on the trajectory points in the first aerial photography trajectory sequence and the aerial photography control parameters to obtain an aerial photography picture.

[0006] The embodiment of the present application automatically performs aerial photography trajectory planning based on aerial photography mission information and an aerial photography trajectory planning model, and obtains corresponding aerial photography control parameters, so that the flight equipment can automatically shoot according to the aerial photography control parameters when reaching the trajectory point, thereby simplifying the user's aerial photography control operation process, lowering the aerial photography operation threshold, and facilitating the capture of aerial photography images that meet user needs.

[0007] In some embodiments, after controlling the flying device to perform aerial photography based on the trajectory points in the first aerial photography trajectory sequence and the aerial photography control parameters to obtain an aerial photography picture, the method further includes: Obtaining evaluation information of the aerial photography picture; Adjusting the first aerial photography trajectory sequence based on the evaluation information to obtain a second aerial photography trajectory sequence; The flying device is controlled to perform aerial photography based on the trajectory points in the second aerial photography trajectory sequence and the aerial photography control parameters.

[0008] In some embodiments, after receiving the aerial photography task information, the method further includes: Detecting whether the aerial photography task information exists in the historical task information, wherein the historical task information is used to indicate the aerial photography task information corresponding to the aerial photography task historically executed by the flight equipment; If the aerial photography task information does not exist in the historical task information, the step of inputting the aerial photography task information into the pre-trained aerial photography trajectory planning model to obtain a first aerial photography trajectory sequence is performed; If the aerial photography mission information exists in the historical mission information, the step of controlling the flying device to perform aerial photography based on the trajectory points in the second aerial photography trajectory sequence and the aerial photography control parameters is executed. In some embodiments, adjusting the first aerial photography trajectory sequence based on the evaluation information to obtain a second aerial photography trajectory sequence includes: When the evaluation information indicates that the aerial photography picture does not conform to the aerial photography task information, the first aerial photography trajectory sequence is adjusted based on the evaluation information to obtain a second aerial photography trajectory sequence.

[0009] In some embodiments, obtaining evaluation information of the aerial photography picture includes: The aerial photography picture and the aerial photography task information are input into a pre-trained image evaluation model to obtain the evaluation information.

[0010] In some embodiments, the training steps of the aerial photography trajectory planning model include: Acquire an aerial photography trajectory planning sample, wherein the aerial photography trajectory planning sample includes an aerial photography task sample and an aerial photography trajectory sequence label corresponding to the aerial photography task; Inputting the aerial photography task samples into an initial aerial photography trajectory planning model to obtain a predicted aerial photography trajectory sequence; The network parameters of the initial aerial photography trajectory planning model are updated based on the aerial photography trajectory sequence label and the predicted aerial photography trajectory sequence to obtain the aerial photography trajectory planning model.

[0011] In a second aspect, an embodiment of the present application further provides an aerial photography control device, which includes a processor and a memory, wherein the memory is used to store instructions, and the processor is used to call the instructions in the memory so that the aerial photography control device executes the aerial photography control method described in the first aspect.

[0012] In a third aspect, an embodiment of the present application further provides an aerial photography control system, comprising: an interactive device and the aerial photography control device described in the second aspect; The interactive device is used to receive the user's aerial photography interaction instructions, identify the aerial photography intention based on the pre-trained semantic understanding model and the aerial photography interaction instructions, obtain the user's aerial photography task information, and transmit the aerial photography task information to the aerial photography control device.

[0013] In some embodiments, the interactive device is also used to obtain the user's aerial photography evaluation instructions, perform semantic recognition on the aerial photography evaluation instructions based on the semantic understanding model, obtain evaluation information of the aerial photography image, and transmit the evaluation information to the aerial photography control device.

[0014] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the aerial photography control method as described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a structural schematic diagram of an aerial photography control system provided according to one embodiment of the present application.

[0016] Figure 2 The present invention is a flowchart of the steps of an aerial photography control method provided according to an embodiment of the present application.

[0017] Figure 3 The present invention is a flowchart of the steps of an aerial photography control method provided according to another embodiment of the present application.

[0018] Figure 4 It is a schematic diagram of the structure of an aerial photography control device provided according to one embodiment of the present application.

[0019] Figure 5 It is a structural schematic diagram of an aerial photography control device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application is described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the implementation methods of the present application and the features in the implementation methods can be combined with each other without conflict.

[0021] In the following description, many specific details are set forth to facilitate a full understanding of the present application. The described implementations are only part of the implementations of the present application, rather than all the implementations.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0023] It should be further noted that, in this article, the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0024] In this application, "at least one" means one or more, and "more" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0025] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0026] Aerial photography refers to the act of taking photos in the air. Users can use flying devices to take aerial photos. The flying devices are equipped with cameras. When taking aerial photos, users need to determine the object to be photographed, plan the path of the flying device, control the flight of the flying device, and control the camera device according to the position of the flying device during the flight to collect aerial pictures. If the aerial pictures do not meet expectations, users may need to control the flying device again to take aerial photos so that the flying device can take aerial pictures that meet the user's requirements.

[0027] In order to smoothly control the flying device for aerial photography, users need to spend a long time learning the control skills of the flying device, for example, learning to control the direction of the gimbal camera of the flying device and controlling the flying device to rise, descend, fly left, fly right, etc. Moreover, when controlling the flying device, users also need to observe the flying device on the ground, but there will be parallax when observing from the ground. Without sufficient experience in controlling the flying device, it is easy for the flying device to collide with obstacles, which will cause the aircraft to crash.

[0028] It can be seen from the above that the operation of users controlling the flying equipment for aerial photography is rather complicated and has a high operating threshold.

[0029] In view of the above, embodiments of the present application provide an aerial photography control method, device, system, and computer-readable storage medium.

[0030] refer to Figure 1 As shown, the aerial photography control system may include an interactive device and an aerial photography control device.

[0031] The interactive device may be a mobile phone, a remote control or a handle of an aircraft device, but is not limited thereto.

[0032] The interactive device can be used to receive the user's aerial photography interaction instructions. The user can issue the aerial photography interaction instructions in the form of voice, body sensation, text, etc. The interactive device can collect the voice, body sensation or text issued by the user, thereby receiving the user's aerial photography interaction instructions.

[0033] The interactive device can also be used to identify the user's aerial photography intention based on a pre-trained semantic understanding model and aerial photography interaction instructions, obtain the user's aerial photography task information, and transmit the aerial photography task information to the aerial photography control device.

[0034] The semantic understanding model can adopt Natural Language Processing (NLP) technology.

[0035] For example, the semantic understanding model can be trained based on a large language model (LLM) and an aerial photography task semantic understanding sample. The aerial photography semantic understanding sample can include an aerial photography interaction instruction sample presented in the form of voice or text, and an aerial photography task information label corresponding to the aerial photography interaction instruction sample.

[0036] The aerial photography task information can be used to describe the aerial photography images that the user expects to capture.

[0037] Aerial photography mission information may include, but is not limited to, the subject of the aerial photography, the type of photography, the shooting technique, and the shooting effect. The subject of the aerial photography refers to the object being photographed. For example, if a selfie is required, the subject of the aerial photography is the user who controls the flight device. The shooting type refers to whether the shooting image is presented in the form of a video or an image. The shooting effect and shooting technique can reflect information such as composition, scene, and camera movement. For example, a user issues an aerial photography interaction command in the form of voice, "Help me take a Hitchcock zoom portrait video." The interactive device receives the aerial photography interaction command, converts it into text information, and inputs the text information into the semantic understanding model to obtain the aerial photography task information.

[0038] In the aerial photography mission information, the subject of the shooting can be the user who controls the flight equipment; the shooting type can be video; the shooting technique can be: the shooting field of view is based on the portrait, and when the camera device approaches or moves away from the portrait, the position and size of the portrait in the camera field of view are kept unchanged through optical zoom, digital cropping zoom, etc.; shooting effect: use the environmental information captured by the lens zoom to create changes in the user's psychological activities.

[0039] After acquiring the aerial photography mission information, the interactive device may transmit the aerial photography mission information to the aerial photography control device.

[0040] The aerial photography control device can be used to execute the aerial photography control method provided in the embodiment of the present application. The aerial photography control method includes: receiving aerial photography task information; inputting the aerial photography task information into a pre-trained aerial photography trajectory planning model to obtain a first aerial photography trajectory sequence, the first aerial photography trajectory sequence includes trajectory points arranged in time sequence, and aerial photography control parameters corresponding to the trajectory points, the aerial photography control parameters include but are not limited to camera orientation, drone orientation, camera parameters, etc., then, based on the trajectory points and aerial photography control parameters in the first aerial photography trajectory sequence, the flight equipment is controlled to perform aerial photography to obtain an aerial photography picture.

[0041] The aerial photography control device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes but is not limited to processors, microprogrammed control units (MCU), application specific integrated circuits (ASIC), programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0042] The aerial photography control device can be mounted on a flying device and be communicatively connected with a motor, a camera device, etc. of the flying device to control the flying device to perform aerial photography.

[0043] The flying device may be an airplane, a helicopter, a hot air balloon, a small spacecraft, a rocket, a kite, a parachute, a drone, etc., but is not limited thereto.

[0044] The embodiment of the present application mounts the aerial photography control device on the flight device, which is beneficial for the aerial photography control device to timely obtain relevant information such as the flight angle and location of the flight device, and to timely control the flight device, thereby reducing information transmission delays and improving the real-time performance of flight control.

[0045] In some embodiments, after the flying device obtains the aerial photography image, the interactive device can also obtain the user's aerial photography evaluation instructions, and then perform semantic recognition on the aerial photography evaluation instructions based on the semantic understanding model to obtain evaluation information of the aerial photography image, and transmit the evaluation information to the aerial photography control device.

[0046] The aerial photography control device may adjust the first aerial photography trajectory sequence based on the evaluation information to obtain a second aerial photography trajectory sequence. For example, the aerial photography control device may adjust the trajectory points in the first aerial photography trajectory sequence, as well as aerial photography control parameters, such as flight speed, gimbal angle, etc., but is not limited thereto.

[0047] Then, the aerial photography control device can store the second aerial photography trajectory sequence to the cloud or locally (for example, stored in the memory of the aerial photography control device), so that when the flying device performs the same aerial photography task again, it can shoot based on the second aerial photography trajectory sequence, and then gradually optimize the aerial photography picture to make it more in line with the aerial photography task information.

[0048] For example, a user may issue an aerial photography evaluation instruction such as "The video just now moved too fast, fly slower". The aerial photography evaluation instruction may be parsed through a semantic understanding model to obtain evaluation information of reducing the flight speed of the flying device during aerial photography. Then, the aerial photography control device may receive the evaluation information and, in response to the evaluation information, reduce the flight speed of the flying device in the first aerial photography trajectory sequence to obtain an adjusted first aerial photography trajectory sequence, which is the second aerial photography trajectory sequence. In other embodiments, the user end (such as a user bracelet, etc.) may also transmit user behavior information to the interactive device. The user behavior information may include user movements, postures, expressions, blood oxygen, heartbeat, etc. The interactive device may obtain whether the user is satisfied with the aerial photography image based on the information, thereby obtaining evaluation information.

[0049] In the above embodiments, the interactive device may determine the evaluation information based on the user's aerial photography evaluation instruction and / or user behavior information. In other embodiments, the aerial photography control device may also input the aerial photography image and aerial photography task information into a pre-trained image evaluation model to obtain the evaluation information. The embodiments of the present application are not limited to this.

[0050] The embodiment of the present application adjusts the first aerial photography trajectory sequence in a targeted manner through evaluation information, which can not only timely correct the first aerial photography trajectory sequence so that the aerial photography images meet the aerial photography requirements indicated by the aerial photography task information, but also further realize aerial photography automation, reduce the difficulty of aerial photography operations, and improve aerial photography efficiency.

[0051] In addition, the embodiment of the present application can also use the optimized aerial photography trajectory sequence (ie, the second aerial photography trajectory sequence) when encountering the same aerial photography task next time, thereby ensuring the aerial photography effect for the next time.

[0052] The above is combined with the aerial photography control system to explain the process of aerial photography by the flight equipment. Figure 2 Describe the process of aerial photography control equipment controlling the flying equipment to perform aerial photography.

[0053] Figure 2 This is a flowchart of the steps of an embodiment of the aerial photography control method of the present application. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0054] See also Figure 2 As shown, the aerial photography control method may include the following steps.

[0055] Step 201, receiving aerial photography mission information.

[0056] Specifically, the aerial photography control device may receive aerial photography task information from the interactive device.

[0057] In some embodiments, the interactive device can receive the user's aerial photography interaction instructions, identify the user's aerial photography intention based on a pre-trained semantic understanding model and the aerial photography interaction instructions, obtain the user's aerial photography task information, and transmit the aerial photography task information to the aerial photography control device so that the aerial photography control device can receive the aerial photography task information.

[0058] The aerial photography task information can be used to describe the aerial photography images that the user expects to capture.

[0059] The aerial photography mission information may include: shooting subject, shooting type, shooting technique and shooting effect, but is not limited thereto.

[0060] In some embodiments, after receiving the aerial photography mission information, the aerial photography control device can detect whether the aerial photography mission information exists in the historical mission information, wherein the historical mission information is used to indicate the aerial photography mission information corresponding to the aerial photography mission historically performed by the flight device; if the aerial photography mission information does not exist in the historical mission information, it means that the flight device may be performing the corresponding aerial photography mission for the first time, and the aerial photography control device can execute the following steps 202 to 203.

[0061] Step 202: input the aerial photography task information into a pre-trained aerial photography trajectory planning model to obtain a first aerial photography trajectory sequence.

[0062] The first aerial photography trajectory sequence includes trajectory points arranged in time sequence, and aerial photography control parameters corresponding to the trajectory points.

[0063] Aerial photography control parameters may refer to control parameters of the flight action, camera orientation, camera parameters, etc. of the flying device during aerial photography.

[0064] The flying action may include the direction of the flying device. For example, the direction of the flying device may be that the body of the flying device is facing the photographed subject.

[0065] The aerial photography control device can determine the pitch angle, yaw angle and roll angle of the fuselage of the flying device at the trajectory point based on the orientation of the flying device, so as to control the fuselage of the flying device.

[0066] For example, the flying device is a drone, the shooting subject is a building, and the drone is facing the front of the building. The aerial photography control device can determine the pitch angle, yaw angle and roll angle of the drone's body at the corresponding trajectory point based on the drone's direction, so that the drone can face the front of the building.

[0067] The flight action may also include the speed of the flight device, whether the flight device is hovering, etc., but is not limited thereto.

[0068] The camera orientation may be an angle of the camera relative to the subject being photographed. For example, the aerial photography control device may obtain a target gimbal orientation based on the camera orientation, thereby controlling the camera orientation through the gimbal.

[0069] Camera parameters may include, but are not limited to: aperture, shutter, sensitivity, focal length, depth of field, exposure, etc.

[0070] In step 202, the aerial photography control device can control the flight device to use simultaneous localization and mapping (SLAM) technology to establish a point cloud map of the surrounding environment of the flight device when performing the shooting task, and mark key information such as the shooting subject in the point cloud map; the aerial photography trajectory planning model uses key information such as obstacles and shooting subjects marked by the point cloud map as constraints, and takes shooting techniques and shooting effects, such as camera movement and composition, as planning goals to obtain a first aerial photography trajectory sequence.

[0071] The embodiment of the present application establishes a point cloud map of the surrounding environment through SLAM technology. The point cloud map can reflect the terrain, obstacles, shooting subjects, etc. of the surrounding location, so that the aerial photography trajectory planning model can combine the terrain and obstacles, shooting subjects, etc. of the surrounding location to plan the aerial photography trajectory, avoid collision with obstacles during the aerial photography process, improve the safety of aerial photography, and help to make the shooting effect better.

[0072] Take the flying device as a drone as an example, and assume that the aerial photography interactive command is "Help me take a Hitchcock zoom portrait video". The first aerial photography trajectory sequence is used to control the drone to hover about 10 meters in front of the user, and control the gimbal in the drone to face the user, ensuring that the size and position of the portrait in the field of view meet the aerial photography task information.

[0073] That is, the first aerial photography trajectory sequence can instruct the drone to start filming when it reaches the target trajectory point, and gradually fly slowly towards the user according to each trajectory point. During this process, the focal length of the camera is controlled to gradually shorten, and the aerial photography control device can control the camera direction and the drone direction through the shooting control parameters of each trajectory point, so that the size and position of the user's portrait in each image frame taken by the camera conform to the aerial photography task information, until the shooting task is completed, and the recording is turned off when the target trajectory point is reached.

[0074] After turning off the recording, the drone can transmit the captured aerial images to designated electronic devices, such as mobile phones, remote controls or handles, etc. The drone can also hover at a certain location and wait for the user's next instruction.

[0075] In some embodiments, the training steps of the above-mentioned aerial photography trajectory planning model may include: First, the training device obtains an aerial photography trajectory planning sample, where the aerial photography trajectory planning sample includes an aerial photography task sample and an aerial photography trajectory sequence label corresponding to the aerial photography task sample.

[0076] For example, the training device may obtain aerial photography trajectory planning samples from a data set, which may include various types of aerial photography trajectory samples.

[0077] Specifically, the data set may include multiple video data.

[0078] The multiple video data may correspond to different types of shooting task samples. For example, the multiple video data may correspond to different shooting subjects, compositions, and / or camera movement models (such as tracking, translation, rotation, etc.).

[0079] Each frame of the video data can also be labeled with the location of the flight device when the frame was taken, as well as the shooting control parameters at that location, such as the direction of the flight device, the direction of the gimbal, etc., to form an aerial photography trajectory sequence label.

[0080] Next, the training device can input the aerial photography task samples into the aerial photography trajectory planning model to obtain a predicted aerial photography trajectory sequence, and then update the network parameters of the aerial photography trajectory planning model based on the aerial photography sequence labels and the predicted aerial photography trajectory sequence until the training accuracy requirements are met.

[0081] For example, reinforcement learning models such as deep Q-network or policy gradient algorithm can be used to train aerial photography trajectory planning models.

[0082] It is understandable that the above-mentioned training device and aerial photography control device can be the same device or different devices.

[0083] After the aerial photography trajectory planning model is trained, the trained aerial photography trajectory planning model can be deployed on the aerial photography control device using embedded artificial intelligence technology, so that the aerial photography control can obtain the first aerial photography trajectory sequence.

[0084] Step 203: Control the flying device to perform aerial photography based on the first aerial photography trajectory sequence to obtain aerial photography images.

[0085] Specifically, the aerial photography control device controls the flight device to perform aerial photography based on the trajectory points in the first aerial photography trajectory sequence and the aerial photography control parameters to obtain aerial photography images.

[0086] That is, the aerial photography control device controls the flying device to reach each trajectory point in sequence according to the timing of the trajectory points in the first aerial photography trajectory sequence, and controls the flying device to perform aerial photography according to the aerial photography control parameters each time it reaches a trajectory point, thereby obtaining an aerial photography image. For example, the flying device is controlled to perform aerial photography according to the drone attitude, gimbal attitude, and camera parameters in the aerial photography control parameters.

[0087] For example, suppose that there is a trajectory point in the first aerial photography trajectory sequence that is 10 meters in front of the user, and the shooting control parameters include hovering at the trajectory point and controlling the gimbal direction toward the user. The aerial photography control device can obtain the user's position, for example, by obtaining the user's GPS location through the remote control, handle, mobile phone, etc. held by the user, and obtaining the geomagnetic data of the handle or remote control, thereby obtaining the user's position and direction, setting the 10 meters in front of the handle or remote control as the drone's hovering position, and controlling the drone to fly to the hovering position. After the drone flies to the hovering position, it rotates on the spot to make the gimbal direction face the handle or remote control position.

[0088] In some embodiments, the aerial photography control device can also obtain evaluation information of the aerial photography image; adjust the first aerial photography trajectory sequence based on the evaluation information to obtain a second aerial photography trajectory sequence; and control the flight device to perform aerial photography based on the trajectory points in the second aerial photography trajectory sequence and the aerial photography control parameters.

[0089] Furthermore, when the evaluation information indicates that the aerial photography picture meets the aerial photography task information, the aerial photography control device may not optimize the first aerial photography trajectory sequence.

[0090] That is, in this case, if the aerial photography control device receives the same aerial photography task information again, it controls the flight device to perform aerial photography based on the first aerial photography trajectory sequence.

[0091] The aerial photography control device may also adjust the first aerial photography trajectory sequence based on the evaluation information to obtain a second aerial photography trajectory sequence when the evaluation information indicates that the aerial photography picture does not conform to the aerial photography task information.

[0092] For example, after acquiring the second aerial photography trajectory sequence, the aerial photography control device can automatically control the flight device to re-perform aerial photography based on the second aerial photography trajectory sequence to obtain the aerial photography picture again; if the evaluation information of the aerial photography picture obtained again indicates that the aerial photography picture does not meet the aerial photography task information, the second aerial photography trajectory sequence is continued to be adjusted based on the evaluation information to obtain the third aerial photography trajectory sequence, and then the flight device is automatically controlled to re-perform aerial photography based on the third aerial photography trajectory sequence until the aerial photography picture meets the aerial photography task information.

[0093] The embodiment of the present application obtains evaluation information of the aerial photography image, so that the aerial photography control device can know the advantages and disadvantages of the current shooting image, and adjust the first aerial photography trajectory sequence in a targeted manner based on the evaluation information, thereby reducing the number of adjustments to the aerial photography trajectory sequence, and quickly locating the aerial photography trajectory sequence that can capture the desired image. In addition, the embodiment of the present application uses an automatic adjustment method to adjust the first aerial photography trajectory sequence, which can further lower the operating threshold.

[0094] For another example, after acquiring the second aerial photography trajectory sequence, the aerial photography control device may store the second aerial photography trajectory sequence. If the same aerial photography task information is received again, the flight device may be controlled to perform aerial photography based on the trajectory points in the second aerial photography trajectory sequence and the aerial photography control parameters.

[0095] In an embodiment of the present application, the aerial photography control device can record the second aerial photography trajectory sequence of the aerial photography task, so that when the same aerial photography task is received subsequently, the second aerial photography trajectory sequence can be quickly located, thereby ensuring the quality of aerial photography and improving the efficiency of aerial photography.

[0096] For example, refer to Figure 3 As shown, after step 203, the following steps may also be included: Step 204, determining whether the aerial photography image meets the aerial photography mission information.

[0097] In some embodiments, the aerial photography control device may obtain evaluation information, and the evaluation information may indicate whether the aerial photography image complies with the aerial photography task information.

[0098] In some embodiments, the aerial photography control device may obtain evaluation information from the interactive device. Specifically, the interactive device obtains the user's aerial photography evaluation instruction, performs semantic recognition on the aerial photography evaluation instruction based on the semantic understanding model, obtains evaluation information of the aerial photography image, and transmits the evaluation information to the aerial photography control device, so that the aerial photography control device can obtain the evaluation information.

[0099] The embodiment of the present application can adjust the first aerial photography trajectory sequence in combination with the user's personalized evaluation of the aerial photography images, so that the aerial photography images can better meet the user's specific needs and improve the user experience.

[0100] In other embodiments, the aerial photography control device may input the aerial photography image and the aerial photography task information into a pre-trained image evaluation model to obtain the evaluation information.

[0101] Specifically, when the type of aerial photography is video, the image evaluation model can analyze each frame of the aerial photography to obtain evaluation information, which may include a score for each image frame. The more the image frame conforms to the aerial photography task information, the higher the score.

[0102] The evaluation information may also include suggestions for adjusting the first aerial photography trajectory planning sequence. For example, the evaluation information may include: reducing the flight speed between a certain trajectory point and another trajectory point, adjusting the camera orientation of a certain trajectory point, and the like.

[0103] Compared with the subjective differences in manual evaluation, the embodiment of the present application adopts an image evaluation model to obtain evaluation information, which can provide unified and objective feedback to ensure the consistency of the evaluation results. Moreover, the image evaluation model can evaluate quickly and has strong real-time feedback capabilities, thereby ensuring the optimization efficiency of aerial photography images.

[0104] Furthermore, the image evaluation model may include a convolutional neural network and an image quality assessment module.

[0105] The image evaluation model can analyze the camera movement, composition, light and shadow, and color in the input aerial photos based on the convolutional neural network to extract features such as composition (such as the golden ratio, rule of thirds, etc.), light and shadow (brightness, contrast, etc.), and color (color balance, saturation, etc.).

[0106] Then, the image evaluation model processes these features based on image quality assessment modules, such as BRISQUE, NIQE, etc., to quantify the image quality.

[0107] The training steps of the image evaluation model may include: obtaining a large-scale dataset, such as the Atomic Visual Actions (AVA) dataset, which may include images and videos, performing preprocessing such as resizing the images and videos in the dataset to obtain a preprocessed dataset, and then using the preprocessed dataset to train a deep learning model for aesthetic evaluation and use methods such as transfer learning to improve performance by fine-tuning the image evaluation model. After obtaining the trained image evaluation model through the above steps, the trained image evaluation model can be deployed on the flight equipment.

[0108] The aerial photography control device collects images and videos in real time through the camera of the flight device, uses the image evaluation model to evaluate them, and outputs evaluation information.

[0109] In other embodiments, the aerial photography control device may obtain evaluation information from the interactive device and from the image evaluation model, and integrate the evaluation information of the two to obtain the evaluation information of the aerial photography image.

[0110] For example, if the evaluation information obtained from the interactive device indicates that the flight speed of the flying device needs to be reduced, and the evaluation information obtained based on the image evaluation model indicates that the direction of the flying device at a certain trajectory point needs to be adjusted, it means that the aerial photography image does not conform to the aerial photography mission information, and the evaluation information of the aerial photography image includes reducing the flight speed of the flying device and adjusting the direction of the flying device at a certain trajectory point.

[0111] The evaluation information obtained from the interactive device often reflects the user's views based on subjective factors such as his or her own aesthetics and specific needs. The evaluation information obtained through the image evaluation model is determined based on established objective standards (such as picture clarity, color accuracy, composition, etc.). The embodiment of the present application can make the final aerial photography picture evaluation information include both subjective feeling level content and objective professional measurement indicators, thereby realizing the organic integration of subjective and objective evaluation. The two can complement and verify each other, making the evaluation results more comprehensive, thereby ensuring the quality of aerial photography pictures.

[0112] In some embodiments, the aerial photography control device may create a storage space, which may be used to store the aerial photography task information and the aerial photography trajectory sequence corresponding to the aerial photography task information.

[0113] When the aerial photography image conforms to the aerial photography mission information, the aerial photography control device can store the aerial photography mission information and the first aerial photography trajectory sequence in a storage space, so that the next time the aerial photography control device receives the same aerial photography mission information, it can find the aerial photography trajectory sequence (i.e., the first aerial photography trajectory sequence) corresponding to the aerial photography mission information in the storage space to perform aerial photography.

[0114] That is, if the aerial photography image meets the aerial photography task information, the aerial photography control device may execute step 207 .

[0115] If the aerial photography image does not conform to the aerial photography task information, the aerial photography control device may execute steps 205 and 206 .

[0116] Step 205: adjust the first aerial photography trajectory sequence based on the evaluation information to obtain a second aerial photography trajectory sequence.

[0117] For example, the evaluation information includes reducing the flight speed, and the flight speed of each track point in the first aerial photography track sequence may be reduced by a preset amplitude. Alternatively, the flight speed of a certain flight segment may be reduced, and the flight segment may include a plurality of track points.

[0118] In some embodiments, when the aerial photography picture does not conform to the aerial photography task information, the aerial photography control device adjusts the first aerial photography trajectory sequence to obtain a second aerial photography trajectory sequence, and then uses the second aerial photography trajectory sequence as the aerial photography trajectory sequence corresponding to the aerial photography task information, and stores it in the above-mentioned storage space, so that when the aerial photography control device receives the same task information again, the aerial photography trajectory sequence corresponding to the aerial photography task information (i.e., the second aerial photography trajectory sequence) can be found in the storage space to perform aerial photography. That is, the aerial photography control device can execute step 206.

[0119] Step 206: If the aerial photography mission information is received again, the flying device is controlled to perform aerial photography based on the second aerial photography trajectory sequence.

[0120] Specifically, if the aerial photography control device receives the aerial photography task information again, it controls the flight device to perform aerial photography based on the trajectory points in the second aerial photography trajectory sequence and the aerial photography control parameters.

[0121] That is, when the aerial photography image does not conform to the aerial photography task information, the aerial photography control device may execute the above steps 205 to 206.

[0122] When the aerial photography image meets the aerial photography task information, the aerial photography control device may execute the following step 207 .

[0123] Step 207: If the aerial photography mission information is received again, the flying device is controlled to perform aerial photography based on the first aerial photography trajectory sequence.

[0124] Specifically, if the aerial photography control device receives the aerial photography task information again, it controls the flight device to perform aerial photography based on the trajectory points in the first aerial photography trajectory sequence and the aerial photography control parameters.

[0125] The embodiment of the present application can automatically perform trajectory planning based on aerial photography mission information and an aerial photography trajectory planning model, and obtain corresponding aerial photography control parameters, so that the flight equipment can shoot according to the aerial photography control parameters when reaching the trajectory point, thereby simplifying the aerial photography operation process, lowering the aerial photography operation threshold, and facilitating the capture of aerial photography images that meet user needs.

[0126] Furthermore, the embodiment of the present application can automatically adjust the aerial photography trajectory sequence based on the evaluation information, which is conducive to gradually optimizing the aerial photography image to make it more consistent with the aerial photography task information.

[0127] The embodiment of the present application can automatically evaluate the aerial photography images based on the image evaluation model, which is conducive to promoting the automatic optimization of the aerial photography trajectory sequence.

[0128] Moreover, the embodiment of the present application uses a semantic understanding model to identify the user's aerial photography interaction commands, so that the flight equipment can obtain the user's core needs for aerial photography, improve the human-computer interaction performance, and ensure the accuracy of aerial photography tasks. On the other hand, configuring the semantic understanding model on the interactive device is conducive to ensuring the timeliness of user intention recognition and achieving lightweight flight equipment.

[0129] Based on the same idea as the aerial photography control method in the above embodiment, the present application also provides an aerial photography control device, which can be used to execute the above aerial photography control method. For ease of explanation, the structural diagram of the aerial photography control device embodiment only shows the parts related to the embodiment of the present application. It can be understood by those skilled in the art that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown in the diagram, or combine certain components, or arrange the components differently.

[0130] like Figure 4 As shown, the aerial photography control device may include an information receiving module 401, a trajectory planning module 402, and a flight control module 403. In some embodiments, the above modules may be programmable software instructions stored in a memory and can be called and executed by a processor. It is understood that in other embodiments, the above modules may also be program instructions or firmware solidified in the processor.

[0131] The information receiving module 401 is used to receive the aerial photography task information; The trajectory planning module 402 is used to input the aerial photography task information into a pre-trained aerial photography trajectory planning model to obtain a first aerial photography trajectory sequence, wherein the first aerial photography trajectory sequence includes trajectory points arranged in time sequence and aerial photography control parameters corresponding to the trajectory points; The flight control module 403 is used to control the flight device to perform aerial photography based on the trajectory points in the first aerial photography trajectory sequence and the aerial photography control parameters to obtain aerial photography images. Figure 5 This is a schematic diagram of an embodiment of the aerial photography control device of the present application.

[0132] The aerial photography control device 100 includes a memory 20, a processor 30, and a computer program 40 stored in the memory 20 and executable on the processor 30. When the processor 30 executes the computer program 40, the steps in the above-mentioned aerial photography control method embodiment are implemented, for example Figure 2 Steps 201 to 203 shown in FIG. 10 , or executing the steps Figure 3 Steps 201 to 207 are shown.

[0133] Exemplarily, the computer program 40 can also be divided into one or more modules / units, which are stored in the memory 20 and executed by the processor 30. The one or more modules / units can be a series of computer program instruction segments that can perform specific functions, and the instruction segments are used to describe the execution process of the computer program 40 in the aerial photography control device 100.

[0134] Those skilled in the art will understand that the schematic diagram is merely an example of the aerial photography control device 100 and does not constitute a limitation of the aerial photography control device 100. The aerial photography control device 100 may include more or fewer components than shown in the diagram, or a combination of certain components, or different components. For example, the aerial photography control device 100 may also include input and output devices, network access devices, buses, etc.

[0135] The processor 30 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, a single-chip microcomputer, or the processor 30 may also be any conventional processor, etc.

[0136] The memory 20 can be used to store the computer program 40 and / or the module / unit. The processor 30 realizes various functions of the aerial photography control device 100 by running or executing the computer program and / or the module / unit stored in the memory 20 and calling the data stored in the memory 20. The memory 20 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data (such as audio data) created according to the use of the aerial photography control device 100, etc. In addition, the memory 20 can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

[0137] If the module / unit integrated in the aerial photography control device 100 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electrical carrier signals and telecommunication signals.

[0138] The present application also provides an aerial photography control device, referring to Figure 1 As shown, the aerial photography control device may include an interactive device and the aerial photography control device in the above-mentioned embodiment.

[0139] The interactive device is used to receive the user's aerial photography interaction instructions, identify the user's aerial photography intention based on the pre-trained semantic understanding model and the aerial photography interaction instructions, obtain aerial photography task information, and transmit the aerial photography task information to the aerial photography control device.

[0140] In some embodiments, the interactive device is also used to obtain the user's aerial photography evaluation instructions, perform semantic recognition on the aerial photography evaluation instructions based on the pre-trained semantic understanding model, obtain evaluation information of the aerial photography image, and transmit the evaluation information to the aerial photography control device.

[0141] In the several embodiments provided in this application, it should be understood that the disclosed aerial photography control device and method can be implemented in other ways. For example, the aerial photography control device embodiment described above is only illustrative, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation.

[0142] In addition, each functional unit in each embodiment of the present application may be integrated into the same processing unit, each unit may exist physically separately, or two or more units may be integrated into the same unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0143] For those skilled in the art, it is obvious that the present application is not limited to the details of the above-mentioned exemplary embodiments, and the present application can be implemented in other specific forms without departing from the spirit or basic characteristics of the present application. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units or aerial control devices stated in the claims of the aerial control device can also be implemented by the same unit or aerial control device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any specific order.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the above embodiments, a person of ordinary skill in the art should understand that the technical solution of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present application.

Claims

1. An aerial photography control method for controlling a flying device to perform aerial photography, characterized in that: The aerial photography control method comprises: Receive aerial photography mission information; Inputting the aerial photography task information into a pre-trained aerial photography trajectory planning model to obtain a first aerial photography trajectory sequence, wherein the first aerial photography trajectory sequence includes trajectory points arranged in time sequence and aerial photography control parameters corresponding to the trajectory points; The flying device is controlled to perform aerial photography based on the trajectory points in the first aerial photography trajectory sequence and the aerial photography control parameters to obtain an aerial photography picture.

2. The aerial photography control method according to claim 1, characterized in that: After controlling the flying device to perform aerial photography based on the trajectory points in the first aerial photography trajectory sequence and the aerial photography control parameters to obtain an aerial photography picture, the method further includes: Obtaining evaluation information of the aerial photography picture; Adjusting the first aerial photography trajectory sequence based on the evaluation information to obtain a second aerial photography trajectory sequence; The flying device is controlled to perform aerial photography based on the trajectory points in the second aerial photography trajectory sequence and the aerial photography control parameters.

3. The aerial photography control method according to claim 2, characterized in that: After receiving the aerial photography task information, the method further includes: Detecting whether the aerial photography task information exists in the historical task information, wherein the historical task information is used to indicate the aerial photography task information corresponding to the aerial photography tasks that the flight equipment has historically performed; If the aerial photography task information does not exist in the historical task information, the step of inputting the aerial photography task information into the pre-trained aerial photography trajectory planning model to obtain a first aerial photography trajectory sequence is performed; If the aerial photography mission information exists in the historical mission information, the step of controlling the flying device to perform aerial photography based on the trajectory points in the second aerial photography trajectory sequence and the aerial photography control parameters is executed.

4. The aerial photography control method according to claim 2, characterized in that: The step of adjusting the first aerial photography trajectory sequence based on the evaluation information to obtain a second aerial photography trajectory sequence includes: When the evaluation information indicates that the aerial photography picture does not conform to the aerial photography task information, the first aerial photography trajectory sequence is adjusted based on the evaluation information to obtain the second aerial photography trajectory sequence.

5. The aerial photography control method according to claim 2, characterized in that: The obtaining of evaluation information of the aerial photography picture includes: The aerial photography picture and the aerial photography task information are input into a pre-trained image evaluation model to obtain the evaluation information.

6. The aerial photography control method according to any one of claims 1 to 5, characterized in that: The training steps of the aerial photography trajectory planning model include: Acquire an aerial photography trajectory planning sample, wherein the aerial photography trajectory planning sample includes an aerial photography task sample and an aerial photography trajectory sequence label corresponding to the aerial photography task sample; Inputting the aerial photography task samples into an initial aerial photography trajectory planning model to obtain a predicted aerial photography trajectory sequence; The network parameters of the initial aerial photography trajectory planning model are updated based on the aerial photography trajectory sequence label and the predicted aerial photography trajectory sequence to obtain the aerial photography trajectory planning model.

7. An aerial photography control device, comprising a processor and a memory, characterized in that: The memory is used to store instructions, and the processor is used to call the instructions in the memory, so that the aerial photography control device executes the aerial photography control method according to any one of claims 1 to claim 6.

8. An aerial photography control system, characterized in that: include: An interactive device and an aerial photography control device as claimed in claim 7; The interactive device is used to receive the user's aerial photography interaction instructions, identify the user's aerial photography intention based on the pre-trained semantic understanding model and the aerial photography interaction instructions, obtain aerial photography task information, and transmit the aerial photography task information to the aerial photography control device.

9. The aerial photography control system according to claim 8, characterized in that: The interactive device is also used to obtain the user's aerial photography evaluation instructions, perform semantic recognition on the aerial photography evaluation instructions based on the pre-trained semantic understanding model, obtain evaluation information of the aerial photography picture, and transmit the evaluation information to the aerial photography control device.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the aerial photography control method according to any one of claims 1 to 6.

Images