Shooting control method, device and system and computer readable storage medium

Abstract

The invention relates to the technical field of shooting, in particular to a shooting control method, device and system and a computer readable storage medium. The shooting control method can comprise the following steps: acquiring surrounding environment information of first flight equipment and task information used for indicating a shooting task; planning a flight path based on the task information and the surrounding environment information; based on the surrounding environment information and a pre-trained shooting model, a shooting strategy is determined, and the shooting model is obtained through training based on a first training sample set and is updated based on a second training sample set; the second training sample set comprises historical flight shooting data of the first flight equipment and historical flight shooting data of second flight equipment associated with the first flight equipment; and executing the shooting task based on the flight path and the shooting strategy. The flight equipment provided by the invention can provide personalized shooting service.

Description

Technical Field

[0001] The present application relates to the field of photographing technology, and in particular to a photographing control method, device, system and computer-readable storage medium. Background Art

[0002] Aerial photography refers to the act of taking photos in the air using the camera configured on the aircraft. When using the aircraft to take photos in the air, the user needs to independently control the flight trajectory of the aircraft to take photos. The aircraft cannot learn to take photos based on the user's shooting habits and cannot meet the user's personalized shooting needs. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a shooting control method, device, system and computer-readable storage medium, which can solve the problem that existing flight equipment cannot autonomously learn shooting according to the user's shooting habits and cannot meet the user's personalized shooting needs.

[0004] In a first aspect, an embodiment of the present application provides a shooting control method, which is applied to a first flying device, and the shooting control method includes: obtaining surrounding environment information of the first flying device, and task information used to indicate a shooting task; based on the task information and the surrounding environment information, planning a flight trajectory corresponding to the shooting task; based on the surrounding environment information and a pre-trained shooting model, determining a shooting strategy corresponding to the shooting task, wherein the shooting model is trained based on a first training sample set, and the shooting model is also updated based on a second training sample set, and the second training sample set includes historical flight shooting data of the first flying device and historical flight shooting data of a second flying device associated with the first flying device; executing the shooting task based on the flight trajectory and the shooting strategy.

[0005] By adopting the above technical scheme, the flight trajectory is planned and the shooting strategy is formulated based on the user's shooting instructions, which simplifies the operation process of using the flight equipment for aerial photography, lowers the operation threshold of aerial photography, and uses artificial intelligence technology to formulate shooting strategies to achieve intelligent photography. By using the flight equipment's own historical flight shooting data and the historical flight shooting data of the flight equipment associated with it, the shooting model is optimized, which can enhance the self-learning and optimization capabilities of the flight equipment, so that the flight equipment can self-optimize the shooting model according to the historical flight shooting data that represents the user's shooting habits, and provide more personalized shooting services.

[0006] In some embodiments, the second flying device associated with the first flying device includes: a user of the second flying device and a user of the first flying device belong to the same user group portrait.

[0007] By adopting the above technical solution, since the same user group portrait represents that users in the group have similar shooting preferences, by optimizing the shooting model using historical flight shooting data belonging to the same user group portrait, not only can the types and quantities of training samples used to update the shooting model be enriched, but also the self-learning and optimization capabilities of the shooting model can be further enhanced.

[0008] In some embodiments, the step of updating the shooting model includes: updating the shooting model through federated learning based on historical flight shooting data of the first flight device and historical flight shooting data of the second flight device associated with the first flight device, the historical flight shooting data including historical camera movement data, historical flight trajectory data, historical shooting results, the historical shooting results including camera shooting results and / or shooting results after processing the camera shooting results.

[0009] By adopting the above technical solution and updating the shooting model through federated learning, the privacy and security of historical flight shooting data during transmission and model training can be ensured, and user privacy leakage can be avoided. While improving the intelligent shooting capabilities of flight equipment, user privacy can be protected and the personalized shooting needs of users can be met. In addition, by learning the camera movement data under the historical flight trajectory data and the processing method of historical shooting results during the training stage, the flight equipment deployed with the shooting model can autonomously perform intelligent camera movement according to the current flight trajectory data, and intelligently process the shooting results, such as intelligently beautifying, editing, rendering, etc. the shooting results.

[0010] In some embodiments, the surrounding environment information includes image information and point cloud information of the surrounding environment. Based on the task information and the surrounding environment information, a flight trajectory corresponding to the shooting task is planned, including: based on the image information and point cloud information of the surrounding environment, device positioning and three-dimensional map construction are performed through real-time positioning and map construction algorithms; based on the location information, task information and three-dimensional map of the first flying device, a flight trajectory corresponding to the shooting task is planned.

[0011] By adopting the above technical solution, a three-dimensional map is constructed and the position of the flight equipment is located through the simultaneous localization and mapping (SLAM) algorithm, so that the path planning and navigation of the flight equipment can be realized even in signal blind spots, which can enhance the navigation and obstacle avoidance capabilities of the flight equipment in complex environments and enable the flight equipment to work stably in complex environments.

[0012] In some embodiments, a shooting result corresponding to the shooting task and a shooting score of the shooting result are output, and the shooting score is obtained based on the shooting model score.

[0013] By adopting the above technical solution, by outputting the shooting results and the shooting scores, it is convenient for users to view the shooting results and the shooting scores. Users can also evaluate whether to reshoot based on the shooting scores of the shooting results, or share images / videos with high scores to other flight equipment users.

[0014] In some embodiments, the training step of the shooting model includes: obtaining a first training sample set, the first training sample set includes multiple shooting samples, and a shooting label corresponding to each of the multiple shooting samples, the multiple shooting samples include multiple shooting environments, and the shooting labels include sample shooting scores and sample shooting strategies; based on the first training sample set, a preset neural network is trained to obtain a shooting model.

[0015] By adopting the above technical solution, the shooting model is trained using shooting samples including sample shooting scores and labels of sample shooting strategies and deployed on the flight equipment. The flight equipment can have the ability to formulate shooting strategies and score shooting results.

[0016] In some embodiments, a shooting strategy corresponding to the shooting task is determined based on surrounding environment information and a pre-trained shooting model, including: when the shooting subject is a person, generating a reference shooting posture based on task information and information of the shooting subject, so that the shooting subject can adjust the shooting posture based on the reference shooting posture; obtaining the shooting posture of the shooting subject, and determining the shooting strategy based on the surrounding environment information, the shooting posture and the shooting model.

[0017] By adopting the above technical solution, when the subject is a person, by generating a reference shooting posture, it is possible to better guide the subject to adjust the shooting posture and improve the shooting effect. The shooting strategy is determined by comprehensively considering the surrounding environment information, shooting posture and shooting model to meet the needs of shooting strategies under different shooting postures, facilitate displaying the posture details of the subject, and further improve the shooting effect.

[0018] In a second aspect, an embodiment of the present application further provides a shooting control device, which includes a processor and a memory, the memory is used to store instructions, and the processor is used to call the instructions in the memory, so that the shooting control device executes the shooting control method described in the first aspect.

[0019] In the third aspect, an embodiment of the present application also provides a shooting control system, comprising: an interactive device and the shooting control device described in the second aspect; the interactive device is used to receive the user's shooting instructions, determine the task information of the shooting task corresponding to the shooting instructions based on a pre-trained semantic understanding model, and transmit the task information to the shooting control device.

[0020] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions. When the computer instructions are executed on a flying device, the flying device executes the shooting control method as described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 The figure is a schematic diagram of the structure of a shooting control system provided according to an embodiment of the present application.

[0022] Figure 2 The present invention is a schematic diagram of functional modules of a flight device provided according to an embodiment of the present application.

[0023] Figure 3 Schematic diagram of functional modules of a shooting control device provided according to an embodiment of the present application.

[0024] Figure 4 The present invention is a flowchart of a shooting control method according to an embodiment of the present application.

[0025] Figure 5 The present invention is a flowchart of a shooting control method according to another embodiment of the present application.

[0026] Figure 6 Schematic diagram of the structure of a shooting control device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application is described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the implementation methods of the present application and the features in the implementation methods can be combined with each other without conflict.

[0028] In the following description, many specific details are set forth to facilitate a full understanding of the present application. The described implementations are only part of the implementations of the present application, rather than all the implementations.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0030] It should be further noted that, in this article, the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0031] In this application, "at least one" means one or more, and "more" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0032] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0033] In the related art, when using a flying device to take aerial photos, the user needs to autonomously control the flying device to take photos. The flying device cannot autonomously learn to take photos according to the user's shooting habits, and cannot meet the user's personalized shooting needs.

[0034] In view of this, the embodiments of the present application provide a shooting control method, device, system and computer-readable storage medium, which can solve the problem that the flight equipment cannot autonomously learn shooting according to the user's shooting habits and cannot meet the user's personalized shooting needs.

[0035] refer to Figure 1 As shown, the shooting control system 10 may include an interactive device 11 and a flying device 12 that can fly in the air. The interactive device 11 and the flying device 12 may communicate wirelessly (including but not limited to wireless fidelity (Wi-Fi), Bluetooth, dedicated radio waves, etc.).

[0036] In some embodiments, the interactive device 11 includes but is not limited to a portable terminal device (e.g., a mobile phone, a tablet computer, etc.), a vehicle device, a remote controller of the flying device 12, etc. The flying device 12 includes but is not limited to a drone, a flying camera, and other devices capable of aerial photography. Figure 1 The interactive device 11 is a mobile phone, and the flying device 12 is a drone.

[0037] The interactive device 11 may be used to receive a user's shooting instruction. For example, the shooting instruction may be an instruction issued by the user in the form of voice input, text input, etc. The interactive device may also identify the shooting intention corresponding to the shooting instruction based on the pre-trained semantic understanding model, that is, identify the task information of the shooting task initiated by the user, and transmit the task information to the shooting control device.

[0038] In some embodiments, the semantic understanding model may adopt relevant natural language processing technology (NLP). For example, the semantic understanding model may be trained based on a large language model (LLM) and semantic understanding samples related to the aerial photography task. The semantic understanding sample may include an interactive instruction sample presented in the form of voice or text, and a label of the aerial photography task corresponding to the interactive instruction sample.

[0039] In some embodiments, task information may be used to describe the shooting requirements of the picture that the user expects to capture. For example, task information may include: shooting subject, shooting type, expected shooting technique, expected shooting effect, etc., but is not limited thereto. The shooting subject may refer to the object being photographed. For example, if a selfie is required, the shooting subject is the user who controls the flight device. The shooting type may refer to whether the shooting result is presented in the form of a video or an image. The expected shooting effect and shooting technique may reflect information such as the user's expected composition, camera movement, and camera shooting mode used. For example, the user issues a shooting command in the form of voice: "Help me shoot a Hitchcock zoom portrait video". After the interactive device 11 collects the voice shooting command, it converts it into text information and inputs the text information into the semantic understanding model to obtain the task information of the shooting task. The task information can indicate: the shooting subject is the user who controls the flying device 12, the shooting type is video, and the desired shooting technique is: with the portrait as the subject, when the camera of the flying device 12 approaches or moves away from the portrait, the position and size of the portrait in the camera field of view are kept unchanged through optical zoom, digital cropping zoom, etc.; the desired shooting effect: using the environmental information captured by the lens zoom to create changes in the user's psychological activities.

[0040] like Figure 2As shown, the flying device 12 may include a camera module 121, a shooting control device 122, and a power device 123. The camera module 121 is used to realize the shooting of images or videos. For example, the camera module 121 may include a camera, a gimbal, etc. The power device 123 is used to provide power for the flying device 12. For example, the power device 123 may include a battery, a motor, a rotor, etc. The shooting control device 122 is used to control the shooting of the camera module 121 and the operation of the power device 123, so that the flying device 12 can perform the shooting task according to the planned flight trajectory.

[0041] In some embodiments, after acquiring the task information of the shooting task, the interactive device 11 may transmit the task information to the shooting control device 122. The shooting control device 122 may be used to execute the shooting control method provided in the following embodiments of the present application. The shooting control device 122 may be a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a processor, a microprogrammed control unit (MCU), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), etc.

[0042] In the embodiment of the present application, the shooting control device 122 is mounted on the flight equipment 12, which is beneficial for the shooting control device 122 to obtain relevant information such as the flight attitude and location of the flight equipment 12 in a timely manner, reduce information transmission delay, and improve the real-time performance of flight control.

[0043] like Figure 3 As shown, in some embodiments, the shooting control device 122 may include a positioning and navigation module 1221 , an intelligent shooting module 1222 , a federated learning module 1223 , a flight control module 1224 , and a system monitoring module 1225 .

[0044] For example, the flying device 12 may be equipped with a camera and a laser radar for positioning and navigation. After acquiring the image information and point cloud information of the surrounding environment of the flying device 12 through the camera and the laser radar, the positioning and navigation module 1221 may be used to process the image information and the point cloud information. For example, a three-dimensional map that can be updated in real time may be generated based on the image information and the point cloud information through a simultaneous localization and mapping (SLAM) algorithm, and the location information of the flying device 12 in the surrounding environment may be determined. The three-dimensional map may be used for flight trajectory planning and environmental perception (e.g., perception of obstacles) of the flying device. For example, the positioning and navigation module 1221 may plan the flight trajectory based on the location information of the flying device 12 and the three-dimensional map to avoid obstacles and ensure the flight safety of the flying device.

[0045] In some embodiments, when planning the flight trajectory, the positioning and navigation module 1221 also needs to consider the task information of the shooting task, so that the flying device 12 can perform the shooting task according to the planned flight trajectory.

[0046] The intelligent shooting module 1222 can formulate shooting strategies based on deep learning algorithms. For example, the intelligent shooting module 1222 can formulate shooting strategies based on the surrounding environment information and the shooting subject by calling a pre-trained shooting model. The shooting model can be obtained based on a large number of images and video training, and the shooting model can learn the composition strategy, mirror movement strategy, camera parameter setting strategy, etc. for shooting these images and videos. For the camera parameters determined based on the shooting model or the camera parameters set by the user, the intelligent shooting module 1222 can also dynamically adjust the camera parameters based on the light intensity of the current environment to optimize the shooting effect.

[0047] In some embodiments, the intelligent shooting module 1222 can also perform aesthetic scoring on the captured images / videos. For example, during the training process of the shooting model, the images and videos used for training also have scoring tags. The shooting model can learn the scoring rules of these images and videos, and then subsequently score the captured images / videos, so that the user can evaluate whether to reshoot based on the scoring of the captured images / videos, or share images / videos with higher scores with other flight equipment users for shooting exchanges.

[0048] The federated learning module 1223 can be used to collect historical flight shooting data of the flight equipment, and use the historical flight shooting data for updating and optimizing the shooting model. Updating the shooting model by means of federated learning can ensure the privacy and security of the historical flight shooting data during transmission and model training, avoid user privacy leakage, and increase the diversity of training samples used for model updating, thereby improving the intelligent shooting capabilities of the flight equipment and meeting the personalized shooting needs of users.

[0049] For example, historical flight shooting data may include historical camera movement data, historical flight trajectory data, and historical shooting results.

[0050] The flight control module 1224 may be used to be responsible for the flight control of the flight device 12. For example, the flight control module 1224 may control the flight attitude and flight path of the flight device 12 based on the location information, three-dimensional map, and planned flight trajectory provided by the positioning and navigation module 1221. The flight control module 1224 may also monitor the flight status of the flight device in real time, and adjust the flight attitude, speed, etc. based on the flight status to ensure the flight safety of the flight device.

[0051] In some embodiments, the flight control module 1224 can also be used for the management and storage of flight data and photographing data, and can support real-time processing and transmission of flight data and photographing data. Data transmission can adopt relevant encrypted transmission protocols to ensure the privacy and security of data during transmission.

[0052] The system monitoring module 1225 can be used to monitor the operating status of each module / component in the flight device 12 in real time to ensure the stable operation and flight safety of the flight device 12. For example, the system monitoring module 1225 can be used to monitor the operating status of the camera module 121 and the power device 123 in real time. When the system monitoring module 1225 monitors that the operating status of each module / component is abnormal, it can output corresponding warning information to the interactive device 11 so that the user can intervene in the abnormal processing in time.

[0053] The following combination Figure 4 and Figure 5 The process of how the shooting control device in the first flying device controls the first flying device to shoot is described. The first flying device may refer to Figure 1 The flying device 12 is shown.

[0054] Figure 4 This is a flowchart of the steps of the shooting control method provided by an embodiment of the present application. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0055] See also Figure 4 As shown, the shooting control method may include the following steps.

[0056] Step 401: Acquire surrounding environment information of a first flying device and task information indicating a shooting task.

[0057] In some embodiments, the first flying device may be equipped with a camera, or a camera and a laser radar, and the surrounding environment information of the first flying device may be acquired by the camera, or acquired by the camera and the laser radar. The shooting control device may acquire the surrounding environment information of the first flying device by communicating with the camera or the laser radar.

[0058] In some embodiments, the shooting control device may receive task information indicating a shooting task from an interactive device, and the task information may indicate the shooting requirements of the shooting task. For example, the interactive device may receive a user's shooting instruction, identify the task information of the shooting task represented by the shooting instruction based on a pre-trained semantic understanding model, and transmit the task information to the shooting control device so that the shooting control device can receive the task information of the shooting task. The task information can be used to describe the shooting requirements of the picture or video that the user expects to capture. For example, the task information may include: shooting subject, shooting type, expected shooting technique, expected shooting effect, etc., but is not limited thereto.

[0059] Step 402: planning a flight trajectory corresponding to the shooting task based on the task information and the surrounding environment information.

[0060] In some embodiments, the flight trajectory may include multiple trajectory points arranged in time sequence, and the connection line of the multiple trajectory points may represent the planned flight trajectory. The shooting control device may construct a three-dimensional map based on the surrounding environment information, locate the position of the flying device, and then plan the flight trajectory corresponding to the shooting task based on the task information.

[0061] Taking the first flying device equipped with a camera and a laser radar as an example, the surrounding environment information includes the image information of the surrounding environment collected by the camera and the point cloud information of the surrounding environment collected by the laser radar. The shooting control device can construct a three-dimensional map and locate the position of the flying device based on the image information and point cloud information of the surrounding environment through the SLAM algorithm, and then plan the flight trajectory corresponding to the shooting task based on the position information, mission information and three-dimensional map of the first flying device. The embodiment of the present application performs positioning and composition based on the SLAM algorithm, which can enhance the navigation and obstacle avoidance capabilities of the flying device in a complex environment, so that the flying device can work stably in a complex environment.

[0062] For example, the mission information indicates information such as the shooting subject, the desired shooting technique, and the desired shooting effect. The shooting control device can obtain the planned flight trajectory by marking the shooting subject, obstacles and other information in the three-dimensional map, using the identified obstacles, shooting subject and other information as constraints, and using the desired shooting technique and shooting effect and other information as planning goals.

[0063] In some embodiments, the shooting control device can plan the flight trajectory corresponding to the shooting task through a pre-trained trajectory planning model. For example, in order to avoid the training trajectory shooting model occupying more hardware resources (e.g., processor resources, memory resources, storage space resources, etc.) of the first flight device, the trajectory planning model can be trained through a training device (e.g., a server, a server cluster, etc.), and the pre-trained trajectory planning model can be deployed on the first flight device before the first flight device leaves the factory.

[0064] For example, the training steps for a trajectory planning model may include: a1. Obtaining trajectory planning samples, the trajectory planning samples may include multiple shooting task samples, and the flight trajectory corresponding to each shooting task sample; For example, multiple shooting task samples may include multiple aerial videos, which may correspond to multiple different types of shooting tasks. Different types of shooting tasks may refer to having different shooting subjects, different shooting techniques, different shooting effects, etc.

[0065] b1. Input the shooting task samples into the first preset model in batches to obtain the predicted flight trajectory, and then update the model parameters of the first preset model based on the flight trajectory corresponding to the shooting task samples and the predicted flight trajectory until the first preset model converges. The trained first preset model can be used as a trajectory planning model, or the trained first preset model can be used as a trajectory planning model after quantization. In this embodiment, the first preset model can be a model built based on a neural network, but is not limited thereto.

[0066] Step 403: Determine a shooting strategy corresponding to the shooting task based on the surrounding environment information and the pre-trained shooting model.

[0067] In some embodiments, the shooting strategy may include one or more of a camera movement strategy, a composition strategy, a camera parameter setting strategy, etc. For example, the surrounding environment information may represent the light intensity of the surrounding environment, whether the subject to be shot is a person, etc. The shooting control device may determine the shooting strategy corresponding to the shooting task based on the surrounding environment information and the pre-trained shooting model. By determining the shooting strategy corresponding to the shooting task, the shooting requirements such as the expected shooting subject, the expected shooting technique, the expected shooting effect, etc. indicated by the task information can be achieved.

[0068] For example, the camera movement strategy may include a strategy for controlling the flying action of the flying device, the camera direction, etc. during shooting, so as to achieve camera movement techniques such as pushing, pulling, shaking, moving, following, rising, lowering, and throwing.

[0069] The flight action may include, but is not limited to, the direction of the flight device, the speed of the flight device, whether the flight device is hovering, the pitch angle, yaw angle, and roll angle during flight, etc.

[0070] The camera orientation may be an angle of the camera relative to the subject being photographed. For example, the photographing control device may control the orientation of the camera by controlling the orientation of the pan / tilt.

[0071] For example, the composition strategy may be a strategy for setting the layout of the captured image or video. The composition strategy may include a nine-square grid composition, a triangle composition, a symmetrical composition, a rule of thirds composition, a golden ratio composition, and the like.

[0072] For example, the camera parameter setting strategy may be a strategy for setting camera parameters, and the camera parameters may include, but are not limited to, parameters such as aperture, shutter, sensitivity, focal length, depth of field, and exposure.

[0073] In some embodiments, the shooting model can be trained based on the first training sample set, and the trained shooting model can be deployed on the first aircraft device. For example, in order to avoid the training shooting model occupying more hardware resources of the first aircraft device, the shooting model can also be trained by a training device, and the pre-trained shooting model can be deployed on the first aircraft device before the first aircraft device leaves the factory.

[0074] For example, the training steps for a capture model may include: a2 obtain a first training sample set, the first training sample set includes a plurality of shooting samples, and each shooting sample corresponding to the shooting label, the plurality of shooting samples may include a plurality of shooting environments, the shooting sample may be an image or video, the shooting label may include a sample shooting strategy; b2. Train the second preset model based on the first training sample set to obtain a shooting model. For example, the shooting samples can be input into the second preset model in batches to obtain a predicted shooting strategy, and then the network parameters of the second preset model are updated based on the shooting strategy corresponding to the shooting samples and the predicted shooting strategy until the second preset model converges. The trained second preset model can be used as a shooting model. In this embodiment, the second preset model can be a model constructed based on a neural network, but is not limited thereto.

[0075] In some embodiments, the shooting model deployed in the first flight device may also be updated based on the second training sample set. For example, the shooting model deployed in the first flight device is trained using the second training sample set so that the updated shooting model can provide more personalized shooting services. The second training sample set may include historical flight shooting data of the first flight device and historical flight shooting data of at least one second flight device associated with the first flight device, so that the shooting model can not only self-optimize according to the data of the user's historical use of the first flight device, but also self-optimize according to the data of at least one second flight device, and can overcome the defect of the limited amount of historical flight shooting data of the first flight device, enrich the training samples, and enable the shooting model to provide more personalized shooting services.

[0076] For example, the second flight device associated with the first flight device may mean that the user of the second flight device and the user of the first flight device belong to the same user group portrait. The same user group portrait indicates that the users in the group have similar shooting preferences, and can greatly enrich the types and quantities of training samples used to update the shooting model, thereby improving the self-learning and optimization capabilities of the shooting model.

[0077] In some embodiments, the user of the flight device may decide whether to use the historical flight shooting data of the flight device for federated learning. For example, the user of the flight device may determine whether to use the historical flight shooting data of the flight device for federated learning through the authorization option provided by the interactive device. If it is determined that the historical flight shooting data of the flight device is used for federated learning, the shooting model of the flight device may perform self-learning and optimization based on federated learning.

[0078] In some embodiments, when the user determines to use the historical flight shooting data of the flight device for federated learning, the user's basic information and shooting preferences can be uploaded to the cloud in an encrypted manner. The cloud can construct a user group portrait based on the basic information and shooting preferences of each user. For flight devices belonging to the same user group portrait, the shooting model of the flight device under the user group portrait is updated through federated learning. The user's basic information may include age, gender, education level, occupation and other information, and the shooting preferences may include the user's preference for the use of the device functions of the flight device, shooting technique preference, shooting effect preference, etc. In some embodiments, the updating method of the shooting model may include: updating the shooting model through federated learning based on the historical flight shooting data of the first flying device and the historical flight shooting data of at least one second flying device associated with the first flying device. The federated learning method can ensure that a large amount of historical flight shooting data is used to update the shooting model, and the privacy of each user's historical flight shooting data can be guaranteed.

[0079] For example, the process of updating the shooting model through federated learning can be: the shooting control device trains the shooting model based on the historical flight shooting data of the first flight device. After the training is completed, the shooting control device can upload the obtained model parameters to the cloud in an encrypted manner, and the cloud can receive the model parameters uploaded by the shooting control devices of each flight device (the first flight device and one or more second flight devices). The cloud can combine all the model parameters for unified aggregation processing, such as obtaining new model parameters through weighted average, and then send the new model parameters to the shooting control devices of each flight device, so that each flight device can update the shooting model locally.

[0080] In some embodiments, the historical flight shooting data may include historical camera movement data, historical flight trajectory data, and historical shooting results. The historical shooting results may include camera shooting results and / or shooting results after processing the camera shooting results. Processing the camera shooting results may include beautifying, editing, rendering, and other processing of the camera shooting results. The shooting model learns the camera movement data under the historical flight trajectory data, as well as the beautification, editing, rendering, and other processing methods of the historical shooting results during the training phase, so that the flight equipment deployed with the shooting model can autonomously perform intelligent camera movement according to the current flight trajectory data, and intelligently beautify, edit, render, and other processing of the shooting results.

[0081] Step 404, executing the shooting task based on the flight trajectory and the shooting strategy.

[0082] After determining the flight trajectory and the shooting strategy, the shooting control device can perform the shooting task based on the flight trajectory and the shooting strategy. For example, the shooting control device can control the power device based on the flight trajectory so that the first flying device flies according to the flight trajectory. The shooting control device can also control the camera module based on the shooting strategy so that the camera module can capture images or videos corresponding to the user's shooting instructions.

[0083] The above-mentioned shooting control method can plan the flight trajectory and formulate the shooting strategy based on the user's shooting instructions, simplify the operation process of using the flight equipment for aerial shooting, and lower the operation threshold. The three-dimensional environment perception capability based on SLAM technology can enhance the navigation and obstacle avoidance capabilities of the flight equipment in complex environments, so that the flight equipment can work stably in complex environments. At the same time, artificial intelligence technology is used to formulate shooting strategies, improve the composition and camera movement capabilities of the flight equipment, and realize intelligent shooting. Through federated learning technology, the self-learning and optimization capabilities of the flight equipment can be enhanced, so that the flight equipment can self-optimize the shooting model according to historical flight shooting data, provide more personalized shooting services, and ensure the privacy of shooting data.

[0084] See also Figure 5 , is a flowchart of the steps of a shooting control method provided by another embodiment of the present application. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0085] See also Figure 5 As shown, the shooting control method may include the following steps.

[0086] Step 501: Acquire surrounding environment information of a first flying device and task information for indicating a shooting task.

[0087] Step 501 of the embodiment of the present application is similar to step 401 of the aforementioned embodiment, and will not be described again here to avoid repetition.

[0088] Step 502: planning a flight trajectory corresponding to the shooting task based on the task information and the surrounding environment information.

[0089] Step 502 of the embodiment of the present application is similar to step 402 of the aforementioned embodiment, and will not be described again here to avoid repetition.

[0090] Step 503: Determine a shooting strategy corresponding to the shooting task based on the surrounding environment information and the pre-trained shooting model.

[0091] In this embodiment, in addition to determining the shooting strategy corresponding to the shooting task, the shooting model also has the ability to perform aesthetic scoring on the images / videos obtained. In the pre-training stage of the shooting model, the shooting samples (images, videos) in the first sample set also have scoring labels. The shooting model can learn the scoring rules of these shooting samples, so that the shooting model can score the images / videos obtained by shooting, so that the user can also evaluate whether to reshoot based on the shooting score of the shooting result. For example, the scoring value of the shooting sample can be pre-scored based on the camera movement, composition, light and shadow, and color of the shooting sample to form a scoring label.

[0092] For another example, the shooting model may include two sub-networks, one sub-network for formulating shooting strategies, and the other sub-network for scoring the captured images / videos. The training process of the sub-network for formulating shooting strategies can refer to the description of the aforementioned embodiment, and will not be repeated here to avoid repetition.

[0093] The training process of the sub-network for scoring the captured images / videos may include: inputting the captured samples into the sub-network in batches to obtain predicted scoring values, and then updating the network parameters of the sub-network based on the scoring values ​​corresponding to the captured samples and the predicted scoring values ​​until the sub-network converges. The trained sub-network may be used to score the captured images / videos.

[0094] In some embodiments, when the subject is a person, the shooting control device may further generate a shooting posture suggestion or a reference shooting posture, and transmit the shooting posture suggestion or the reference shooting posture to the interactive device for reference by the shooting subject to adjust the shooting posture. For example, the shooting control device may generate a shooting posture suggestion or a reference shooting posture based on the task information and the information of the shooting subject, so that the shooting subject can adjust the shooting posture based on the shooting posture suggestion or the reference shooting posture.

[0095] The information of the subject may represent the subject's gender, age, etc. Generally speaking, users of different ages and genders may have different preferences for shooting postures, and different task information may also indicate different shooting requirements. Different shooting requirements may also be adapted to different shooting postures. By further setting shooting posture suggestions or reference shooting postures based on task information and subject information, the subject can be better guided to adjust the shooting posture to improve the shooting effect.

[0096] In some embodiments, the shooting subject may have different requirements for shooting strategies in different shooting postures. For example, in some shooting postures, the shooting subject uses posture details to represent his or her mood / inner activities. In this case, it may be necessary to pay attention to the posture details of the shooting subject (for example, it is necessary to shoot hand details or facial details, etc.), while some shooting postures do not require special attention to the posture details of the shooting subject. In order to solve this problem, the shooting control device can also obtain the shooting posture of the shooting subject, and determine the shooting strategy based on the surrounding environment information, shooting posture and shooting model, so as to comprehensively consider the surrounding environment information, shooting posture and shooting model to determine the shooting strategy, meet the needs for shooting strategies in different shooting postures, facilitate the display of the posture details of the shooting subject, and further improve the shooting effect.

[0097] Step 504, executing the shooting task based on the flight trajectory and the shooting strategy.

[0098] Step 504 of the embodiment of the present application is similar to step 404 of the aforementioned embodiment, and will not be described again here to avoid repetition.

[0099] Step 505: output the shooting result corresponding to the shooting task and the shooting score of the shooting result.

[0100] In some embodiments, after completing the shooting task, the shooting control device may output the shooting result corresponding to the shooting task and the shooting score of the shooting result. For example, the shooting control device may transmit the shooting result and the shooting score to the interactive device so that the user can view the shooting result and the shooting score on the interactive device. By displaying the shooting score of the shooting result, it is convenient for the user to evaluate whether to reshoot based on the shooting score of the shooting result, or share images / videos with high scores to other flight device users for shooting exchanges.

[0101] In some embodiments, the shooting result may be an image or a video. For example, if the shooting type indicated by the task information is image shooting, the generated shooting result is an image, and if the shooting type indicated by the task information is video shooting, the generated shooting result is a video.

[0102] The above-mentioned shooting control method can plan the flight trajectory and formulate the shooting strategy based on the user's shooting instructions, simplify the operation process of using the flight equipment for aerial shooting, and lower the operation threshold. The three-dimensional environment perception capability based on the SLAM technology can enhance the navigation and obstacle avoidance capabilities of the flight equipment in complex environments, so that the flight equipment can work stably in complex environments. At the same time, artificial intelligence technology is used to formulate shooting strategies, improve the composition and camera movement capabilities of the flight equipment, and realize intelligent shooting. Through federated learning technology, the self-learning and optimization capabilities of the flight equipment can be enhanced, so that the flight equipment can self-optimize the shooting model according to historical flight shooting data, provide more personalized shooting services, and ensure the privacy of shooting data. It can also output the shooting score of the shooting results, so that users can evaluate whether to reshoot or share the shooting results based on the shooting score of the shooting results. At the same time, shooting posture suggestions can be made and shooting strategies can be adjusted based on the shooting posture, which can further improve the shooting effect.

[0103] Please refer to Figure 6 , is a schematic diagram of the hardware structure of the shooting control device 122 provided in the embodiment of the present application. The shooting control device 122 can be integrated into Figure 1 In the flying device 12 shown, Figure 6 As shown, the shooting control device 122 may include a processor 1001 and a memory 1002. The memory 1002 is used to store one or more computer programs 1003. The one or more computer programs 1003 are configured to be executed by the processor 1001. The one or more computer programs 1003 include instructions, and the above instructions can be used to implement the above shooting control method in the shooting control device 122.

[0104] It is understandable that the structure shown in this embodiment does not constitute a specific limitation on the shooting control device 122. In other embodiments, the shooting control device 122 may include more or fewer components than shown, or combine or separate some components, or arrange the components differently.

[0105] The processor 1001 may include one or more processing units, for example, the processor 1001 may include an application processor (AP), a modem, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0106] The processor 1001 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 1001 is a cache memory. The memory may store instructions or data that the processor 1001 has just used or circulated. If the processor 1001 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 1001, and thus improves the efficiency of the system.

[0107] In some embodiments, the processor 1001 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM interface, and / or a USB interface, etc.

[0108] In some embodiments, processor 1001 is used to execute acceleration schemes such as single instruction multiple data (SIMD) and very long instruction word (VLIW).

[0109] In some embodiments, memory 1002 may include high-speed random access memory and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0110] This embodiment also provides a computer-readable storage medium, in which computer instructions are stored. When the instructions are executed on the flying device, the flying device executes the above-mentioned related method steps to implement the shooting control method in the above-mentioned embodiment.

[0111] Among them, the shooting control device and computer storage medium provided in this embodiment are used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0112] In practical applications, the above functions can be distributed to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0113] In several embodiments provided in the present application, the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are schematic. For example, the division of the modules or units is a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0114] The unit described as a separate component may or may not be physically separated, and the component shown as a unit may be one physical unit or multiple physical units, that is, it may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment.

[0115] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0116] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

[0117] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application.

Claims

1. A shooting control method, applied to a first flying device, characterized in that: The shooting control method comprises: Acquire surrounding environment information of the first flying device and task information indicating a shooting task; Planning a flight trajectory corresponding to the shooting task based on the task information and the surrounding environment information; Determining a shooting strategy corresponding to the shooting task based on the surrounding environment information and a pre-trained shooting model, wherein the shooting model is trained based on a first training sample set, and the shooting model is also updated based on a second training sample set, and the second training sample set includes historical flight shooting data of the first flying device and historical flight shooting data of a second flying device associated with the first flying device; The shooting task is performed based on the flight trajectory and the shooting strategy.

2. The shooting control method according to claim 1, characterized in that: The second flying device associated with the first flying device includes: a user of the second flying device and a user of the first flying device belong to the same user group portrait.

3. The shooting control method according to claim 1 or 2, characterized in that: The updating step of the shooting model comprises: Based on the historical flight shooting data of the first flying device and the historical flight shooting data of the second flying device associated with the first flying device, the shooting model is updated through federated learning, the historical flight shooting data includes historical camera movement data, historical flight trajectory data, and historical shooting results, and the historical shooting results include camera shooting results and / or shooting results after processing the camera shooting results.

4. The shooting control method according to claim 1 or 2, characterized in that: The surrounding environment information includes image information and point cloud information of the surrounding environment, and planning a flight trajectory corresponding to the shooting task based on the task information and the surrounding environment information includes: Based on the image information and point cloud information of the surrounding environment, the device is positioned and a three-dimensional map is constructed through a real-time positioning and map construction algorithm; A flight trajectory corresponding to the shooting task is planned based on the location information of the first flying device, the task information, and the three-dimensional map.

5. The shooting control method according to claim 1, characterized in that: The shooting control method further includes: Outputting a shooting result corresponding to the shooting task and a shooting score of the shooting result, wherein the shooting score is obtained based on the shooting model score.

6. The shooting control method according to claim 5, characterized in that: The training steps of the shooting model include: Acquire the first training sample set, the first training sample set including a plurality of shooting samples and a shooting label corresponding to each of the plurality of shooting samples, the plurality of shooting samples including a plurality of shooting environments, and the shooting label including a sample shooting score and a sample shooting strategy; A preset neural network is trained based on the first training sample set to obtain the shooting model.

7. The shooting control method according to claim 1 or 2, characterized in that: The determining, based on the surrounding environment information and the pre-trained shooting model, a shooting strategy corresponding to the shooting task includes: In the case where the photographing subject is a person, generating a reference photographing posture based on the task information and the information of the photographing subject, so that the photographing subject can adjust the photographing posture based on the reference photographing posture; The shooting posture of the shooting subject is obtained, and the shooting strategy is determined according to the surrounding environment information, the shooting posture and the shooting model.

8. A shooting control device, comprising a processor and a memory, characterized in that: The memory is used to store instructions, and the processor is used to call the instructions in the memory, so that the shooting control device executes the shooting control method according to any one of claims 1 to claim 7.

9. A shooting control system, characterized in that: include: An interactive device and a flying device, wherein the flying device comprises the shooting control device according to claim 8; The interactive device is used to receive a user's shooting instruction, determine task information of a shooting task corresponding to the shooting instruction based on a pre-trained semantic understanding model, and transmit the task information to the shooting control device.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on the aerial device, the aerial device executes the shooting control method according to any one of claims 1 to 7.

Images