Unmanned aerial vehicle video synthesis method based on multi-modal large model

Abstract

The invention relates to the technical field of video synthesis, and provides an unmanned aerial vehicle video synthesis method based on a multi-modal large model. The method comprises the following steps: shooting a basic video through an unmanned aerial vehicle, and receiving a user video preference instruction; analyzing the instruction, obtaining a shooting attitude parameter, a main body and a key frame, and generating an attitude adjustment scheme including adjustment time according to the shooting attitude parameter and the real-time attitude parameter of the unmanned aerial vehicle; extracting an associated subject video in the basic video, and predicting key frame time; if the key frame time is earlier than the adjustment time, optimizing the scheme, obtaining the optimal adjustment time and adjusting the attitude; otherwise, adjusting and collecting the image according to the original scheme; and generating synthetic content through a multi-modal large model, analyzing and synthesizing key image positions, and outputting an unmanned aerial vehicle video synthesis result, so as to solve the technical problems that the shooting attitude is difficult to adjust in real time according to the personalized demand of a user in the unmanned aerial vehicle video shooting process, and the video content synthesis is difficult to accurately match the expectation of the user.

Description

Technical Field

[0001] The present application relates to the technical field of video synthesis, and in particular to a method for synthesizing unmanned aerial vehicle videos based on a multimodal large model. Background Art

[0002] In the process of modern drone video shooting, users usually need to make personalized adjustments and edits to the video to meet specific shooting needs and visual effects. However, traditional drone video shooting methods can only capture the main image by manually adjusting the drone's flight posture, and are often unable to flexibly and accurately adjust according to the user's real-time needs, resulting in unsatisfactory video shooting effects. In addition, the post-editing and synthesis process of the video usually requires a lot of manpower and time, especially when it involves complex scenes and dynamic subjects. Traditional methods are difficult to respond quickly and generate high-quality video content that meets user requirements. Summary of the invention

[0003] This application provides a drone video synthesis method based on a multimodal large model, aiming to solve the technical problems that it is difficult to adjust the shooting posture in real time according to the user's personalized needs during drone video shooting, and that it is difficult to accurately match the user's expectations in video content synthesis.

[0004] In view of the above problems, the present application provides a drone video synthesis method based on a multimodal large model.

[0005] The present application provides a method for synthesizing unmanned aerial vehicle videos based on a multimodal large model, the method comprising: collecting a basic video through unmanned aerial vehicle shooting, receiving a user's video preference instruction; parsing the content of the video preference instruction to obtain shooting posture parameters, a shooting subject and a shooting subject key frame, generating a posture adjustment scheme according to the shooting posture parameters and the real-time posture parameters of the current unmanned aerial vehicle, the posture adjustment scheme comprising a posture adjustment time; extracting a subject video associated with the shooting subject in the basic video, predicting the appearance of the shooting subject key frame, and obtaining the key frame appearance time; when the key frame appearance time is before the posture adjustment time, adjusting and optimizing the posture adjustment scheme according to the key frame appearance time, obtaining an optimal posture adjustment scheme and an optimal posture adjustment time before the key frame appearance time, performing unmanned aerial vehicle posture adjustment, and when the key frame appearance time is not before the posture adjustment time, performing unmanned aerial vehicle posture adjustment according to the posture adjustment scheme, and acquiring a shooting subject key image; generating synthesized content according to the video preference instruction through the multimodal large model, analyzing and obtaining a synthesis position based on the shooting subject key image, performing video synthesis on the synthesized content at the synthesis position within the subject key image, and obtaining a unmanned aerial vehicle video synthesis result.

[0006] One or more technical solutions provided in this application have at least the following technical effects or advantages: The above-mentioned method for synthesizing drone videos based on a multimodal large model first shoots a basic video through a drone, receives user instructions about the video content, and then parses these instructions to determine the shooting posture, the subject to be shot, and the key frames of these subjects in the video. Subsequently, based on this information and the current posture of the drone, a posture adjustment scheme is generated to adjust the posture of the drone during the shooting process to capture the best picture. After that, other video clips related to the subject to be shot are identified and extracted in the basic video, and the specific time when the key frame appears is predicted. If the predicted key frame appearance time is before the originally planned posture adjustment time, the posture adjustment scheme will be adjusted accordingly to ensure that the drone is in the best shooting posture before the key frame appears. If the key frame time is not before the adjustment time, the posture is adjusted as originally planned and the key image is captured. Then, the multimodal large model is used to generate additional synthetic content according to the user's video preference instructions. By analyzing the key images of the subject shot, the appropriate synthesis position is determined, and the generated synthetic content is integrated into these key images, so as to obtain the drone video synthesis result. The whole process aims to meet the user's demand for high-quality and personalized video content through the flexible shooting and video processing technology of drones.

[0007] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0009] Figure 1 A schematic diagram of a flow chart of a method for synthesizing unmanned aerial vehicle videos based on a multimodal large model in one embodiment; Figure 2 The present invention is a flowchart of content analysis of a method for synthesizing drone videos based on a multimodal large model in one embodiment. DETAILED DESCRIPTION

[0010] The embodiments of the present application provide a drone video synthesis method based on a multimodal large model to solve the technical problems that it is difficult to adjust the shooting posture in real time according to the user's personalized needs during drone video shooting, and that it is difficult to synthesize video content accurately to match user expectations.

[0011] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0012] It should be noted that the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or modules that are not explicitly listed or inherent to these processes, methods, products or devices.

[0013] Examples, such as Figure 1 As shown, the present application provides a method for synthesizing drone videos based on a multimodal large model, the method comprising: Collect basic videos through drone shooting and receive users' video preference instructions.

[0014] In an embodiment of the present application, when a video is shot by a drone, the drone will first execute a predetermined flight trajectory and collect basic video data of the surrounding environment in real time. During this process, the user can input personalized video preference instructions according to their own needs, and the instruction content involves the shooting subject, posture parameters, etc. For example, the instruction input by the user is "shoot a video of the setting sun from an upward angle." The system terminal will receive these video preference instructions input by the user so as to make adjustments to the subsequent shooting process according to the user's needs.

[0015] The video preference instruction is content parsed to obtain shooting posture parameters, shooting subjects and shooting subject key frames, and a posture adjustment scheme is generated according to the shooting posture parameters and the real-time posture parameters of the current UAV, wherein the posture adjustment scheme includes a posture adjustment time.

[0016] In one embodiment, after receiving the user's video preference instruction, the system terminal performs content analysis on the video preference instruction through the constructed instruction parser. The instruction parser analyzes the input video preference instruction according to the learned mapping relationship, and generates shooting posture parameters, shooting subjects and shooting subject key frames. Taking the instruction "shooting a video of the setting sun at an upward angle" as an example, the shooting posture parameters identified by the instruction parser are the upward angle, the shooting subject is the sun, and the shooting subject key frames are the continuous frames of the setting sun. After parsing the video preference instruction, the system terminal obtains the real-time posture parameters of the current drone, which include the current flight altitude, shooting angle, direction and other data of the drone. By comparing these real-time posture parameters with the shooting posture parameters required by the user, the height, angle, direction and other data that need to be adjusted are calculated, and a posture adjustment plan is generated according to the calculation results. The plan clarifies how the drone needs to adjust its posture to meet the user's shooting needs, for example, adjusting to an upward angle and maintaining stable shooting. The posture adjustment plan also includes the time required for adjustment. For example, it takes 1 minute to adjust the drone to a suitable shooting posture, so the posture adjustment time is 1 minute. In the generated attitude adjustment plan, the drone will gradually complete the attitude adjustment according to the attitude adjustment time to ensure that when the key frame of setting sun appears, it can be in the shooting position and angle required by the user to ensure the shooting effect.

[0017] Further, if Figure 2 As shown, the present application provides a method for parsing the content of the video preference instruction to obtain shooting posture parameters, shooting subjects and shooting subject key frames, including: A set of sample video preference instructions is collected, and shooting posture parameters, shooting subjects and shooting subject key frames in each sample video preference instruction are annotated to obtain a sample shooting posture parameter set, a sample shooting subject set and a sample shooting subject key frame set; an instruction parser is trained using the sample video preference instruction set, the sample shooting posture parameter set, the sample shooting subject set and the sample shooting subject key frame set; and the instruction parser is used to perform content analysis on the video preference instructions to obtain shooting posture parameters, shooting subjects and shooting subject key frames.

[0018] Preferably, in order to enable the instruction parser to accurately parse the content of the video preference instruction, the system terminal first collects a large number of sample video preference instructions and constructs a sample video preference instruction set. These sample instructions may contain user requirements for different shooting scenes, such as "looking down to shoot the city night view" or "looking up to shoot the setting sun". These instructions provide basic data for training the instruction parser. Subsequently, each collected sample video preference instruction is annotated, and the annotation content includes shooting posture parameters, shooting subjects and shooting subject key frames, and these annotation information are organized into a sample shooting posture parameter set, a sample shooting subject set and a sample shooting subject key frame set. After that, the sample video preference instruction set, the sample shooting posture parameter set, the sample shooting subject set and the sample shooting subject key frame set are divided into a training set and a verification set, and the constructed initial instruction parser is trained by the training set, and the verification set is used for verification after the training is completed to evaluate the performance of the model. Among them, the initial instruction parser can be constructed based on the RNN model, the Transformer model, etc. Taking the Transformer model as an example, an initial instruction parser structure is constructed using the Transformer, including an input layer, an output layer, a fully connected layer, an encoder, etc. Then, the weight of the initial instruction parser is initialized by random numbers or other methods, and the training set is input into the initialized initial instruction parser for forward propagation. It is passed layer by layer through the input layer, encoder, fully connected layer, output layer, etc., and the prediction results including posture parameters, shooting subjects and key frames are calculated. Then, the cross entropy loss function is used to calculate the loss value between the prediction result and the sample data, and the gradient of the loss to the weight of each layer is calculated layer by layer through back propagation. The Adam optimizer is then used to optimize the model parameters and adjust the weights to minimize the value of the loss function. Repeat the above process until the maximum number of iterations is reached. After the training is completed, the validation set is used to test the model performance and evaluate the accuracy of the model in the shooting posture, shooting subject and key frame classification tasks. If the accuracy meets the expected expectations, the current initial instruction parser is output as the final instruction parser. Otherwise, the hyperparameters such as the learning rate and the number of training batches are adjusted to further improve the content parsing effect of the instruction parser. Finally, the system terminal inputs the video preference instructions into the trained instruction parser, which will automatically analyze the instructions, identify the shooting posture parameters, shooting subjects and key frames, and use this information in subsequent video shooting and synthesis.

[0019] Furthermore, the present application provides a method for generating an attitude adjustment scheme according to the shooting attitude parameters and the real-time attitude parameters of the current drone, wherein the attitude adjustment scheme includes an attitude adjustment time, including: According to the controller of the UAV, a posture adjustment scheme is generated based on the shooting posture parameters and the real-time posture parameters of the current UAV; and the posture adjustment time of the posture adjustment scheme is obtained.

[0020] Preferably, after the content parsing is completed, the system terminal uses the shooting attitude parameters obtained by the parsing and the real-time attitude parameters of the current drone to perform attitude difference comparison and calculate the difference between the two. Among them, the real-time attitude parameters of the current drone are obtained from the controller of the drone, and these data usually include the pitch angle, direction, height, etc. of the drone. In the attitude difference comparison process, the system terminal calculates the difference between each attitude parameter in the shooting attitude parameters and the corresponding attitude parameters in the real-time attitude parameters of the current drone, obtains the attitude difference between the shooting attitude parameters and the real-time attitude parameters of the current drone, and then adds the calculated attitude difference to the attitude adjustment scheme. Subsequently, the system terminal obtains the motion capability of the drone, including adjustment speed, response time, etc., calculates the ratio of each attitude parameter difference in the attitude difference to the corresponding adjustment speed, and then accumulates it with the response time to obtain the time required to complete the attitude adjustment, as the attitude adjustment time of the attitude adjustment scheme, and adds it to the attitude adjustment scheme to ensure that the adjustment process meets the shooting requirements in the user instruction.

[0021] In the basic video, the associated subject video of the shooting subject is extracted, the appearance of the key frame of the shooting subject is predicted, and the appearance time of the key frame is obtained.

[0022] In one embodiment, in order to extract the images related to the subject, the system terminal inputs the basic video and the subject into a pre-built subject video extractor. The subject video extractor can automatically identify and extract the subject specified by the user in the basic video, and generate an associated subject video of the subject. For example, if the user's target is the sun, the subject video extractor accurately extracts all video clips including the sun by detecting the shape, color and lighting characteristics of the sun in the video frame, and summarizes these video clips in the output layer as the associated subject video of the subject. Subsequently, the associated subject video of the subject is input into the pre-built key frame time predictor for appearance time prediction, and the specific moment when the sun reaches the predetermined key position, that is, the key frame appearance time, is calculated, which provides an important basis for subsequent shooting posture adjustment and video synthesis.

[0023] Furthermore, the present application provides extracting the associated subject video of the shooting subject in the basic video, predicting the appearance of the key frame of the shooting subject, and obtaining the appearance time of the key frame, including: A sample basic video set and a sample shooting subject set are obtained, and the associated subject videos in the sample basic videos are cropped and divided according to different sample shooting subjects to obtain a sample associated subject video set; a subject video extractor is trained using the sample basic video set, the sample shooting subject set and the sample associated subject video set; the associated video of the shooting subject is extracted from the basic video using the subject video extractor to obtain an associated subject video; and the appearance of the key frame of the shooting subject is predicted based on the associated subject video to obtain the appearance time of the key frame.

[0024] Preferably, the system terminal first collects a sample basic video set, which contains video data of multiple different scenes, and each sample basic video contains different shooting subjects. For example, the sun, the moon, the mountains, etc., and at the same time obtains a sample shooting subject set, which corresponds to the different shooting objects in the sample video. For example, the sun, the moon, etc. Subsequently, each sample basic video is processed. For each sample basic video, it is cropped for different shooting subjects to ensure that each subject has a separate associated subject video, thereby generating a sample associated subject video set. The set contains all cropped associated subject videos, and each video contains only a specific subject and its motion trajectory. Afterwards, the same method as the aforementioned training instruction parser is used, through forward propagation, loss calculation, back propagation and optimization processes, the sample basic video set, the sample shooting subject set and the sample associated subject video set are used to train the initial subject video extractor to obtain the final subject video extractor. Among them, the initial subject video extractor can be constructed based on a convolutional neural network, etc. After constructing the subject video extractor, the system terminal inputs the basic video and the subject into the subject video extractor, which automatically identifies the subject in the video and extracts the part related to the subject as the associated subject video. After obtaining the associated subject video, the associated subject video is analyzed, that is, the key frame appearance time is obtained by inputting the associated subject video into the pre-constructed key frame time predictor for appearance time prediction.

[0025] Furthermore, the present application provides a method for predicting the occurrence of a key frame of the shooting subject according to the associated subject video to obtain the key frame occurrence time, including: Based on the sample-associated subject video set and the sample shooting subject set, the appearance time of the key frame of the sample shooting subject that appears after each sample-associated subject video is marked to obtain a sample key frame appearance time set; the sample-associated subject video set, the sample shooting subject set and the sample key frame appearance time set are used to train a key frame time predictor; the key frame time predictor is used to predict the appearance time of the shooting subject key frame to obtain the key frame appearance time.

[0026] Optionally, the system terminal defines key frames according to different shooting subjects. For example, for the setting sun, the key frame can be defined as the moment when the sun approaches the top of the mountain or sets. In each sample-associated subject video, the system terminal analyzes the motion trajectory of the shooting subject and marks the moment when the defined key frame appears. The system terminal uses the time points of these key frames (such as the setting sun after 30 seconds) as labels to form a sample key frame appearance time set. After marking all the key frame times of the sample-associated subject video set, the initial key frame time predictor is trained based on the sample-associated subject video set, the sample shooting subject set and the sample key frame appearance time set using the same training method as described above to construct the final key frame time predictor. Among them, the initial key frame time predictor can be constructed based on a long short-term memory network, a Transformer model, and the like. After the key frame time predictor is constructed, the system terminal inputs the shooting subject key frame and the shooting subject into the key frame time predictor, and the key frame time predictor automatically analyzes the characteristics of the shooting subject in the shooting subject key frame and predicts the key frame appearance time as a basis for subsequent adjustment and synthesis.

[0027] When the key frame appearance time is before the attitude adjustment time, the attitude adjustment scheme is adjusted and optimized according to the key frame appearance time, and the optimal attitude adjustment scheme and the optimal attitude adjustment time before the key frame appearance time are obtained, and the UAV attitude adjustment is performed; when the key frame appearance time is not before the attitude adjustment time, the UAV attitude adjustment is performed according to the attitude adjustment scheme, and the key image of the shooting subject is acquired.

[0028] In one embodiment, after obtaining the key frame appearance time, in order to avoid missing the key frame, the system terminal must ensure that the drone can complete the posture adjustment before the key frame time, so the key frame appearance time is compared with the posture adjustment time. If the key frame appearance time is less than or equal to the posture adjustment time, it means that the posture of the drone has not been adjusted when the key frame appears. At this time, the system terminal sets the posture adjustment constraint with the goal of reducing the posture adjustment time so that the drone completes the posture adjustment before the key frame appears, and obtains the optimal posture adjustment scheme by randomly adjusting and optimizing the posture adjustment scheme for multiple times. This optimal posture adjustment scheme has the highest similarity with the shooting posture parameters provided by the user, which can ensure that the shooting effect is close to the user's ideal state. Subsequently, the shooting posture parameters in the optimal posture adjustment scheme are calculated in the same way as above to obtain the optimal posture adjustment time, and add it to the optimal posture adjustment scheme. After obtaining the optimal posture adjustment scheme, the system terminal performs posture adjustment according to the optimized posture adjustment scheme. At this time, the adjustment time is shortened, and the angle or direction of the posture change is optimized to ensure that the drone completes the adjustment before the key frame time and enters the shooting state. If the key frame time is greater than the posture adjustment time, it means that when the key frame appears, the drone has been adjusted and entered the shooting state. At this time, the system terminal will execute the original attitude adjustment plan without optimization. After the optimal attitude adjustment plan is determined, the system terminal will control the drone to shoot at the moment when the key frame appears to obtain the key image of the subject. Through the above process of optimizing the attitude adjustment plan, it can be ensured that the drone completes the adjustment before the key frame arrives to avoid missing the shooting opportunity.

[0029] Furthermore, the present application provides an adjustment and optimization of the posture adjustment scheme according to the occurrence time of the key frame, including: According to the key frame appearance time, the optimized posture adjustment time is set as the posture adjustment constraint before the key frame appearance time; the posture adjustment scheme is randomly adjusted to obtain a first posture adjustment scheme, and the first adjustment posture parameter and the first posture adjustment time of the first posture adjustment scheme are obtained; when the posture adjustment constraint is met, the similarity between the first adjustment posture parameter and the shooting posture parameter is analyzed, and the fitness of the first scheme is calculated in combination with the first posture adjustment time and the key frame appearance time; the first posture adjustment scheme is continued to be randomly adjusted and iteratively optimized until convergence, and an optimal posture adjustment scheme with the largest scheme fitness is output, and the optimal posture adjustment time of the optimal posture adjustment scheme is obtained.

[0030] Preferably, when the key frame appearance time is less than or equal to the attitude adjustment time, in order to ensure that the drone can complete the attitude adjustment before the key frame appears, the system terminal uses the key frame time as a constraint condition, that is, the completion time of the attitude adjustment must be less than the key frame appearance time. For example, if the key frame appearance time is 30 seconds, the target time of the attitude adjustment must be completed within 30 seconds. Under the attitude adjustment constraint, the system terminal generates a new attitude adjustment scheme, that is, the first attitude adjustment scheme, by randomly adjusting the current attitude adjustment scheme. During the adjustment process, the system terminal randomly adjusts the shooting attitude parameters in the attitude adjustment scheme to obtain the first shooting attitude parameters, performs attitude difference analysis by the same method as above, obtains the first attitude difference, and calculates the adjusted first attitude adjustment time, and then integrates the first shooting attitude parameters, the first attitude difference and the first attitude adjustment time to generate the first attitude adjustment scheme. After obtaining the first attitude adjustment scheme, check whether the generated first attitude adjustment scheme meets the constraint condition of the attitude adjustment time, that is, whether the adjustment can be completed before the key frame appearance time. If the first attitude adjustment time meets the condition, the system terminal will analyze other characteristics of the scheme. If the conditions are not met, the scheme will be abandoned and a new adjustment scheme will continue to be randomly generated until an adjustment scheme that meets the constraints is found. For the first posture adjustment scheme that meets the posture adjustment constraints, the system terminal uses a pre-built posture similarity analyzer to calculate the similarity between the adjustment posture parameters of the scheme and the shooting posture parameters provided by the user. The higher the similarity, the closer the adjusted posture is to the optimal shooting posture expected by the user. After obtaining the posture similarity, the system terminal combines the posture adjustment time for evaluation and calculates the fitness of the first posture adjustment scheme to measure the overall pros and cons of the adjustment scheme. Repeat the above process until the fitness of the posture adjustment scheme is no longer significantly improved and reaches a convergence state. At this time, the system terminal outputs the posture adjustment scheme with the highest fitness and the corresponding posture adjustment time as the final optimal posture adjustment scheme and optimal posture adjustment time. This optimal posture adjustment scheme can ensure that the drone can capture key images at the optimal shooting posture.

[0031] Further, the present application provides a method of analyzing the similarity between the first adjustment posture parameter and the shooting posture parameter when the posture adjustment constraint is satisfied, combining the first posture adjustment time and the key frame appearance time, and calculating the fitness of the first solution, including: Collect a set of sample adjustment posture parameters and a set of sample shooting posture parameters, perform similarity annotation according to the shooting position and shooting angle of each set of sample adjustment posture parameters and sample shooting posture parameters, and obtain a sample posture similarity set; use the sample adjustment posture parameter set and the sample shooting posture parameter set, as well as the sample posture similarity set, to train a posture similarity analyzer; use the posture similarity analyzer to perform similarity analysis on the first adjustment posture parameters and the shooting posture parameters to obtain a first posture similarity; calculate the ratio of the key frame appearance time to the first posture adjustment time, and combine the first posture similarity to obtain a first scheme fitness through weighted calculation.

[0032] Optionally, the system terminal collects the adjustment attitude parameter data of the drone in actual operation, which contains the attitude of the drone at each moment during the adjustment process. Each adjustment attitude parameter reflects the details of how the drone transitions from the current attitude to the target attitude. Then collect the best shooting attitude parameter set expected by the user, indicating the attitude that the user expects the drone to take at a specific moment, including the same pitch angle, roll angle and heading angle. For each set of sample adjustment attitude parameters and sample shooting attitude parameters, by comparing their specific shooting positions and angles, the similarity between them is calculated in combination with the Euclidean distance, and the result is recorded as a sample attitude similarity set. The higher the similarity, the closer the posture of the drone after adjustment is to the posture expected by the user. Subsequently, the initial attitude similarity analyzer is trained based on the sample adjustment attitude parameter set, the sample shooting attitude parameter set and the sample attitude similarity set using the same training method as described above, and the final attitude similarity analyzer is constructed. Among them, the initial attitude similarity analyzer can be constructed based on linear regression, support vector machine, neural network and the like. After constructing the posture similarity analyzer, the system terminal inputs the first adjustment posture parameter and the shooting posture parameter into the posture similarity analyzer. The similarity analyzer will calculate the first posture similarity between the two based on the input first adjustment posture parameter and the shooting posture parameter. This similarity indicates the closeness between the adjusted posture and the optimal shooting posture. The higher the similarity, the closer it is. After that, the ratio of the key frame appearance time to the first posture adjustment time is calculated. This ratio is used to measure the relative efficiency of the adjustment speed. The closer it is to 1, the more appropriate the adjustment speed. Then, combining the first posture similarity and the calculated ratio, the overall fitness of the first posture adjustment scheme is weightedly calculated to obtain the fitness of the first scheme. The purpose of fitness is to comprehensively consider the accuracy and time efficiency of posture adjustment to ensure that the drone can accurately adjust to the target shooting posture before the key frame arrives.

[0033] Through the multimodal large model, synthetic content is generated according to the video preference instructions, and based on the key image of the shooting subject, the synthesis position is analyzed and obtained. At the synthesis position within the key image of the subject, the synthetic content is subjected to video synthesis to obtain the drone video synthesis result.

[0034] In one embodiment, after obtaining the key image of the subject, the system terminal inputs the video preference instruction and the key image of the subject into the multimodal large model, and analyzes the video preference instruction and the key image of the subject through the synthetic content generator and synthetic position identifier inside the multimodal large model, respectively, to generate synthetic content and synthetic position. Subsequently, the subject key image is marked according to the synthetic position, and the synthetic content is synthesized by the marked position to obtain the drone video synthesis result. This drone video synthesis result meets the user's preference requirements, while retaining the key image of the subject, which can ensure the beauty of the overall video and user experience.

[0035] Further, the present application provides generating synthetic content according to the video preference instruction through a multimodal large model, analyzing and obtaining a synthetic position based on the key image of the shooting subject, and performing video synthesis on the synthetic content at the synthetic position within the key image of the subject, including: A sample video preference instruction set is obtained, and a sample synthesis content set is set; the sample video preference instruction set and the sample synthesis content set are used to train a synthesis content generator, and the video preference instruction is input into the synthesis content generator to generate synthesis content; a sample shooting subject key image set is collected, and the position for synthesizing the synthesis content in each sample shooting subject key image is marked to obtain a sample synthesis position set; the sample shooting subject key image set and the sample synthesis position set are used to train a synthesis position identifier, and the shooting subject key image is analyzed to obtain a synthesis position; at the synthesis position in the subject key image, the synthesis content is synthesized to obtain a drone video synthesis result.

[0036] Preferably, the system terminal first constructs a multimodal large model, which is obtained by encapsulating a synthetic content generator and a synthetic location identifier. For the synthetic content generator, the system terminal collects a variety of video preference instructions input by the user, such as "add the text 'sunset afterglow' to the sunset scene" or "add animation effects to the mountain background". By summarizing these video preference instructions, a sample video preference instruction set is obtained, and this sample video preference instruction set is used to help understand the user's needs and preferences. Then, according to the video preference instructions, a corresponding synthetic content set is set. For example, the text instruction "sunset afterglow" input by the user will be used as a sample synthetic content, or the animation effect (such as a flashing halo) generated according to the user instruction also belongs to the sample synthetic content set. Subsequently, the initial synthetic content generator is trained based on the sample video preference instruction set and the sample synthetic content set using the same training method as described above to construct the final synthetic content generator. Among them, the initial synthetic content generator can be constructed based on a long short-term memory network, a gated recurrent unit, and the like. After constructing the synthetic content generator, the system terminal inputs the video preference instruction into the synthetic content generator, and the synthetic content generator automatically generates the corresponding synthetic content, including text, animation, image effects, etc. The generator can dynamically adjust the content style according to different scenes. For the synthetic position identifier, the system terminal collects key frame images containing the subject, which reflect the specific position and state of the subject in the video. Then analyze each sample subject key image to determine the synthetic position suitable for the synthetic content. For example, if the subject is above the image, the position of the synthetic content may be placed below to avoid overlapping with the subject. According to the analysis results, the synthetic position is marked to obtain a sample synthetic position set. There will be one or more synthetic position annotations in each image to ensure that the synthetic content and the subject are reasonably distributed during the synthesis process. Subsequently, the initial synthetic position identifier is trained based on the sample subject key image set and the sample synthetic position set using the same training method as mentioned above to construct the final synthetic position identifier. Among them, the initial synthetic position identifier can be constructed based on a convolutional neural network, a target detection model, etc. After building the synthesis position identifier, the system terminal inputs the key image of the subject into the synthesis position identifier, and the synthesis position identifier automatically detects the corresponding synthesis position. After that, the generated synthesis content (such as the text "sunset afterglow" or animation effect) will be matched according to the identified synthesis position, and the synthesis content will be embedded in the key image of the subject to ensure that the synthesis content does not conflict with the subject, while achieving the best visual effect, and then generate the drone video synthesis result.

[0037] In summary, the embodiments of the present application have at least the following technical effects: The embodiment of the present application collects basic videos through drone shooting, and parses the instruction content in combination with the user's video preference instructions to obtain shooting posture parameters, shooting subjects and key frame information of the shooting subjects. Subsequently, according to the shooting posture parameters and the real-time posture parameters of the drone, a posture adjustment scheme is generated, and the key frames of the associated videos of the shooting subjects are predicted. If the key frame appears earlier than the posture adjustment time, the posture adjustment scheme is optimized to ensure that the drone completes the posture adjustment before the key frame appears. After that. Through a multimodal large model, synthetic content is generated according to the user's video preference instructions, and the key images of the shooting subject are analyzed to determine the synthesis position of the synthetic content, and finally generate the drone video synthesis result. These technical effects jointly solve the technical problems that it is difficult to adjust the shooting posture in real time according to the user's personalized needs during drone video shooting, and that the video content synthesis is difficult to accurately match the user's expectations, and realize the analysis of user instructions through a multimodal large model, automatically adjust the drone posture and optimize the shooting process, combine key frame prediction and video synthesis to ensure that the video content is highly consistent with the user's preferences, and improve the intelligence and accuracy of video shooting and synthesis.

[0038] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. And the above-mentioned specific embodiments of this specification are described. The processes depicted in the accompanying drawings do not necessarily require the specific order and continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0039] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0040] This specification and drawings are merely exemplary illustrations of the present application and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application intends to include these modifications and variations.

Claims

1. A method for synthesizing drone videos based on a multimodal large model, characterized in that: The method comprises: Collect basic videos through drone shooting and receive users' video preference instructions; Parsing the content of the video preference instruction to obtain shooting posture parameters, shooting subjects and shooting subject key frames, and generating a posture adjustment plan according to the shooting posture parameters and the real-time posture parameters of the current UAV, wherein the posture adjustment plan includes a posture adjustment time; Extracting the associated subject video of the subject in the basic video, predicting the appearance of the key frame of the subject, and obtaining the appearance time of the key frame; When the key frame appearance time is before the attitude adjustment time, the attitude adjustment scheme is adjusted and optimized according to the key frame appearance time, the optimal attitude adjustment scheme and the optimal attitude adjustment time before the key frame appearance time are obtained, and the attitude adjustment of the drone is performed; when the key frame appearance time is not before the attitude adjustment time, the attitude adjustment of the drone is performed according to the attitude adjustment scheme, and the key image of the shooting subject is acquired; Through the multimodal large model, synthetic content is generated according to the video preference instructions, and based on the key image of the photographed subject, the synthesis position is analyzed and obtained. At the synthesis position within the key image of the subject, the synthetic content is subjected to video synthesis to obtain the drone video synthesis result.

2. The method for synthesizing unmanned aerial vehicle videos based on a multimodal large model according to claim 1, characterized in that: The video preference instruction is analyzed for content to obtain shooting posture parameters, shooting subjects and shooting subject key frames, including: Collecting a set of sample video preference instructions, annotating the shooting posture parameters, shooting subjects and shooting subject key frames in each sample video preference instruction, and obtaining a sample shooting posture parameter set, a sample shooting subject set and a sample shooting subject key frame set; Using the sample video preference instruction set, the sample shooting posture parameter set, the sample shooting subject set and the sample shooting subject key frame set, to train an instruction parser; The instruction parser is used to parse the video preference instruction to obtain shooting posture parameters, shooting subjects and shooting subject key frames.

3. The method for synthesizing unmanned aerial vehicle videos based on a multimodal large model according to claim 1, characterized in that: Generate an attitude adjustment scheme according to the shooting attitude parameters and the real-time attitude parameters of the current UAV, wherein the attitude adjustment scheme includes an attitude adjustment time, including: According to the controller of the UAV, based on the shooting attitude parameters and the real-time attitude parameters of the current UAV, a posture adjustment plan is generated; Obtain the posture adjustment time of the posture adjustment scheme.

4. The method for synthesizing unmanned aerial vehicle videos based on a multimodal large model according to claim 1, characterized in that: Extracting the associated subject video of the subject in the basic video, predicting the appearance of the key frame of the subject, and obtaining the appearance time of the key frame include: Obtain a sample basic video set and a sample shooting subject set, and cut and divide the associated subject videos in the sample basic video according to different sample shooting subjects to obtain a sample associated subject video set; Using the sample basic video set, the sample shooting subject set and the sample associated subject video set to train a subject video extractor; Utilizing the subject video extractor, extracting the associated video of the shooting subject from the basic video to obtain the associated subject video; Based on the associated subject video, the occurrence prediction of the key frame of the shooting subject is performed to obtain the key frame occurrence time.

5. The method for synthesizing unmanned aerial vehicle videos based on a multimodal large model according to claim 4, characterized in that: According to the associated subject video, predicting the appearance of the key frame of the shooting subject to obtain the appearance time of the key frame includes: Based on the sample-associated subject video set and the sample-shot subject set, annotating the appearance time of the key frame of the sample-shot subject after each sample-associated subject video to obtain a sample key frame appearance time set; Using the sample associated subject video set, the sample shot subject set and the sample key frame appearance time set, training a key frame time predictor; The key frame time predictor is used to predict the appearance time of the key frame of the shooting subject to obtain the key frame appearance time.

6. The method for synthesizing unmanned aerial vehicle videos based on a multimodal large model according to claim 1, characterized in that: According to the key frame appearance time, the posture adjustment scheme is adjusted and optimized, including: According to the key frame appearance time, setting the optimized posture adjustment time before the key frame appearance time as the posture adjustment constraint; Randomly adjust the posture adjustment scheme to obtain a first posture adjustment scheme, and obtain a first adjustment posture parameter and a first posture adjustment time of the first posture adjustment scheme; When the posture adjustment constraint is met, the similarity between the first adjustment posture parameter and the shooting posture parameter is analyzed, and the fitness of the first solution is calculated by combining the first posture adjustment time and the key frame appearance time; Continue to perform random adjustment and iterative optimization on the first posture adjustment scheme until convergence, output an optimal posture adjustment scheme with the maximum scheme fitness, and obtain an optimal posture adjustment time of the optimal posture adjustment scheme.

7. The method for synthesizing unmanned aerial vehicle videos based on a multimodal large model according to claim 6, characterized in that: When the posture adjustment constraint is met, analyzing the similarity between the first adjustment posture parameter and the shooting posture parameter, combining the first posture adjustment time and the key frame appearance time, and calculating the fitness of the first solution, including: Collecting a set of sample adjustment posture parameters and a set of sample shooting posture parameters, and performing similarity annotation according to the shooting position and shooting angle of each set of sample adjustment posture parameters and sample shooting posture parameters to obtain a sample posture similarity set; Using the sample adjustment posture parameter set and the sample shooting posture parameter set, as well as the sample posture similarity set, to train a posture similarity analyzer; Using the posture similarity analyzer, performing similarity analysis on the first adjustment posture parameter and the shooting posture parameter to obtain a first posture similarity; The ratio of the key frame appearance time to the first posture adjustment time is calculated, and combined with the first posture similarity, a weighted calculation is performed to obtain the fitness of the first solution.

8. The method for synthesizing unmanned aerial vehicle videos based on a multimodal large model according to claim 1, characterized in that: Generate synthetic content according to the video preference instruction through the multimodal large model, analyze and obtain a synthetic position based on the key image of the shooting subject, and perform video synthesis on the synthetic content at the synthetic position within the key image of the subject, including: Obtain a sample video preference instruction set and set a sample synthesis content set; Using the sample video preference instruction set and the sample synthetic content set to train a synthetic content generator, inputting the video preference instruction into the synthetic content generator to generate synthetic content; Collect a set of key images of the sample shooting subject, mark the position where the synthetic content is synthesized in each key image of the sample shooting subject, and obtain a set of sample synthesis positions; Using the sample subject key image set and the sample synthesis position set, training a synthesis position identifier, analyzing the subject key image, and obtaining a synthesis position; At the synthesis position in the subject key image, video synthesis is performed on the synthesis content to obtain a drone video synthesis result.

Images