Mobile phone combined with large model control flying carrier's photographing device and method

Abstract

This invention relates to a mobile phone-based camera system and its operating method for controlling an aircraft using a large-scale model. The system enables automatic photography with a low cost, high image clarity, and the ability to automatically adjust the camera position for optimal shooting. The system involves a mobile phone with a selfie app mounted on the aircraft. The selfie app communicates with a backend intelligent system via a wireless network, sending temporarily captured photos to the system. The system's composition analysis module analyzes the photos, providing movement direction and distance, and generating flight control commands to adjust the aircraft's position. These commands are then transmitted to the selfie app via the wireless network. The selfie app then sends these commands to the aircraft. Once the aircraft has executed the flight control commands and adjusted to the appropriate position, the selfie app optimizes the shooting parameters and timing, and controls the phone's camera to take the picture.

Description

Technical Field

[0001] This invention belongs to the technical field of mobile phone photography and background data processing, and particularly relates to a photography device that combines a mobile phone with a large model to control a flying vehicle, as well as a method for operating this photography device that combines a mobile phone with a large model to control a flying vehicle. Background Technology

[0002] While mobile phone photography is becoming increasingly widespread, automatic photo taking is still difficult to achieve. Furthermore, using a selfie stick is often insufficient to capture a truly satisfying photo where the photographer blends seamlessly with the surrounding scenery. Asking for help on the spot can be awkward, and the varying skill levels of photographers make it challenging to obtain images that accurately capture precious moments.

[0003] To avoid the above situations, current technology uses the camera built into the flying vehicle (such as a drone) for shooting. However, drones with camera functions are expensive, and these lenses generally have low resolution, failing to achieve the clarity of a mobile phone photo, and cannot be adjusted to the optimal shooting position. Summary of the Invention

[0004] To overcome the shortcomings of the prior art, the technical problem to be solved by the present invention is to provide a mobile phone combined with a large model to control the flight vehicle to take pictures. It can realize automatic shooting of mobile phone, has low cost, high image clarity, and can adjust to the best position to control the mobile phone to take pictures automatically.

[0005] The technical solution of this invention is as follows: This mobile phone combined with a large model controls a flying vehicle for taking pictures. The mobile phone with a selfie app is installed on the flying vehicle. The selfie app communicates with the background intelligent system through a wireless network and sends temporarily taken photos to the background intelligent system. The composition analysis module of the background intelligent system performs composition analysis on the temporarily taken photos, gives the direction and distance of movement, and generates flight control commands to adjust the position of the flying vehicle. These commands are transmitted to the selfie app through the wireless network. The selfie app then sends them to the flying vehicle. When the flying vehicle executes the flight control commands and adjusts to a suitable position, the selfie app optimizes the shooting parameters and shooting time and controls the mobile phone camera to take pictures.

[0006] This invention enables mobile photography by controlling a mobile vehicle via a mobile phone, replacing manual photo taking. The intelligent system in the background selects the lens for the photo and controls the flying vehicle to reach the best shooting point, achieving automatic photo taking with the mobile phone. It is low-cost, produces high-definition photos, and can adjust to the optimal position to control the mobile phone to take photos automatically.

[0007] A method for operating a mobile phone combined with a large model to control a flight vehicle for taking photos is also provided, which includes the following steps: (1) A mobile phone with a selfie app is installed on an aircraft; (2) The selfie app communicates with the background intelligent system via wireless network and sends the temporarily taken photos to the background intelligent system; (3) The composition analysis module of the background intelligent system performs composition analysis on the temporarily taken photos, gives the direction and distance of movement and generates flight control commands to adjust the position of the flying vehicle, and transmits them to the selfie APP through the wireless network; (4) The selfie app then distributes the photos to the flying vehicles; (5) The flight vehicle executes flight control commands and adjusts to a suitable position; (6) The selfie app optimizes the shooting parameters, shooting time and controls the phone camera to take pictures; (7) Determine whether the flight vehicle has moved to the optimal position. If the overall score of the photo is higher than the adjustable threshold, it has moved to the optimal position. If so, proceed to step (8); otherwise, jump to step (2). (8) The selfie app controls the flying vehicle to land, ending the entire photo-taking process. Attached Figure Description

[0008] Figure 1 A flowchart illustrating the working method of the photographing device for controlling a flying vehicle using a mobile phone combined with a large model according to the present invention. Detailed Implementation

[0009] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0010] To make the description of this disclosure more detailed and complete, illustrative descriptions of embodiments and specific examples of the present invention are provided below; however, these are not the only forms of implementing or utilizing the specific examples of the present invention. The embodiments cover features of multiple specific examples and methods and steps for constructing and operating these specific examples, and their order. However, other specific examples may also be used to achieve the same or equivalent functions and order of steps.

[0011] This mobile phone-based camera device, which combines a smartphone with a large model to control a flying vehicle, is installed on the flying vehicle. The smartphone with a selfie app is connected to a backend intelligent system via a wireless network. It sends temporarily captured photos to the backend intelligent system, where the composition analysis module analyzes the photos, provides the direction and distance of movement, and generates flight control commands to adjust the position of the flying vehicle. These commands are then transmitted to the selfie app via the wireless network and sent to the flying vehicle. When the flying vehicle executes the flight control commands and adjusts to the appropriate position, the selfie app optimizes the shooting parameters and shooting time, and controls the phone's camera to take the picture.

[0012] This invention enables mobile photography by controlling a mobile vehicle via a mobile phone, replacing manual photo taking. The intelligent system in the background selects the lens for the photo and controls the flying vehicle to reach the best shooting point, achieving automatic photo taking with the mobile phone. It is low-cost, produces high-definition photos, and can adjust to the optimal position to control the mobile phone to take photos automatically.

[0013] Preferably, the background intelligent system includes: an image acquisition and processing module, a composition analysis module, a large model aesthetic scoring module, and a flight command generation module; The image acquisition and processing module acquires image information from temporarily captured photos and performs scaling, noise reduction, white balance adjustment, and dynamic range optimization. The attitude and timestamp are synchronized, and the image frames are synchronized with the current attitude, timestamp, and inertial measurement unit metadata of the flight vehicle to provide contextual information for mapping decisions. The composition analysis module analyzes the current image in real time according to the rules of photographic composition, determines whether its composition is acceptable, and if not, provides the direction and distance for movement. The large-model aesthetic scoring module uses an image aesthetic evaluation model based on a large model to score the current image, providing a basis for whether to take a picture, and also for model feedback to optimize composition strategies. The flight command generation module outputs three-dimensional motion commands based on the current image composition state, the flight vehicle pose, and the target composition rules.

[0014] Preferably, the image acquisition and processing module performs image quality assessment to determine whether the image is overexposed, underexposed, or blurry, and discards low-quality image frames in advance.

[0015] Preferably, the composition analysis module includes photographic composition rules such as the rule of thirds, symmetrical composition, leading lines, and centering the subject.

[0016] Preferably, the composition analysis module performs the following steps: (3.1) Image semantic segmentation and subject detection: a deep learning-based object detection model that outputs the location, bounding box, and category of objects in the image; (3.2) Determine whether the current image conforms to the following rules: determine the target according to the instructions given by the user, calculate the geometric / visual symmetry of the image, detect the proximity of the subject to the intersection of the nine-square grid, the depth of field and the prominence of the subject, and select low position, high position or horizontal position according to the known paradigm.

[0017] Preferably, in the large model aesthetic scoring module, a large number of outstanding works by photographers are first collected, and artificial labels are formulated for their composition, including the position of a single object, multiple objects, symmetry, and area ratio in a grid. Based on the above data, a large model is trained to perform composition analysis on the images and output a metric that meets the criteria for excellent composition.

[0018] Preferably, in the large model aesthetic scoring module, a large number of high-scoring images are produced, a large number of excellent photographic works by photographers are collected, and they are labeled and scored as full or close to full marks; a large number of low-scoring images are produced, and the above photographic images are asymmetrically expanded and their colors are randomly adjusted, and they are labeled and scored as 0 or close to 0 marks; then a regression model is trained, and each frame of the image is sent to the large model aesthetic scoring module. If the score is higher than a set adjustable threshold, a result indicating that the image can be photographed is returned.

[0019] Preferably, in the flight command generation module, flight control commands are generated based on the difference between the coordinates of the object in the current photo and the ideal composition.

[0020] like Figure 1 As shown, a method for operating a mobile phone combined with a large model to control a flight vehicle for taking photos is also provided, which includes the following steps: (1) A mobile phone with a selfie app is installed on an aircraft; (2) The selfie app communicates with the background intelligent system via wireless network and sends the temporarily taken photos to the background intelligent system; (3) The composition analysis module of the background intelligent system performs composition analysis on the temporarily taken photos, gives the direction and distance of movement and generates flight control commands to adjust the position of the flying vehicle, and transmits them to the selfie APP through the wireless network; (4) The selfie app then distributes the photos to the flying vehicles; (5) The flight vehicle executes flight control commands and adjusts to a suitable position; (6) The selfie app optimizes the shooting parameters, shooting time and controls the phone camera to take pictures; (7) Determine whether the flight vehicle has moved to the optimal position. If the overall score of the photo is higher than the adjustable threshold, it has moved to the optimal position. If so, proceed to step (8); otherwise, jump to step (2). (8) The selfie app controls the flying vehicle to land, ending the entire photo-taking process.

[0021] Preferably, step (3) includes: (3.1) Image semantic segmentation and subject detection: a deep learning-based object detection model that outputs the location, bounding box, and category of objects in the image; (3.2) Determine whether the current image conforms to the following rules: determine the target according to the instructions given by the user, calculate the geometric / visual symmetry of the image, detect the proximity of the subject to the intersection of the nine-square grid, the depth of field and the prominence of the subject, and select low position, high position or horizontal position according to the known paradigm.

[0022] The present invention will now be described in more detail.

[0023] First, install a selfie app on your smartphone.

[0024] Then, secure the phone to the flying vehicle.

[0025] In selfie apps, users can manually select the shooting scene or input the desired shooting method via voice. When in front of people who need to take photos, the phone can control the flying vehicle under the command of the background intelligent system.

[0026] After the aircraft takes off, the mobile phone controls the aircraft.

[0027] The background intelligent system intelligently selects the content of photos taken from the phone's camera lens and controls the flying vehicle to move to the best shooting angle based on the selection.

[0028] The background intelligent system performs multi-dimensional analysis on temporary photos and dynamically determines the best shooting angle based on scene requirements. Specifically, it can be divided into the following core steps: After the image is acquired by the server, the composition analysis module performs composition analysis on the image to determine whether its composition is acceptable. If it is not acceptable, it provides the direction and distance for movement.

[0029] Module 1: Image Acquisition and Processing Module Design

[0030] This module serves as the data entry point for the entire intelligent photography system for flying vehicles and is a fundamental component for composition analysis and aesthetic scoring.

[0031] 1. The module collects image information from the mobile phone camera (not limited to photos or real-time image streams).

[0032] 2. Perform preliminary processing on image information to improve quality and reduce noise, including image preprocessing such as scaling, denoising, white balance adjustment, and dynamic range optimization (HDR) of the original image.

[0033] 3. Attitude and timestamp synchronization: Synchronize image frames with metadata such as the current pose of the flight vehicle, timestamp, and IMU, to provide contextual information for mapping decisions.

[0034] 4. Image quality assessment (optional): Quickly determine whether an image is overexposed, underexposed, or blurry, and discard low-quality image frames in advance to save backend computing resources.

[0035] Module 2: Flight Command Generation System Based on Graph Analysis

[0036] Based on photographic composition rules (such as the rule of thirds, symmetrical composition, leading lines, and subject centering), the system analyzes the current image in real time and dynamically adjusts the position and angle of the flight vehicle to achieve automatic framing and composition.

[0037] 1. Image semantic segmentation and subject detection

[0038] Method: A deep learning-based human / object detection model.

[0039] Output: Location, bounding box, and category of the main objects (people, buildings, landmarks, etc.) in the image.

[0040] 2. Composition Rules Determination

[0041] Determine if the current image meets the following rules:

[0042] Photo instruction: Determine the target (person / object) based on the instructions given by the user.

[0043] Symmetrical composition: Calculate the geometric / visual symmetry of an image.

[0044] Rule of Thirds: Detects whether the subject is close to the intersection of the 3x3 grid.

[0045] Foreground / background separation: Determine the depth of field and the prominence of the subject.

[0046] Viewpoint matching: Select low camera position, high camera position, or horizontal position based on the known paradigm.

[0047] The specific implementation of composition analysis is as follows: First, a large number of excellent works by photographers are collected, and their compositions are manually labeled, including the position of single and multiple main people and objects, symmetry, and area proportion in the rule of thirds; based on the above data, a large model is trained to perform composition analysis on the images and output a metric that conforms to excellent composition.

[0048] 3. Flight Command Generation

[0049] Input: Current image composition state, vehicle pose, target composition rules.

[0050] Output: 3D motion commands (forward / backward, left / right, rise / fall, pitch, turn).

[0051] Algorithm: Based on the coordinates of people / objects in the current photo and the difference from the ideal composition, a flight command is generated.

[0052] Module 3: Current Image Aesthetic Scoring System Based on Large Models

[0053] 1. Model selection and implementation method:

[0054] (1) Pre-trained image aesthetic model

[0055] An image aesthetic evaluation model based on a large model is adopted.

[0056] Input: Image.

[0057] Output: Aesthetic score of 1-10.

[0058] The image aesthetics model is used to score the current image, providing a basis for whether to take a picture, and can also be used for model feedback to optimize composition strategies.

[0059] The training process of the aesthetic image scoring model is as follows: A large number of high-scoring images are created, specifically by collecting a large number of excellent photographs by photographers and labeling them, i.e., scoring them as full marks or close to full marks; a large number of low-scoring images are created by performing asymmetric expansion and random color adjustments on the above photographs and labeling them, i.e., scoring them as 0 marks or close to 0 marks; then, a regression model is trained so that the model can score any image aesthetically.

[0060] (2) Online scoring process

[0061] Each frame of the image is fed into the aesthetic model.

[0062] If the score is higher than the set adjustable threshold (e.g., ≥ 6.5), the result indicating that the photo can be taken is returned.

[0063] Module coordination mechanism

[0064] 1. Receive instruction and image → Subject recognition → Composition analysis → Move / take photo

[0065] Detect the main object in the image and generate optimal composition suggestions → Control the flight vehicle to approach the ideal shooting point or take a picture.

[0066] 2. Real-time preview → Aesthetic rating → Shooting decision

[0067] The system inputs images into a large model for evaluation every fixed frame (e.g., 1 second).

[0068] If the composition is reasonable and the aesthetic score meets the standard, the camera will automatically take a picture; otherwise, the flight attitude will continue to be fine-tuned (flight commands are affected by the score feedback).

[0069] The camera image is obtained by using image object detection methods, so that the position of the main person or the object being photographed in the current frame is located in the image.

[0070] The background intelligent system calculates the difference between the current frame and the pre-learned optimal composition of the photograph. It also calculates the proportion and relative position of the photographed object in the entire image in the current frame. The intelligent system evaluates the above feature data of the current frame image and calculates whether its composition is reasonable. If it is unreasonable, it should be moved by an angle, distance, and height.

[0071] The background intelligent system server program sends movement commands to the aircraft. After the aircraft moves, it acquires the current frame of the camera image and sends it to the intelligent system for further analysis. This process is repeated multiple times until the overall score of the photo exceeds a threshold, at which point the final photo-taking stage begins, driving the phone's camera to take a picture.

[0072] After the photo is taken, the mobile app controls the flying vehicle to land, completing the entire photo-taking process.

[0073] The beneficial effects of the present invention are as follows: The method of taking pictures by controlling a mobile vehicle with a mobile phone can replace manual use of mobile phone to take pictures. The intelligent system selects the lens for the picture and controls the vehicle to reach the best shooting point. It can use a flying device, thus realizing automatic shooting of various scenes by mobile phone.

[0074] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A mobile phone combined with a large model to control a flying vehicle for taking photos, characterized in that: A mobile phone with a selfie app is installed on the aircraft. The selfie app communicates with the backend intelligent system via a wireless network, sending temporarily taken photos to the backend intelligent system. The composition analysis module of the backend intelligent system analyzes the composition of the temporarily taken photos, provides the direction and distance of movement, and generates flight control commands to adjust the position of the aircraft. These commands are then transmitted to the selfie app via the wireless network. The selfie app then sends the commands to the aircraft. When the aircraft executes the flight control commands and adjusts to the appropriate position, the selfie app optimizes the shooting parameters and shooting time, and controls the phone's camera to take a picture.

2. The mobile phone combined with a large model to control a flight vehicle for taking photos according to claim 1, characterized in that: The background intelligent system includes: an image acquisition and processing module, a composition analysis module, a large model aesthetic scoring module, and a flight command generation module; The image acquisition and processing module acquires image information from temporarily captured photos and performs scaling, noise reduction, white balance adjustment, and dynamic range optimization. The attitude and timestamp are synchronized, and the image frames are synchronized with the current attitude, timestamp, and inertial measurement unit metadata of the flight vehicle to provide contextual information for mapping decisions. The composition analysis module analyzes the current image in real time according to the rules of photographic composition, determines whether its composition is acceptable, and if not, provides the direction and distance for movement. The large-model aesthetic scoring module uses an image aesthetic evaluation model based on a large model to score the current image, providing a basis for whether to take a picture, and also for model feedback to optimize composition strategies. The flight command generation module outputs three-dimensional motion commands based on the current image composition state, the flight vehicle pose, and the target composition rules.

3. The mobile phone combined with a large model to control a flight vehicle for taking photos, as described in claim 2, is characterized in that: The image acquisition and processing module performs image quality assessment to determine whether the image is overexposed, underexposed, or blurry, and discards low-quality image frames in advance.

4. The mobile phone combined with a large model to control a flying vehicle for taking photos according to claim 3, characterized in that: The composition analysis module includes photographic composition rules such as the rule of thirds, symmetrical composition, leading lines, and centering the subject.

5. The mobile phone combined with a large model to control a flight vehicle for taking photos according to claim 4, characterized in that: The composition analysis module performs the following steps: (3.1) Image semantic segmentation and subject detection: a deep learning-based object detection model that outputs the location, bounding box, and category of objects in the image; (3.2) Determine whether the current image conforms to the following rules: determine the target according to the instructions given by the user, calculate the geometric / visual symmetry of the image, detect the proximity of the subject to the intersection of the nine-square grid, the depth of field and the prominence of the subject, and select low position, high position or horizontal position according to the known paradigm.

6. The mobile phone combined with a large model to control a flight vehicle for taking photos, as described in claim 5, is characterized in that: In the large-scale model aesthetic scoring module, a large number of outstanding works by photographers are first collected, and artificial labels are assigned to their compositions, including the position of a single object, multiple objects, symmetry, and area ratio in a rule of thirds grid. Based on the above data, a large model is trained to analyze the composition of the images and output a metric that meets the criteria for excellent composition.

7. The mobile phone combined with a large model to control a flight vehicle for taking photos according to claim 5, characterized in that: In the large-scale model aesthetic scoring module, a large number of high-scoring images are produced, and a large number of excellent photographs by photographers are collected, labeled, and scored as full or close to full marks. A large number of low-scoring images are produced, and the above photographs are asymmetrically expanded and their colors are randomly adjusted. They are then labeled and scored as 0 or close to 0 marks. Then, a regression model is trained, and each frame of the image is sent to the large-scale model aesthetic scoring module. If the score is higher than a set adjustable threshold, a result indicating that the image can be photographed is returned.

8. The mobile phone combined with a large model to control a flight vehicle for taking pictures, as described in claim 6 or 7, is characterized in that: In the flight command generation module, flight control commands are generated based on the difference between the coordinates of the object in the current photo and the ideal composition.

9. The working method of the mobile phone combined with a large model to control the photographing device of the flight vehicle according to claim 1, characterized in that: It includes the following steps: (1) A mobile phone with a selfie app is installed on an aircraft; (2) The selfie app communicates with the background intelligent system via wireless network and sends the temporarily taken photos to the background intelligent system; (3) The composition analysis module of the background intelligent system performs composition analysis on the temporarily taken photos, gives the direction and distance of movement and generates flight control commands to adjust the position of the flying vehicle, and transmits them to the selfie APP through the wireless network; (4) The selfie app then distributes the photos to the flying vehicles; (5) The flight vehicle executes flight control commands and adjusts to a suitable position; (6) The selfie app optimizes the shooting parameters, shooting time and controls the phone camera to take pictures; (7) Determine whether the flight vehicle has moved to the optimal position. If the overall score of the photo is higher than the adjustable threshold, it has moved to the optimal position. If so, proceed to step (8); otherwise, jump to step (2). (8) The selfie app controls the flying vehicle to land, ending the entire photo-taking process.

10. The working method of the mobile phone combined with a large model to control the photographing device of the flight vehicle according to claim 9, characterized in that: Step (3) includes: (3.1) Image semantic segmentation and subject detection: a deep learning-based object detection model that outputs the location, bounding box, and category of objects in the image; (3.2) Determine whether the current image conforms to the following rules: determine the target according to the instructions given by the user, calculate the geometric / visual symmetry of the image, detect the proximity of the subject to the intersection of the nine-square grid, the depth of field and the prominence of the subject, and select low position, high position or horizontal position according to the known paradigm.

Images