Photographing method, apparatus and electronic device

Abstract

The application discloses a photographing method, a photographing device and an electronic device, and belongs to the technical field of camera shooting. The method comprises the following steps: acquiring first information collected by a holder device and second information collected by an electronic device, wherein the first information comprises at least one of the following: first motion information and first image information, the second information comprises at least one of the following: second motion information and second image information; and an image or a video is obtained based on the first information and the second information.

Description

Technical Field

[0001] This application belongs to the field of camera technology, specifically relating to a shooting method, device, and electronic equipment. Background Technology

[0002] Currently, users are taking photos with electronic devices more and more frequently, which in turn increases the requirements for the stability of the photos taken.

[0003] In related technologies, in order to maintain the stability of the captured footage and the quality of the captured image during motion, optical image stabilization or electronic image stabilization can be used to compensate for the captured footage during motion to obtain a compensated image.

[0004] However, optical image stabilization increases the size and weight of the lens, and there are still limitations to low-speed telephoto image stabilization. Electronic image stabilization relies on cropping the image, resulting in a loss of image quality. Neither of these methods can completely counteract severe and rapid shaking, leading to poor image quality when images are captured by electronic devices. Summary of the Invention

[0005] The purpose of this application is to provide a shooting method, apparatus, electronic device, storage medium, and program product that can improve the stability and quality of shooting by electronic devices.

[0006] In a first aspect, embodiments of this application provide a shooting method, which includes: acquiring first information collected by a gimbal device and second information collected by an electronic device, wherein the first information includes at least one of the following: first motion information and first image information, and the second information includes at least one of the following: second motion information and second image information; and capturing an image or video based on the first information and the second information.

[0007] Secondly, embodiments of this application provide a shooting device, which includes an acquisition module and a processing module. The acquisition module is used to acquire first information collected by a gimbal device and second information collected by an electronic device, wherein the first information includes at least one of the following: first motion information and first image information; and the second information includes at least one of the following: second motion information and second image information. The processing module is used to capture images or videos based on the first and second information acquired by the acquisition module.

[0008] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions, when executed by the processor, implementing the steps of the method described in the first aspect.

[0009] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0010] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0011] In a sixth aspect, embodiments of this application provide a computer program / program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0012] In this embodiment, by combining the first motion information collected by the external gimbal device and the second motion information collected by the electronic device, the images or videos captured by the electronic device are processed, breaking through the stabilization accuracy bottleneck of the single motion information of the electronic device, thereby improving the stabilization effect; and by using the first image information collected by the gimbal device to obtain image information other than the second image information, the image enhancement effect of the electronic device is improved. Attached Figure Description

[0013] Figure 1 This is one of the flowcharts of a shooting method provided in the embodiments of this application;

[0014] Figure 2 This is a second flowchart of a shooting method provided in an embodiment of this application;

[0015] Figure 3 This is a schematic diagram of a motion data fusion method provided in an embodiment of this application;

[0016] Figure 4 This is the third flowchart of a shooting method provided in the embodiments of this application;

[0017] Figure 5 This is the fourth flowchart of a shooting method provided in the embodiments of this application;

[0018] Figure 6 This is a schematic diagram illustrating an application scenario of a shooting method provided in an embodiment of this application;

[0019] Figure 7 This is a schematic diagram of the structure of a shooting device provided in an embodiment of this application;

[0020] Figure 8 This is one of the hardware structure diagrams of an electronic device provided in the embodiments of this application;

[0021] Figure 9 This is a second schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0023] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects. For example, a first object can be one or more, where "more" means at least two. Furthermore, in this application, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0024] In this application, the terms "at least one," "at least one," etc., refer to any one, any two, or a combination of two or more of the included objects. For example, at least one of a, b, and c can mean: "a," "b," "c," "a and b," "a and c," "b and c," and "a, b, and c," where a, b, and c can be single or multiple, and multiple means at least two. Similarly, "at least two" means two or more, and its meaning is similar to "at least one." The identifiers in this application are text, symbols, images, etc., used to indicate information, and can use controls or other containers as the carriers of displayed information, including but not limited to text identifiers, symbol identifiers, and image identifiers.

[0025] The terminology used in the implementation section of this application is only for explaining specific embodiments of this application and is not intended to limit this application. The terminology involved in the embodiments of this application is explained below.

[0026] The shooting method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0027] The shooting method provided in this application embodiment can be applied to shooting scenarios.

[0028] The following examples illustrate the shooting method provided in the embodiments of this application using specific scenarios.

[0029] Scenario 1: In a scenario where a user is running and filming, the gimbal device connected to the electronic device can send the collected motion information to the electronic device. The electronic device can then use the motion information collected by the gimbal device and the electronic device to perform image stabilization on the images or videos captured by the camera module in the electronic device, thus obtaining the images or videos.

[0030] Scenario 2: In scenarios where users want to blur the background, a gimbal device connected to an electronic device can send multiple image information captured from different shooting angles to the electronic device. The electronic device can then determine the background area based on these multiple image information, thereby blurring the background of the images or videos captured by the electronic device to obtain the blurred images or videos.

[0031] It should be noted that the above scenarios 1 to 2 are merely exemplary examples of some scenarios that may be applied to the embodiments of this application. In actual implementation, the embodiments of this application can also be applied to more possible scenarios such as shooting, and the embodiments of this application are not limited here.

[0032] Based on the above-mentioned scenarios applied in the embodiments of this application, the shooting method provided in the embodiments of this application, by combining the first motion information collected by the external gimbal device and the second motion information collected by the electronic device, processes the image or video captured by the electronic device, breaks through the stabilization accuracy bottleneck of the single motion information of the electronic device, thereby improving the stabilization effect; and by using the first image information collected by the gimbal device to obtain image information other than the second image information, thereby improving the image enhancement effect of the electronic device.

[0033] The shooting method provided in this application is executed by a shooting device, which can be an electronic device, or a functional module or entity within an electronic device. This application does not limit the specific device to this type of device. The following will use an electronic device as an example to illustrate the shooting method provided in this application. This electronic device is mounted on a gimbal and is communicatively connected to the gimbal.

[0034] This application provides a shooting method. Figure 1 A flowchart illustrating a shooting method provided in an embodiment of this application is shown. Figure 1 As shown, the shooting method provided in this application embodiment may include the following steps 201 and 202.

[0035] Step 201: The electronic device acquires the first information collected by the gimbal device and the second information collected by the electronic device.

[0036] In some embodiments, the first information includes at least one of the following: first motion information, first image information, and the second information includes at least one of the following: second motion information, second image information.

[0037] It is understood that the aforementioned gimbal devices may include gimbal cameras.

[0038] It is understood that the aforementioned gimbal device can be an external gimbal device that connects to electronic devices.

[0039] Optionally, in this embodiment of the application, the above-mentioned gimbal device can be connected to electronic devices via wireless fidelity, Bluetooth or physical connection.

[0040] Optionally, the gimbal device can acquire initial motion information through its built-in IMU (Inertial Measurement Unit) and Gyroscope sensor.

[0041] Optionally, in this embodiment of the application, the gimbal device can acquire first image information through the camera in the gimbal device.

[0042] Optionally, in this embodiment, the gimbal device can be any of the following: a single-axis gimbal device, a dual-axis gimbal device, a three-axis gimbal device, etc. The specific type can be determined based on actual usage requirements, and this embodiment does not impose any limitations.

[0043] Optionally, in this embodiment, the first motion information may include at least one of the following: jitter information, attitude angle information, acceleration motion information, angular velocity motion information, etc. The specific information can be determined according to actual usage requirements, and this embodiment does not impose any limitations.

[0044] Optionally, in this embodiment, the first image information may include at least one of the following: image brightness information, image white balance information, image resolution information, image exposure time information, foreground image information, background image information, etc. The specific information can be determined according to actual usage requirements, and this embodiment does not impose any limitations.

[0045] It should be noted that the foreground mentioned above refers to the scenery or objects located between the lens and the subject, close to the camera. It is in the foreground of the frame and can be elements such as people, flowers, railings, and window frames, but it is not necessarily the core subject of the shot.

[0046] The aforementioned background refers to all the scenery elements behind the subject in the shooting frame. It can be the environment, buildings, sky, walls, or people, and is generally not the main subject of the image. The background can be clearly presented to explain the environment, era, and atmosphere of the shooting scene, or its presence can be weakened by blurring to highlight the subject in the center of the image.

[0047] Optionally, in this embodiment, the camera can be any of the following: a wide-angle camera, an ultra-wide-angle camera, a macro camera, etc. The specific type can be determined based on actual usage requirements, and this embodiment does not impose any limitations.

[0048] Optionally, in this embodiment of the application, the electronic device can acquire second motion information through the IMU and Gyro sensor in the camera module.

[0049] It should be noted that the data acquisition accuracy of the IMU and Gyro sensors in the aforementioned camera module is lower than that of the IMU and Gyro sensors in the gimbal device.

[0050] Optionally, in this embodiment, the second motion information may include at least one of the following: jitter information, attitude angle information, acceleration motion information, angular velocity motion information, etc. The specific information can be determined according to actual usage requirements, and this embodiment does not impose any limitations.

[0051] Optionally, in this embodiment, the second image information may include at least one of the following: image brightness information, image white balance information, image resolution information, image exposure time information, foreground image information, background image information, etc. The specific information can be determined according to actual usage requirements, and this embodiment does not impose any limitations.

[0052] Step 202: The electronic device captures images or videos based on the first and second information.

[0053] It should be noted that the specific implementation process of step 202 above can be found in the following embodiments, and will not be repeated here to avoid repetition.

[0054] In the shooting method provided in this application embodiment, first information collected by a gimbal device and second information collected by an electronic device are acquired. The first information includes at least one of the following: first motion information and first image information; the second information includes at least one of the following: second motion information and second image information. An image or video is captured based on the first and second information. In this solution, by combining the first motion information collected by an external gimbal device and the second motion information collected by the electronic device, the image or video captured by the electronic device is processed, breaking through the stabilization accuracy bottleneck of single motion information from the electronic device, thereby improving the stabilization effect. Furthermore, the first image information collected by the gimbal device can acquire image information other than the second image information, thereby improving the image enhancement effect of the electronic device.

[0055] In some embodiments of this application, the first information mentioned above includes first motion information, and the second information mentioned above includes second motion information.

[0056] For example, combined Figure 1 ,like Figure 2 As shown, step 202 above can be specifically implemented through step 202a below.

[0057] Step 202a: The electronic device performs image stabilization processing based on the motion state information determined by the first motion information and the second motion information to obtain an image or video.

[0058] In some embodiments of this application, the above-mentioned image stabilization process includes at least one of the following: optical image stabilization, electronic image stabilization, and rolling shutter correction.

[0059] Optionally, in this embodiment of the application, the electronic device can obtain combined motion information in the same coordinate system, i.e., the above-mentioned motion state information, based on the first motion information and the second motion information, and perform anti-shake processing based on the combined motion information to obtain an image or video.

[0060] It is understandable that the above-mentioned combined motion information can characterize the global motion trajectory and motion trend of the electronic device during the shooting process.

[0061] Optionally, in this embodiment of the application, the electronic device can fuse the first motion information and the second motion information by extended Kalman filtering or factor graph optimization, and map the fused data to the same coordinate system to obtain combined motion information.

[0062] For example, such as Figure 3 As shown, after the electronic device obtains motion data collected by the external gimbal and the motion data collected by the electronic device, it can use a fusion algorithm to obtain fused data and map the fused data to the same coordinate system to obtain combined motion information.

[0063] In this embodiment, by combining the first motion information collected by the external gimbal device and the second motion information collected by the electronic device, the image collected by the camera module in the electronic device is processed, which breaks through the stabilization accuracy bottleneck of the single motion information of the electronic device and improves the stabilization effect of the electronic device.

[0064] In some embodiments of this application, combined with Figure 2 ,like Figure 4 As shown, step 202a can be implemented through steps 202a1 and 202a2 as described below.

[0065] Step 202a1: The electronic device determines the shooting scene based on motion state information and determines the image stabilization strategy based on the shooting scene.

[0066] Optionally, in this embodiment, the shooting scenario can be any of the following: walking shooting scenario, vehicle-mounted shooting scenario, fixed-position shooting scenario, handheld slow-pan shooting scenario, telephoto shooting scenario, etc. The specific scenario can be determined according to actual usage requirements, and this embodiment does not impose any limitations.

[0067] Optionally, in this embodiment of the application, the electronic device can distinguish between walking, vehicle-mounted, fixed-position, handheld slow-motion, and telephoto scenes based on the motion frequency, amplitude, direction, and shooting status of the electronic device, such as focal length, contained in the combined motion information.

[0068] For example, when the subject is in a slow and uniform motion, such as when a person is walking slowly, the shooting scene is determined to be a tracking shot scene. The camera moves smoothly and synchronously with the person, and the lens keeps the subject in a relatively fixed position in the frame. By utilizing the flowing changes of the foreground and background of the environment, the complete process of the person's leisurely walk is reproduced, focusing on recording the sense of integration between the person's actions and expressions and the surrounding environment.

[0069] For example, when the subject is moving at high speed in a straight line, such as cycling or sprinting, and the shooting scene is determined to be a fixed-position snapshot scene, the camera position remains stationary, and the lens tracks the moving subject that is moving quickly across the screen. The background quickly sweeps by to create a strong sense of speed, focusing on capturing the peak moment of the subject's movement, which is suitable for expressing intense and impactful dynamic effects.

[0070] For example, when the subject is moving back and forth in place, such as a person dancing in place or waving their arms, the shooting scene is determined to be a circular shooting scene. The camera moves in a circle around the stationary moving subject, and the subject's position remains basically unchanged. The subject's posture is shown from multiple angles by constantly changing background angles. The image is enriched by changes in light and shadow and environment, highlighting the details of the person's body movements.

[0071] For example, when the subject moves from far to near towards the camera, such as a person walking towards the camera, the shooting scene is determined to be a push-in shot. The camera position remains still or slowly moves forward to meet the subject approaching the camera. The foreground is continuously covered by the subject, and the background gradually shrinks backward, which strengthens the sense of pressure and immersion of the subject constantly approaching, and guides the audience's attention to the gradually growing figure.

[0072] For example, when the subject is moving irregularly and unpredictably, such as children running and playing freely, and the shooting scene is determined to be a handheld mobile shooting scene, the camera is handheld and flexibly follows the subject's irregular changes in movement direction, without deliberately pursuing absolute stability of the image, but rather matching the subject's casual movement rhythm, preserving the real dynamic feeling of the scene, and fully recording unpredictable natural moments.

[0073] Optionally, in this embodiment of the application, the electronic device can adjust the image stabilization strategy corresponding to the shooting scene according to the shooting scene.

[0074] Optionally, in this embodiment of the application, the above-mentioned image stabilization strategy may include an image stabilization method and image stabilization parameters corresponding to the image stabilization method.

[0075] Optionally, in the embodiments of this application, the above-mentioned image stabilization parameters include at least one of the following: optical image stabilization parameters, electronic image stabilization parameters, or rolling shutter correction parameters.

[0076] Optionally, in this embodiment of the application, the combined motion information can be input into the optical image stabilization algorithm to determine the aforementioned optical image stabilization parameters.

[0077] For example, electronic devices can convert the high-frequency jitter component information decomposed from combined motion information into optical image stabilization drive commands, thereby more accurately controlling the camera module or sensor to perform reverse displacement compensation. Moreover, since the estimation accuracy of combined motion information far exceeds the estimation accuracy of single motion information from the electronic device, the effective compensation range of optical image stabilization is expanded, enabling compensation for larger amplitude jitter.

[0078] Optionally, in this embodiment of the application, the combined motion information can be input into the electronic image stabilization algorithm to determine the above-mentioned electronic image stabilization parameters.

[0079] For example, electronic devices can obtain high-frequency jitter components from combined motion information. These high-frequency jitter components can provide electronic image stabilization algorithms with more reliable global motion vectors, enabling them to more accurately distinguish between background jitter and subject movement, such as a person walking. This allows them to make better decisions when cropping the image, reducing image cropping while maintaining stability and preserving a wider field of view.

[0080] Optionally, in this embodiment of the application, the combined motion information can be input into the rolling shutter correction algorithm to determine the above-mentioned rolling shutter correction image stabilization parameters.

[0081] For example, electronic devices can use the high-frequency jitter components decomposed from motion state information to perform pixel-level correction of the "jelly effect" caused by line-by-line exposure of a camera module or sensor. The exposure time of each line corresponds to a different motion state, and the precise motion compensation amount for each line can be obtained through interpolation, which is particularly effective in fast translation or vibration scenarios.

[0082] For example, the process of obtaining the precise motion compensation amount for each row through the above interpolation can be described as follows: Because the frequency of the image sensor's line-by-line exposure may differ from the frequency of motion data acquisition corresponding to the motion state information, the exposure time of the image sensor and the motion data acquisition time are not perfectly matched. The motion state information corresponds to times t1 to t... n The motion data can be obtained, but the first exposure time T1 of the image sensor may be between two consecutive sampling times t1 and t2. In this case, the motion data of t1 and t2 can be averaged or weighted and fitted, and the fitted motion data can be used as the motion data corresponding to time T1. This process can be repeated to improve the prediction accuracy of motion compensation.

[0083] Optionally, in this embodiment of the application, the electronic device can determine the image stabilization strategy corresponding to the shooting scene based on the correspondence between the shooting scene and the image stabilization parameters, and determine the image stabilization parameters corresponding to the shooting scene based on the preset image stabilization processing algorithm.

[0084] For example, in walking shooting scenarios, the electronic image stabilization cropping range is dynamically adjusted to ensure image stability; in vehicle shooting scenarios and telephoto shooting scenarios, optical image stabilization shift compensation is enhanced to cope with high-frequency vibrations; and in fixed-position shooting scenarios, the cropping of the image is minimized to retain the best image quality.

[0085] For example, in shooting scenarios where a fixed camera is used to capture fast-moving objects, if strong electronic image stabilization is not enabled, electronic cropping will result in a loss of field of view. Excessive image stabilization compensation should be turned off, and optical image stabilization compensation should be used to deal with high-frequency vibrations, ensuring a complete field of view and preventing the image stabilization algorithm from distorting the outline of the fast-moving subject, so as to clearly capture the moment of the object's high-speed movement.

[0086] For example, in a performance recording scene where the camera is handheld and pans around, a medium-strength electronic image stabilization combined with a small-amplitude gimbal is used. The image stabilization strategy focuses on correcting the offset shaking caused by hand rotation, without forcibly locking the image, allowing for a reasonable sense of camera movement. This eliminates the violent shaking caused by hand instability, while retaining the background rotation changes brought about by panning, thus balancing image stability and the dynamic atmosphere of the shot.

[0087] Step 202a2: The electronic device performs image stabilization processing based on the image stabilization strategy to obtain an image or video.

[0088] It should be noted that the specific image stabilization process performed by the electronic device can be found in the descriptions in the relevant technologies, and will not be repeated here to avoid repetition.

[0089] In this embodiment, the electronic device can determine the shooting scene based on motion state information and adjust the anti-shake strategy corresponding to the shooting scene, thereby enabling different anti-shake parameters to be used for different shooting scenes and improving the flexibility and accuracy of anti-shake.

[0090] In some embodiments of this application, the first information mentioned above includes first image information, and the second information mentioned above includes second image information.

[0091] For example, combined Figure 1 ,like Figure 5 As shown, step 202 above can be specifically implemented through step 301 below.

[0092] Step 301: The electronic device performs image processing on the second image information based on the first image information to obtain an image or video.

[0093] In some embodiments of this application, the shooting parameters corresponding to the second image information are different from the shooting parameters corresponding to the first image information.

[0094] It is understood that the aforementioned first image information can be image information captured by the gimbal device at different shooting angles.

[0095] Optionally, in this embodiment of the application, the different shooting angles can be determined according to the angle adjustment command of the electronic device.

[0096] For example, the aforementioned angle adjustment command can be a preset command in the electronic device or a command input by the user. The specific command can be determined according to actual usage needs, and this application embodiment does not impose any limitations.

[0097] For example, such as Figure 6 As shown, a gimbal device connected to an electronic device can capture different images based on different angle adjustment commands. For example, if the camera module in the electronic device captures a person running, the gimbal device can capture an image containing the sun or an image of another person running, depending on the angle adjustment command.

[0098] Optionally, in the embodiments of this application, the first image information may include at least one of the following: global illumination information, shooting background information contained in each captured image, and shooting foreground information contained in each captured image.

[0099] For example, the above step 301 will be explained in detail below through a specific scenario.

[0100] Electronic devices can use at least one captured image from a gimbal device to identify background and human information in the shooting environment that is not captured by the camera module of the current electronic device, construct a background model and human motion trajectory, and perform video blurring enhancement and pedestrian removal preprocessing.

[0101] The aforementioned video blurring enhancement is achieved by obtaining a complete background model from at least one captured image using a gimbal device. During video blurring, there is no need to perform depth estimation frame by frame; the background model can be directly used for precise image matting, significantly reducing computational latency and resulting in more accurate and natural blurring edges.

[0102] The aforementioned pedestrian removal enhancement is achieved by using at least one captured image from multiple angles via a PTZ device to detect pedestrians about to enter the frame in advance and mark their movement trajectory. As a result, when a pedestrian enters the frame captured by the camera module in the electronic device, the pedestrian is immediately located and removed, resulting in a faster response and cleaner effect.

[0103] Optionally, in this embodiment of the application, the electronic device can enhance the shooting effect based on image information from different shooting angles collected by the gimbal device.

[0104] For example, the electronic device can perform global illumination guidance based on the ambient lighting information collected by the gimbal device. After the gimbal device captures images from multiple angles, the electronic device can synthesize a global illumination map to obtain the overall brightness distribution of the scene. At this time, the local illumination map captured by the electronic device can refer to the global illumination information to avoid local overexposure or underexposure and achieve a more natural tonal transition.

[0105] It is understandable that the image itself can be converted to retain only the brightness information. For example, the AE information of each pixel can be represented by 0-255 to represent the darkest to the brightest, resulting in a global illumination map. Based on the global illumination map, it can be known which area is bright. For example, if the pixel AE is 255, when the camera in the electronic device moves to the corresponding area to take a picture, the AE strategy can be adjusted in advance to avoid overexposure or underexposure of the image.

[0106] Optionally, in this embodiment of the application, the electronic device can eliminate reflective areas based on the ambient light information collected by the gimbal device. When there are reflective areas in the captured image of the electronic device, the electronic device can replace the reflective areas with corresponding pixels that are not reflective in other angle images collected by the gimbal device to eliminate reflective interference.

[0107] In this embodiment, by combining the first image information acquired by the external gimbal device, image information different from the second image information acquired by the electronic device can be obtained. This allows the image processing of the electronic device to be upgraded from blind calculation to pre-sensory enhancement processing, thereby improving the image quality of the images acquired by the electronic device. Furthermore, by acquiring image information outside the frame captured by the electronic device in advance, the electronic device can be assisted in image processing, reducing real-time calculation latency and improving image quality.

[0108] In some embodiments of this application, the first information mentioned above includes first image information, and the second information mentioned above includes second image information.

[0109] For example, prior to step 202 above, the shooting method provided in this application embodiment further includes steps 401 and 402 as described below.

[0110] Step 401: The electronic device determines the image composition method based on the first image information and the second image information.

[0111] Optionally, in this embodiment of the application, the electronic device can optimize the current composition mode of the electronic device based on the content corresponding to the first image information, such as a better subject position, a better light direction, and a cleaner background, so as to determine the image composition mode.

[0112] Step 402: The electronic device displays composition guidance icons based on the image composition method.

[0113] Optionally, in this embodiment, the composition guidance mark can be any of the following: symbol mark, text mark, etc., used to guide the user to adjust the composition. The specific design can be determined according to actual usage needs, and this embodiment does not impose any limitations.

[0114] For example, electronic devices can guide users to the "best view" direction with arrows or prompts to help users achieve a better composition.

[0115] Optionally, in this embodiment of the application, the electronic device may display composition guidance marks in the shooting preview screen.

[0116] In this embodiment, the electronic device can determine the composition method based on the global environmental information collected by the gimbal device, guide the user to compose the image, and make the composition of the processed image or video more reasonable.

[0117] In some embodiments of this application, before step 202 above, the shooting method provided in the embodiments of this application further includes the following step 501.

[0118] Step 501: The electronic device synchronizes the timestamps corresponding to the first and second information.

[0119] Optionally, in this embodiment of the application, the electronic device can synchronize the motion data and image data collected by the gimbal device, and synchronize the motion data and image data collected by the electronic device in real time, thereby realizing the synchronization of the first information and the second information.

[0120] For example, the electronic device can perform clock synchronization calibration during the initialization phase to ensure that the motion data collected by the sensors in the gimbal device is strictly aligned in time with the image data of each frame collected by the camera module of the electronic device.

[0121] For example, the synchronization of motion data and image data is generally based on a unified timestamp. When the camera of the electronic device outputs a frame of image, it adds a precise timestamp. Sensors such as IMU synchronously collect motion data such as angular velocity and acceleration and record the corresponding timestamp. After reading the two types of data, the electronic device aligns and matches them according to the timestamp, compensates for the hardware time offset and transmission delay between the sensor and the camera, and binds the image at the same moment and the motion posture data at that moment one by one, eliminating the time difference caused by the inconsistency of the sampling frequency of the two, and finally realizing the correspondence between the image and the motion state in the time dimension.

[0122] In this embodiment, the electronic device ensures the consistency of motion data and image data by synchronizing the data acquisition timestamps between the gimbal device and the camera module.

[0123] The following describes the shooting method provided in the embodiments of this application in detail through different shooting scenarios. Taking the image stabilization shooting scenario as an example, the shooting method provided in the embodiments of this application can be implemented through the following steps S1 to S4.

[0124] S1: The electronic device synchronizes the clocks of the sensors in the gimbal device with those of the sensors in the electronic device.

[0125] S2: Electronic devices perform multi-source motion data fusion and modeling.

[0126] For example, the electronic device receives higher-precision motion data collected by the gimbal device and fuses it with motion data collected by the IMU and Gyro sensors built into the electronic device.

[0127] For example, electronic devices can use methods such as extended Kalman filtering or factor graph optimization to fuse motion data collected by electronic devices and gimbal devices into the same coordinate system, thereby constructing combined motion information with a precision far exceeding that of single motion information from electronic devices. This combined motion information may include at least one of the following: global motion trajectory, jitter frequency distribution, and motion trend prediction.

[0128] S3: Enhanced hybrid image stabilization based on combined motion information for electronic devices.

[0129] For example, the electronic device performs image stabilization based on combined motion information, and the image stabilization may include at least one of the following:

[0130] Optical image stabilization (OIS) converts the jitter components decomposed from the motion model into OIS drive commands, enabling more precise control of the camera module or sensor of electronic devices to perform reverse displacement compensation. The estimation accuracy of combined motion information far exceeds that of single motion information from electronic devices, thus expanding the effective compensation range of optical image stabilization and allowing for the compensation of greater amplitude jitter.

[0131] Optionally, in the embodiments of this application, the jitter component can be a high-frequency jitter component and / or a low-frequency jitter component.

[0132] Electronic image stabilization (EIS) combines motion information to provide more reliable global motion vectors for EIS algorithms, enabling more accurate differentiation between background shaking and subject movement (such as people walking). This allows for better decision-making when cropping the image, reducing cropping while maintaining stability and preserving a wider field of view.

[0133] Rolling shutter correction utilizes combined motion information to perform pixel-level correction of the "jelly effect" caused by line-by-line exposure of electronic device sensors. The exposure time of each line corresponds to a different motion state, and the precise motion compensation amount for each line is obtained through interpolation, which is particularly effective in fast-moving or vibrating scenes.

[0134] S4: Electronic devices perform intelligent scene recognition and dynamic adjustment of anti-shake parameters.

[0135] For example, electronic devices can intelligently identify the current shooting scene based on combined motion information. By analyzing the motion frequency, amplitude, direction, and mobile phone shooting status (such as focal length) in the combined motion information, they can distinguish specific shooting scenes such as walking, vehicle-mounted, fixed camera position, and handheld slow panning, and dynamically adjust the image stabilization parameters. For example, in walking scenes, the electronic image stabilization cropping range is dynamically adjusted to ensure image stability; in vehicle-mounted and telephoto shooting scenes, optical image stabilization compensation is enhanced to cope with high-frequency vibrations; in fixed camera position scenes, cropping is minimized to preserve image quality.

[0136] In this embodiment, the electronic device constructs high-precision combined motion information by fusing motion data from the gimbal device and the electronic device, breaking through the image stabilization accuracy bottleneck of single motion information of the electronic device. This achieves expanded optical image stabilization compensation range, reduced electronic image stabilization cropping loss, precise rolling shutter correction, and scene adaptive strategy adjustment, significantly improving the image or video quality captured by the electronic device in various motion scenarios.

[0137] Taking gimbal-assisted image enhancement as an example, the shooting method provided in this application embodiment can be specifically implemented through the following steps S10 to S14.

[0138] S10: The electronic device synchronizes the clocks of the sensors in the gimbal device with those of the sensors in the electronic device.

[0139] S11: Electronic equipment controls the pan-tilt device to acquire multi-angle image information.

[0140] For example, during the shooting process of an electronic device, the gimbal device can rotate to outside the shooting angle of the electronic device according to a preset strategy or user command sent by the electronic device to acquire images outside the field of view of the electronic device, and transmit them back to the electronic device in real time via a dedicated data link. The gimbal device can periodically acquire images from multiple angles to obtain panoramic information.

[0141] S12: The electronic device performs background and person pre-recognition processing.

[0142] For example, the electronic device uses the image captured by the gimbal device to identify background and people information in the shooting environment that are not in the current electronic device's shooting preview in advance, constructs a background model and people's movement trajectory, and performs video blur enhancement and pedestrian removal enhancement preprocessing.

[0143] The aforementioned video blurring enhancement is achieved by obtaining a complete background model through at least one captured image pre-captured by the gimbal device. During video blurring, there is no need to perform depth estimation frame by frame. The background model can be directly used for accurate image matting, which greatly reduces computational latency and makes the blurred edges more precise and natural.

[0144] The aforementioned pedestrian removal enhancement is achieved by using a gimbal device to pre-capture multiple angles and frames, which can detect pedestrians about to enter the frame in advance and mark their movement trajectories in multiple frames. When a pedestrian enters the shooting range of the electronic device, it is immediately located and removed, resulting in a faster response and cleaner effect.

[0145] S13: Enhanced shooting effects using electronic devices.

[0146] For example, the electronic device enhances its shooting effect based on the global environmental information collected by the gimbal:

[0147] Optionally, the electronic device can use the ambient lighting information collected by the gimbal device for global illumination guidance. After the gimbal captures images from multiple angles, the electronic device synthesizes a global illumination map to obtain the overall brightness distribution of the scene. The local illumination map captured by the electronic device can refer to the global illumination information to avoid local overexposure or underexposure and achieve a more natural tonal transition.

[0148] Optionally, the electronic device can eliminate reflective areas based on the ambient light information collected by the gimbal device. When there are reflective areas in the image captured by the electronic device, the electronic device can determine the corresponding non-reflective pixels from other angle images collected by the gimbal device, replace the reflective areas, and eliminate reflective interference.

[0149] S14: Electronic devices perform intelligent framing.

[0150] For example, the electronic device combines the panoramic information pre-collected by the gimbal device to determine the composition of the current shooting scene, such as a better subject position, a better light direction, and a cleaner background, and guides the user to the "best picture" direction with arrows or prompt boxes to help the user obtain a better composition.

[0151] In this embodiment, the electronic device acquires off-screen image information in advance through multi-angle pre-acquisition of the gimbal device, which upgrades the image processing of the mobile phone camera from real-time blind calculation to pre-sensory enhancement, greatly reducing the calculation latency and error rate of blurring, removing passers-by, etc., improving the uniformity of overall image brightness and the effect of reflection elimination, and providing users with intelligent framing guidance, significantly improving the image or video quality captured by the electronic device.

[0152] It should be noted that the above-described method embodiments, or the various possible implementations of the method embodiments, can be executed individually, or, provided there are no contradictions, they can be combined with each other. The specific implementation can be determined according to actual usage requirements, and this application embodiment does not impose any restrictions on this.

[0153] It should be noted that the shooting method provided in this application embodiment can be executed by a shooting device. This application embodiment uses a shooting device executing the shooting method as an example to illustrate the shooting device provided in this application embodiment.

[0154] Figure 7 A schematic diagram of a possible structure of the shooting device involved in an embodiment of this application is shown. The shooting device is applied to an electronic device, which is mounted on a gimbal device, and the electronic device and the gimbal device are communicatively connected, such as... Figure 7 As shown, the imaging device 70 may include an acquisition module 71 and a processing module 72.

[0155] The acquisition module 71 is used to acquire first information collected by the pan-tilt device and second information collected by the electronic device. The first information includes at least one of the following: first motion information and first image information. The second information includes at least one of the following: second motion information and second image information. The processing module 72 is used to capture images or videos based on the first and second information acquired by the acquisition module.

[0156] In one possible implementation, the first information includes first motion information, and the second information includes second motion information. The processing module 72 is specifically used to perform image stabilization processing based on motion state information determined by the first and second motion information to obtain an image or video; the image stabilization processing includes at least one of the following: optical image stabilization, electronic image stabilization, and rolling shutter correction.

[0157] In one possible implementation, the processing module 72 is further configured to determine the shooting scene based on the motion state information before performing the image stabilization processing based on the motion state information determined by the first motion information and the second motion information, and to determine the image stabilization strategy based on the shooting scene.

[0158] In one possible implementation, the first information includes first image information, the second information includes second image information, and the processing module 72 is specifically used to perform image processing on the second image information based on the first image information to obtain an image or video; wherein the shooting parameters corresponding to the second image information are different from the shooting parameters corresponding to the first image information.

[0159] In one possible implementation, the first information includes first image information, the second information includes second image information, and the shooting device further includes a display module; the processing module 72 is specifically used to determine the image composition method based on the first image information and the second image information before capturing an image or video based on the first information and the second information; the display module is used to display composition guidance marks based on the image composition method.

[0160] In one possible implementation, the processing module 72 is further configured to synchronize the timestamps corresponding to the first information and the second information before capturing an image or video based on the first information and the second information.

[0161] In the shooting device provided in this application embodiment, by combining the first motion information collected by the external gimbal device and the second motion information collected by the electronic device, the image or video captured by the shooting device is processed, which breaks through the stabilization accuracy bottleneck of the single motion information of the shooting device, thereby improving the stabilization effect; and the first image information collected by the gimbal device can obtain image information other than the second image information, thereby improving the image enhancement effect of the shooting device.

[0162] The shooting device in this application embodiment can be an electronic device or a component of an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, handheld shooting device, aircraft, vehicle-mounted electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. The embodiments of this application do not specifically limit the scope.

[0163] The shooting device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems, and this application embodiment does not specifically limit it.

[0164] The imaging device provided in this application embodiment can realize all the processes implemented in the above method embodiments, and will not be described again here to avoid repetition.

[0165] Optionally, such as Figure 8 As shown, this application embodiment also provides an electronic device 80, including a processor 81 and a memory 82. The memory 82 stores a program or instructions that can run on the processor 81. When the program or instructions are executed by the processor 81, they implement the various steps of the above-described image processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0166] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0167] Figure 9 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0168] The electronic device 900 includes, but is not limited to, components such as: radio frequency unit 901, network module 902, audio output unit 903, input unit 904, sensor 905, display unit 906, user input unit 907, interface unit 908, memory 909, and processor 910.

[0169] Those skilled in the art will understand that the electronic device 900 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 910 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 9 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0170] The processor 910 is used to acquire first information collected by the gimbal device and second information collected by the electronic device. The first information includes at least one of the following: first motion information and first image information. The second information includes at least one of the following: second motion information and second image information. The processor 910 is used to capture images or videos based on the first information and the second information.

[0171] Optionally, in this embodiment of the application, the first information includes first motion information, and the second information includes second motion information; the processor 910 is specifically used to perform image stabilization processing based on the motion state information determined by the first motion information and the second motion information to obtain an image or video; the image stabilization processing includes at least one of the following: optical image stabilization, electronic image stabilization, and rolling shutter correction.

[0172] Optionally, in this embodiment of the application, the processor 910 is specifically used to determine the shooting scene based on the motion state information before performing image stabilization processing based on the motion state information determined based on the first motion information and the second motion information, and to determine the image stabilization strategy based on the shooting scene.

[0173] Optionally, in this embodiment of the application, the first information includes first image information, and the second information includes second image information. The processor 910 is specifically configured to perform image processing on the second image information based on the first image information to obtain an image or video; wherein the shooting parameters corresponding to the second image information are different from the shooting parameters corresponding to the first image information.

[0174] Optionally, in this embodiment, the first information includes first image information, and the second information includes second image information; the processor 910 is further configured to determine an image composition method based on the first image information and the second image information before capturing an image or video based on the first information and the second information. The display unit 906 is configured to display a composition guidance mark based on the image composition method.

[0175] Optionally, in this embodiment of the application, the processor 110 is further configured to synchronize the timestamps corresponding to the first information and the second information before capturing an image or video based on the first information and the second information.

[0176] In the electronic device provided in this application embodiment, by combining the first motion information collected by the external gimbal device and the second motion information collected by the electronic device, the image or video captured by the electronic device is processed, breaking through the stabilization accuracy bottleneck of the single motion information of the electronic device, thereby improving the stabilization effect; and by using the first image information collected by the gimbal device to obtain image information other than the second image information, thereby improving the image enhancement effect of the electronic device.

[0177] The electronic device provided in this application embodiment can implement the various processes implemented in the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0178] For details on the beneficial effects of the various implementation methods in this embodiment, please refer to the beneficial effects of the corresponding implementation methods in the above method embodiments. To avoid repetition, these will not be repeated here.

[0179] It should be understood that, in this embodiment, the input unit 904 may include a graphics processing unit (GPU) 9041 and a microphone 9042. The GPU 9041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 906 may include a display panel 9061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 907 includes at least one of a touch panel 9071 and other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 may include a touch detection device and a touch controller. Other input devices 9072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0180] The memory 909 can be used to store software programs and various data. The memory 909 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 909 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 909 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0181] Processor 910 may include one or more processing units; optionally, processor 910 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 910.

[0182] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0183] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0184] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0185] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0186] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above method embodiments and achieve the same technical effects. To avoid repetition, it will not be described again here.

[0187] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0188] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0189] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A shooting method, characterized in that, Performed by an electronic device mounted on a gimbal, the electronic device being communicatively connected to the gimbal, the method includes: The first information collected by the gimbal device and the second information collected by the electronic device are acquired, wherein the first information includes at least one of the following: first motion information and first image information, and the second information includes at least one of the following: second motion information and second image information; Images or videos are captured based on the first and second information.

2. The method according to claim 1, characterized in that, The first information includes first motion information, the second information includes second motion information, and the process of capturing images or videos based on the first information and the second information includes: Based on the motion state information determined by the first motion information and the second motion information, image stabilization processing is performed to obtain the image or video; The image stabilization process includes at least one of the following: optical image stabilization, electronic image stabilization, and rolling shutter correction.

3. The method according to claim 2, characterized in that, Before performing image stabilization based on the motion state information determined by the first motion information and the second motion information, the method further includes: Based on the motion state information, the shooting scene is determined, and a stabilization strategy is determined based on the shooting scene.

4. The method according to claim 1, characterized in that, The first information includes first image information, and the second information includes second image information; The images or videos captured based on the first information and the second information include: Based on the first image information, the second image information is processed to obtain the image or video; The shooting parameters corresponding to the second image information are different from those corresponding to the first image information.

5. The method according to claim 1, characterized in that, The first information includes first image information, and the second information includes second image information. Before capturing an image or video based on the first information and the second information, the method further includes: The image composition method is determined based on the first image information and the second image information; Based on the image composition method, display composition guidance icons.

6. The method according to any one of claims 1-5, characterized in that, Before capturing images or videos based on the first information and the second information, the method further includes: Synchronize the timestamps corresponding to the first information and the second information.

7. A shooting device, characterized in that, Applied to an electronic device, the electronic device is mounted on a gimbal device and is communicatively connected to the gimbal device; the shooting device includes: an acquisition module and a processing module. The acquisition module is used to acquire first information collected by the gimbal device and second information collected by the electronic device, wherein the first information includes at least one of the following: first motion information and first image information, and the second information includes at least one of the following: second motion information and second image information; The processing module is used to capture images or videos based on the first information and the second information obtained by the acquisition module.

8. The apparatus according to claim 7, characterized in that, The first information includes first motion information, and the second information includes second motion information; The processing module is specifically used for: Based on the motion state information determined by the first motion information and the second motion information, image stabilization processing is performed to obtain the image or video; The image stabilization process includes at least one of the following: optical image stabilization, electronic image stabilization, and rolling shutter correction.

9. The apparatus according to claim 8, characterized in that, The processing module is further configured to: Before performing image stabilization processing on the motion state information determined based on the first motion information and the second motion information, the shooting scene is determined based on the motion state information, and an image stabilization strategy is determined based on the shooting scene.

10. The apparatus according to claim 7, characterized in that, The first information includes first image information, and the second information includes second image information; The processing module is specifically used for: Based on the first image information, the second image information is processed to obtain the image or video; The shooting parameters corresponding to the second image information are different from those corresponding to the first image information.

11. The apparatus according to claim 7, characterized in that, The first information includes first image information, the second information includes second image information, and the device further includes: a display module; The processing module is specifically used for: Before capturing images or videos based on the first information and the second information, the image composition method is determined based on the first image information and the second image information; The display module is used to display composition guidance icons based on the image composition method.

12. The apparatus according to any one of claims 7-11, characterized in that, The processing module is further configured to: Before capturing images or videos based on the first information and the second information, the timestamps corresponding to the first information and the second information are synchronized.

13. An electronic device, characterized in that, It includes a processor and a memory, the memory storing programs or instructions that can run on the processor, the programs or instructions being executed by the processor to implement the steps of the shooting method as described in any one of claims 1 to 6.

Images