A shooting method, device, storage medium and program product

Abstract

The present application relates to the field of terminal technology, and discloses a shooting method, device, storage medium and program product. Among them, the shooting method includes: when the photographer selects a specific shooting mode (such as Hitchcock zoom shooting mode) to shoot a video, the electronic device will first identify the subject being shot, and then adjust the zoom ratio of the camera based on the size change of the identified subject. For example, when it is detected that the change in the size of the subject exceeds a preset amplitude threshold, the camera will be triggered to adjust the zoom ratio, so that the camera continues to shoot the video through the adjusted zoom ratio, so that the size of the subject in the captured video screen remains basically unchanged. In this way, when the photographer moves the camera, the electronic device can automatically adjust the zoom ratio of the camera, so that the captured video presents a visual effect that the size of the subject remains basically unchanged, and the background is far away from or close to the subject, thereby improving the overall visual effect of the video.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a shooting method, device, storage medium and program product. Background Art

[0002] With the development of photography technology, some photographers have proposed a Hitchcock zoom shooting method that can be used to enhance the visual impact of videos. Among them, if the Hitchcock zoom method is used for video shooting, the photographer needs to control the camera to move closer to or farther away from the subject while manually adjusting the camera focus during the shooting process, so that the video presents a visual effect in which the size of the subject remains basically unchanged and the background moves away from or closer to the subject.

[0003] However, usually only photographers with strong coordination ability and rich shooting experience can accurately adjust the camera zoom ratio while moving the camera, so that the background size in the captured video screen changes while the size of the subject remains basically unchanged. However, many photographers with poor coordination ability or little shooting experience find it difficult to coordinate the hand movements to adjust the camera zoom ratio while moving the camera, resulting in obvious changes in the size of the captured subject, which reduces the overall visual effect of the video. Summary of the invention

[0004] To solve the above problems, the present application provides a shooting method, device, storage medium and program product.

[0005] In a first aspect, the present application provides a shooting method, which is applied to an electronic device, and the method includes: detecting a first shooting instruction, wherein the first shooting instruction is used to instruct a first camera of the electronic device to enter a first shooting mode, and in the first shooting mode, a change in the size of a subject captured by the first camera during movement satisfies a shooting condition; at a first shooting moment, the subject captured by the first camera has a first subject size, and the first camera has a first zoom ratio; at a second shooting moment, it is detected that the subject captured by the first camera has a second subject size, and the difference between the first subject size and the second subject size does not meet the shooting condition, and the first zoom ratio is adjusted to a second zoom ratio, wherein the subject captured by the first camera at the second zoom ratio has a third subject size, and the difference between the third subject size and the first subject size meets the shooting condition.

[0006] In the present application, the first shooting instruction may be an operation instruction for the photographer to start a specific shooting mode (such as the Hitchcock zoom shooting mode) mentioned in the present application; the first camera may be the camera for shooting the subject mentioned in the present application; the first shooting mode may be the Hitchcock zoom shooting mode mentioned in the present application; the first shooting moment may be any shooting moment at which the subject size captured is the target subject size mentioned in the present application; the first subject size may be the target subject size mentioned in the present application; the first zoom ratio may be the camera zoom ratio before adjustment mentioned in the present application; the second shooting moment may be the shooting moment after the first shooting moment mentioned in the present application; the second subject size may be the subject size mentioned in the present application, which is captured by the first zoom ratio and is significantly different from the first subject size; the second zoom ratio may be the adjusted zoom ratio mentioned in the present application; the third subject size may be the subject size mentioned in the present application, which is captured by the second zoom ratio and is slightly different from the first subject size.

[0007] In some implementations, the electronic device can respond to the photographer's Hitchcock zoom shooting instruction and control the camera to shoot video using the Hitchcock zoom shooting mode. That is, in the process of the photographer moving the camera, the photographer does not need to manually adjust the camera zoom ratio, but the electronic device controls the change of the camera zoom ratio to achieve zoom shooting. In the process of shooting, the electronic device will first identify the subject to be shot and detect the size of the subject in real time, so as to adjust the zoom ratio of the camera based on the size change of the identified subject. For example, when it is detected that the change between the second subject size and the first subject size of the subject photographed by the camera at the first zoom ratio exceeds a preset amplitude threshold, the camera will be triggered to adjust the zoom ratio, so that the camera continues to shoot video at the adjusted second zoom ratio, so that the background size in the shot video screen changes while the subject size remains basically unchanged.

[0008] In some implementations, since the Hitchcock zoom shooting method requires that the subject size remains substantially unchanged, the electronic device may also use the subject size when the photographer first starts shooting the subject as the first subject size. Thus, in subsequent shooting processes, the first subject size is used as an adjustment reference, so that the subject size remains substantially unchanged throughout the entire shooting process.

[0009] In this way, through the above-mentioned shooting method, when the photographer moves the camera, the electronic device can automatically adjust the zoom ratio of the camera, so that the shot video presents a visual effect that the size of the subject remains unchanged, while the background is far away from or close to the subject.

[0010] In a possible implementation of the first aspect above, the subject size includes at least one of the following: a subject frame size, a proportion of the subject in the picture captured by the first camera; wherein the subject frame size includes the length and width of the subject frame.

[0011] In some implementations, the main body frame may be a rectangular frame surrounding the main body, and the length and width of the main body may be roughly determined based on the length and width of the main body frame.

[0012] In other implementations, the proportion of the subject in the video screen can be determined by the subject mask corresponding to the video screen. Figure 5 As shown, since the data corresponding to the main pixel points (pixel points A to D) in mask 502 is 1, and the data corresponding to the non-main pixel points in mask 502 is 0, it can be determined based on mask 502 that the proportion of the main body in the picture 501 is 4 / 9.

[0013] In this way, the size of the subject can be determined based on the size of the subject frame and the proportion of the subject in the video image, so that the zoom ratio of the camera can be adjusted based on the change in the size of the subject.

[0014] In a possible implementation of the first aspect above, the above-mentioned subject size includes a first subject size and a second subject size, the first subject size includes a first subject frame size and a first occupancy ratio, and the second subject size includes a second subject frame size and a second occupancy ratio; and the shooting conditions include: the absolute value of the difference between the first subject frame size and the second subject frame size is less than a difference threshold; the absolute value of the difference between the first occupancy ratio and the second occupancy ratio is less than a ratio threshold.

[0015] In the present application, the first main body frame size can be the main body frame corresponding to the first shooting moment mentioned in the present application; the second main body frame size can be the main body frame corresponding to the second shooting moment mentioned in the present application; the first occupancy ratio can be the occupancy ratio of the main body in the video screen at the first shooting moment mentioned in the present application; the second occupancy ratio can be the occupancy ratio of the main body in the video screen at the second shooting moment mentioned in the present application.

[0016] In some implementations, when the photographer moves the camera, the electronic device detects the size of the subject in real time and determines whether the change in the size of the subject meets the shooting conditions. For example, the shooting conditions may be: the absolute value of the difference in the size of the subject frame is less than the difference threshold, and the absolute value of the difference in the proportion is less than the proportion threshold. When the electronic device determines that the difference between the first subject size and the second subject size does not meet the above shooting conditions, the electronic device can adjust the zoom ratio of the camera. Among them, the difference threshold and the proportion threshold can be set arbitrarily. For example, the maximum subject frame size error that the video viewer cannot see the change in the subject size can be used as the difference threshold, and the maximum subject occupancy ratio error that the video viewer cannot see the change in the subject size can be used as the proportion threshold. The specific setting of the difference threshold and the proportion threshold is not limited in this application. In this way, the electronic device can detect in real time whether the change in the subject size is too large, so that the camera zoom ratio can be adjusted even when the change in the subject size is large.

[0017] In a possible implementation of the first aspect above, the adjusting the first zoom ratio to the second zoom ratio includes: determining a first reference zoom ratio based on a difference between a first main body frame size and a second main body frame size; determining a second reference zoom ratio based on a relative movement speed of the first camera between a first shooting moment and a second shooting moment; determining a second zoom ratio based on the first reference zoom ratio and the second reference zoom ratio, and adjusting the zoom ratio of the first camera from the first zoom ratio to the second zoom ratio.

[0018] In the present application, the first reference zoom ratio may be the zoom ratio obtained by the proportional integral differential (PID) algorithm mentioned in the present application; the second reference zoom ratio may be the zoom ratio predicted by the Kalman filter model mentioned in the present application.

[0019] In some implementations, when adjusting the zoom ratio of the camera, the electronic device can determine the camera zoom ratio that needs to be adjusted by combining the change parameter of the main frame size, the change parameter of the main body occupancy ratio corresponding to the main body mask, and the relative movement speed of the camera. For example, the main frame size change parameter can be input into an industrial model that determines the control signal value based on the parameter change value to obtain a first reference zoom ratio; the original zoom ratio, the change value of the main body occupancy ratio, and the relative movement speed of the camera are input into an industrial model that predicts a new control signal based on speed, parameter change, and known control signals to obtain a second reference zoom ratio; and then the first reference zoom ratio and the second reference zoom ratio are weighted and summed to obtain the final adjusted zoom ratio.

[0020] In this way, when the photographer moves the camera, the electronic device can automatically adjust the zoom ratio of the camera, so that the captured video presents a visual effect that the size of the subject remains basically unchanged. In addition, since the relative movement speed of the camera and other parameters are referenced when adjusting the zoom ratio, the adjustment of the zoom ratio is smoother and smoother, and there will be no jerky video images, unnatural background changes, or visual focus jumps, thereby improving the smoothness and overall effect of the video.

[0021] In a possible implementation of the first aspect, determining the first reference zoom ratio based on the difference between the first main body frame size and the second main body frame size includes: inputting the difference between the first main body frame size and the second main body frame size into a PID algorithm function to obtain the first reference zoom ratio; wherein the PID algorithm function is constructed in the form of the following formula:

[0022] Where Z is the first reference zoom ratio, e(t) is the difference between the first main frame size and the second main frame size, and K p is the error proportionality coefficient, K p e(t) is the error ratio between the size of the first main frame and the size of the second main frame, K i is the error integral coefficient, t is each shooting moment between the first shooting moment and the second shooting moment, K is the error integral between the size of each main frame corresponding to each shooting moment and the size of the first main frame, d is the error differential coefficient, is the differential error between each main body frame size corresponding to each shooting time and the first main body frame size.

[0023] In some implementations, the size of the first main body frame can be used as the target output parameter in the industrial system; the size difference between the first main body frame and the second main body frame can be used as the difference between the target output parameter and the actual output parameter in the industrial system; and the first reference zoom ratio can be used as the system control signal in the industrial system. Thus, the PID algorithm can be applied to the zoom ratio calculation process of the camera, and the size difference between the first main body frame and the second main body frame can be input into the PID algorithm to obtain the first reference zoom ratio, so that the difference between the main body frame size obtained by the first reference zoom ratio and the first main body frame size is less than the difference threshold.

[0024] In a possible implementation of the first aspect, the second reference zoom ratio is determined based on the relative movement speed of the first camera between the first shooting moment and the second shooting moment, including: determining the relative movement speed of the first camera between the first shooting moment and the second shooting moment based on the change value of the spatial relative distance between the subject and the first camera; generating the second reference zoom ratio through a Kalman filter model based on the first zoom ratio, the difference between the first occupancy ratio and the second occupancy ratio, and the relative movement speed corresponding to the first camera.

[0025] In some implementations, when determining the second reference zoom ratio, the second reference zoom ratio can be obtained through a Kalman filter model. Among them, the Kalman filter model can predict a new control signal based on a known control signal and multiple measurement values ​​(such as sensor data obtained by a sensor) in the presence of noise or error. Therefore, in the present application, the first zoom ratio can be used as a known control signal; multiple parameters such as the first subject occupancy ratio and the second subject occupancy ratio, the relative movement speed of the camera, etc. can be used as multiple measurement values; and the second reference zoom ratio can be used as a new control signal. Therefore, the Kalman filter model can be applied to the zoom ratio calculation process of the camera, and the first zoom ratio and multiple parameters are input into the Kalman filter model to obtain the second reference zoom ratio, so that the process of changing from the first zoom ratio to the second reference zoom ratio is smoother, and there will be no stiff video picture, unnatural background changes, visual focus jumps, etc., which improves the fluency and overall effect of the video.

[0026] In a possible implementation of the first aspect, the determining the relative movement speed of the first camera between the first shooting moment and the second shooting moment based on the change value of the spatial relative distance between the subject and the first camera includes: at the first shooting moment, obtaining the first relative distance between the subject and the first camera; at the second shooting moment, obtaining the second relative distance between the subject and the first camera; and determining the relative movement speed of the first camera between the first shooting moment and the second shooting moment based on the first relative distance and the second relative distance.

[0027] In the present application, the first relative distance may be the relative distance between the subject and the camera at the first shooting moment mentioned in the present application; the second relative distance may be the relative distance between the subject and the camera at the second shooting moment mentioned in the present application.

[0028] In some implementations, the electronic device can determine the relative movement distance of the camera within a certain period of time based on the relative distance between the subject and the camera at different shooting moments, and then determine the relative movement speed of the camera based on the relative movement distance and the shooting time interval.

[0029] In a possible implementation of the first aspect above, the method also includes: at a first shooting moment, obtaining a first subject mask corresponding to a first picture captured by a first camera, wherein each pixel point in the first picture corresponds to each data in the first subject mask one by one, the subject pixel point in the first picture corresponds to the first data in the first subject mask, and the non-subject pixel point in the first picture corresponds to the second data in the first subject mask; based on the number of the first data and the number of data in the first subject mask, determining a first occupancy ratio.

[0030] In the present application, the first picture may be the video picture captured by the camera mentioned in the present application at the first shooting moment; the first subject mask may be the subject mask corresponding to the video picture captured at the first shooting moment mentioned in the present application; the first data may be the data corresponding to the pixel points in the subject mentioned in the present application in the first subject mask; the second data may be the data corresponding to the pixel points in the non-subject mentioned in the present application in the first subject mask.

[0031] In some implementations, the proportion of the subject in the video image can be determined by the subject mask corresponding to the video image. Figure 5 As shown, since the data corresponding to the main pixel points (pixel points A to D) in mask 502 is 1, and the data corresponding to the non-main pixel points in mask 502 is 0, it can be determined based on mask 502 that the proportion of the main body in the picture 501 is 4 / 9.

[0032] In a possible implementation of the first aspect, the electronic device further includes a second camera, and the acquiring the first relative distance between the subject and the first camera at the first shooting moment includes: at the first shooting moment, controlling the second camera to shoot the subject to generate a second picture, wherein at the same shooting moment, the zoom magnification of the second camera is the same as the zoom magnification of the first camera; determining the disparity corresponding to each pixel in the first picture captured by the first camera, wherein the disparity corresponding to each pixel in the first picture is the coordinate position difference between each pixel in the first picture and the corresponding pixel in the second picture; determining the depth information corresponding to each pixel in the first picture based on the disparity corresponding to each pixel in the first picture, the first zoom magnification, the first camera parameter, the second camera parameter, and the spatial distance between the first camera and the second camera; based on the pixel corresponding to the first data in the first subject mask in the first picture, filtering out the depth information corresponding to each pixel of the subject from the depth information corresponding to each pixel in the first picture; determining the first relative distance between the subject and the first camera based on the depth information corresponding to each pixel of the subject.

[0033] In the present application, the second camera may be other cameras mentioned in the present application that assist the first camera in obtaining the relative distance; the second picture may be the picture taken by the second camera mentioned in the present application; the first camera parameters may be the parameter information of the first camera mentioned in the present application; the second camera parameters may be the parameter information of the second camera mentioned in the present application.

[0034] In some implementations, the relative distance between the subject and the camera can be represented by the subject depth information, and the subject depth information can be determined by binocular stereo vision technology. Specifically, the electronic device can determine the depth information corresponding to each pixel in the first picture taken by the first camera by binocular stereo vision technology; then, the electronic device can filter out the depth information of the background in combination with the first data corresponding to the subject in the subject mask corresponding to the first picture, and only retain the depth information corresponding to each pixel of the subject, so as to determine the relative distance between the subject and the first camera based on the depth information corresponding to each pixel of the subject.

[0035] In a possible implementation of the first aspect, the determining the second zoom ratio based on the first reference zoom ratio and the second reference zoom ratio includes: determining the second zoom ratio based on a weighted sum function R; wherein the weighted sum function R is constructed in the form of the following formula:

[0036] Among them, R is the second zoom ratio, R1 is the first reference zoom ratio, w1 is the weight corresponding to the first reference zoom ratio, R2 is the second reference zoom ratio, and w2 is the weight corresponding to the second reference zoom ratio.

[0037] In some implementations, after obtaining the first reference zoom ratio through the PID algorithm and obtaining the second reference zoom ratio through the Kalman filter model, it is also necessary to perform a weighted sum calculation on the first reference zoom ratio and the second reference zoom ratio to obtain the final second zoom ratio.

[0038] Among them, in some embodiments, the weights w1 and w2 can be determined by the developer after multiple tests. For example, the developer can control the camera to move a certain distance, and then obtain the first reference zoom ratio R1 and the second reference zoom ratio R2; next, the developer can manually adjust the camera zoom ratio R so that the subject size does not change much and the video screen does not appear stiff. In this way, the developer can complete a test and obtain a set of test data R, R1, and R2. Similarly, the developer can complete multiple tests and obtain multiple sets of test data. Substituting each set of test data into the above-mentioned weighted sum function R can obtain the weights w1 and w2.

[0039] Thus, through the shooting method provided by the present application, when the photographer moves the camera, the electronic device can automatically adjust the zoom ratio of the camera, so that the captured video presents a visual effect of the subject size remaining unchanged. In addition, since the relative movement speed of the camera and other parameters are referenced when adjusting the zoom ratio, the adjustment of the zoom ratio is smoother and smoother, and there will be no stiff video screen, unnatural background changes, and visual focus jumps, thereby improving the fluency and overall effect of the video.

[0040] In a possible implementation of the first aspect above, during the process of shooting with the first camera using the first shooting mode, the method also includes: obtaining subject frames corresponding to the subject in two consecutive frames respectively; determining the position difference of the subject photographed by the first camera in the two frames based on the position information of the matching feature points in the two frames; and performing anti-shake processing on the latter frame of the two frames based on the position difference and the sensor data collected by the gyroscope sensor of the electronic device.

[0041] In some implementations, the subject of the captured video may shake due to factors such as the shaking of the photographer, the movement of the camera, and the adjustment of the focal length. Therefore, during the video shooting process, the electronic device can also perform anti-shake processing on the subject, so that the captured subject is always in the same position in the video screen, thereby avoiding the visual effect of the video showing the shaking of the subject.

[0042] In some implementations, when the subject is subjected to anti-shake processing, sparse optical flow tracking can be performed on each pixel point of the subject within the subject frame to perform anti-shake processing. Sparse optical flow tracking refers to first selecting some key feature points (for example, human eyes, nose, etc.) within the subject frame, and then determining the same key feature points in two consecutive frames, and calculating the relative position difference of the matched same key feature points in the two frames (such as the displacement from the edge position to the center position), thereby estimating the relative position difference of the subject in the video picture based on the displacement of some key feature points. Next, the electronic device can use the relative position difference of the subject in the video picture obtained by sparse optical flow tracking, as well as the acceleration data collected by the acceleration sensor in the electronic device, the angular velocity data collected by the gyroscope sensor, and other data to perform anti-shake processing on the video picture.

[0043] In a possible implementation of the first aspect, the anti-shake processing includes one or more of moving the subject position, image compensation, and rotating the image.

[0044] In some implementations, the electronic device can perform anti-shake processing on the latter frame of the two frames based on the position difference and the sensor data collected by the gyroscope sensor. For example, if the sensor data determines that the video image is rotated 30 degrees clockwise, the electronic device can control the video image to rotate 30 degrees in the opposite direction (counterclockwise). In this way, the visual effect of the video image shaking can be avoided.

[0045] In a second aspect, the present application provides an electronic device, comprising: a memory and a processor, the memory being used to store instructions executed by one or more processors of the electronic device, and the processor being one of the one or more processors of the electronic device, and being used to execute the shooting method mentioned in the present application.

[0046] In a third aspect, the present application provides a readable storage medium, on which instructions are stored. When the instructions are executed on an electronic device, the electronic device executes the shooting method mentioned in the present application.

[0047] In a fourth aspect, the present application provides a computer program product, comprising: computer instructions, which, when executed on an electronic device, enable the electronic device to execute the shooting method mentioned in the present application.

[0048] The beneficial effects of the second to fourth aspects mentioned above can be referred to the relevant descriptions in the first aspect mentioned above and various possible implementations of the first aspect, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 According to some embodiments of the present application, a schematic diagram of a shooting scene of Hitchcock zoom is shown;

[0050] Figure 2 According to some embodiments of the present application, a schematic diagram of a scene in which the size of a subject changes is shown;

[0051] Figure 3 According to some embodiments of the present application, a schematic diagram of an interface of a Hitchcock zoom shooting mode is shown;

[0052] Figure 4 According to some embodiments of the present application, a schematic diagram of an application scenario of a main frame is shown;

[0053] Figure 5 According to some embodiments of the present application, a schematic diagram of an application scenario of a main body mask is shown;

[0054] Figure 6 According to some embodiments of the present application, a schematic flow chart of a shooting method is shown;

[0055] Figure 7According to some embodiments of the present application, a schematic diagram of a focusing operation interface is shown;

[0056] Figure 8 According to some embodiments of the present application, a schematic diagram of a scene of image jitter and anti-shake processing is shown;

[0057] Fig. 9 According to some embodiments of the present application, a schematic flow chart of a shooting method combining anti-shake processing and zoom ratio adjustment is shown;

[0058] Fig.10 According to some embodiments of the present application, a specific flow chart of a shooting method is shown;

[0059] Fig.11 According to some embodiments of the present application, a schematic diagram of the structure of a PID algorithm is shown;

[0060] Fig.12 According to some embodiments of the present application, a schematic diagram of zoom ratios corresponding to different error proportional coefficients is shown;

[0061] Fig.13 According to some embodiments of the present application, a schematic diagram of a Kalman filter model is shown;

[0062] Fig.14 According to some embodiments of the present application, a schematic diagram of a state model of a Kalman filter is shown;

[0063] Fig.15 According to some embodiments of the present application, a schematic diagram of the hardware structure of an electronic device is shown. DETAILED DESCRIPTION

[0064] The following is a brief description of the background of the shooting method provided in the embodiment of the present application.

[0065] As mentioned above, the photographer can shoot a video using the Hitchcock zoom method, so that when the video viewer watches the video, a visual effect of background space compression or expansion can be produced. For example, Figure 1 FIG. 1 shows a schematic diagram of a shooting scene in which a photographer shoots a video using the Hitchcock zoom method. Figure 1 In the example, the photographer can turn on the video recording function of the mobile phone 100 to shoot the subject 101 to obtain a video with a Hitchcock zoom effect. Figure 1 At the shooting moment corresponding to the image (a) in FIG. 1 , the photographer can adjust the zoom ratio of the camera to 1x. Next, during the video shooting process, the photographer can move the mobile phone 100 closer to the subject 101, and can also reduce the camera zoom ratio while moving the mobile phone 100. For example, Figure 1At the shooting moment corresponding to the (b) figure in FIG. 1 , the photographer can reduce the camera zoom ratio from 1x to 0.8x. Then, the photographer can gradually move the mobile phone 100 away from the subject 101, and can also increase the camera zoom ratio while moving the mobile phone 100. For example, Figure 1 At the shooting moment corresponding to Figure (c) in the figure, the photographer can increase the camera zoom ratio from 0.8x to 1.2x.

[0066] However, many photographers with poor coordination skills or little shooting experience are unable to simultaneously perform the operations of "moving the camera" and "adjusting the camera zoom ratio" during Hitchcock zoom shooting, resulting in obvious changes in the size of the subject being photographed, reducing the overall visual effect of the video.

[0067] For example, Figure 2 FIG. 1 is a schematic diagram showing a shooting scene in which the size of the subject being photographed changes significantly. Figure 2 In the video recording process, the photographer can turn on the video recording function of the mobile phone 100 to shoot the subject 201. In the process of video recording, the photographer can move the mobile phone 100 close to the subject 201, and the photographer can also reduce the camera zoom ratio while moving the mobile phone 100. For example, the camera zoom ratio can be Figure 2 The 1x shown in Figure (a) gradually shrinks to Figure 2 However, if the camera zoom ratio needs to be reduced from 1x to 0.6x to eliminate the effect of the camera's moving distance on the subject size, then Figure 2 In the case of a photographer reducing the camera zoom ratio to a smaller extent, Figure 2 The main body 201 shown in FIG. (b) is compared with Figure 2 The subject 201 shown in FIG. 5 (a) becomes significantly larger, affecting the overall visual effect of the video.

[0068] In summary, many photographers are unable to adjust the camera zoom ratio while moving the camera handheld, resulting in obvious changes in the size of the captured subject, which reduces the overall visual effect of the video.

[0069] In order to solve the above problems, the present application proposes a shooting method, which does not require the photographer to manually adjust the zoom ratio of the camera (which can be the first camera mentioned above), but the electronic device controls the change of the zoom ratio of the camera to achieve zoom shooting. Specifically, when the photographer selects a specific first shooting mode (such as the Hitchcock zoom shooting mode) for video shooting, the method will first identify the subject being shot, and then adjust the zoom ratio of the camera based on the size change of the identified subject. For example, when it is detected that the change in the size of the subject exceeds a preset amplitude threshold, the camera will be triggered to adjust the zoom ratio, so that the camera continues to shoot the video with the adjusted zoom ratio, so that the background size in the captured video screen changes while the size of the subject remains basically unchanged.

[0070] As an example, the following combination Figure 3 The interface diagram shown in the figure briefly introduces the shooting method provided by this application. Figure 3 As shown in Figure (a) of FIG. 1 , the mobile phone 10 can include multiple shooting modes such as photo, video, and Hitchcock zoom. The photographer can use different shooting modes to shoot to obtain different shooting effects. When the photographer points the mobile phone 10 at the subject 301, selects the Hitchcock zoom shooting mode, and clicks the shooting button 302, the shooting mode can be entered. Figure 3 In the video shooting interface shown in FIG. (b) of FIG. 1 , the Hitchcock zoom is performed. During the video shooting process, the mobile phone 10 can detect the size of the subject 301 in real time, and when it is detected that the size of the subject 301 changes greatly, the camera zoom ratio is adjusted in time. For example, when the mobile phone 10 detects Figure 3 The size of the main body 301 shown in FIG. (c) is Figure 3 When the size of the subject 301 shown in FIG. (b) is significantly smaller than that of the subject 301, the mobile phone 10 will immediately adjust the zoom ratio of the camera. For example, the mobile phone 10 can adjust the zoom ratio of the camera from Figure 3 The 1x shown in Figure (c) is adjusted to Figure 3 1.2x as shown in Figure (d) of the figure, so that Figure 3 The size of the main body 301 shown in FIG. (d) is Figure 3 The difference between the sizes of the main body 301 shown in (b) is within the preset difference range.

[0071] It can be understood that in some embodiments, the electronic device can determine the size of the main body based on the main body frame and the main body mask corresponding to the main body. The main body frame refers to a rectangular frame that can surround the main body. Figure 4 As shown, after the electronic device recognizes the subject 4012 in the video screen 401, the subject 4012 can be surrounded by the subject frame 4011. The subject mask can refer to Figure 5As shown, when the electronic device recognizes the subject of the video screen 501 and determines that the pixel points corresponding to the subject are pixel points A, pixel points B, pixel points C, and pixel points D, a mask 502 corresponding to the video screen 501 can be generated. In the embodiment of the present application, the data corresponding to the pixel points A to pixel points D corresponding to the subject in the mask 502 are all 1, and the data corresponding to the pixel points of other non-subject parts in the video screen 501 in the mask 502 are all 0.

[0072] Among them, based on the size (length and width) of the subject frame, the length and width of the subject can be roughly determined. Based on the different data in the subject mask, the specific proportion of the subject in the video screen can be calculated, for example, Figure 5 As shown in , since the data corresponding to the subject pixel points (pixel points A to D) in the mask 502 is 1, and the data corresponding to the non-subject pixel points in the mask 502 is 0, it can be determined based on the mask 502 that the proportion of the subject in the picture 501 is 4 / 9. Therefore, the electronic device can determine whether the change range of the subject size is large based on the changes in the subject frame and the subject mask, and when it is detected that the change range of the subject size is large, the zoom ratio of the camera can be automatically adjusted.

[0073] It can be understood that in some embodiments, when adjusting the zoom ratio of the camera, the electronic device can determine the camera zoom ratio to be adjusted by combining the change parameters of the main frame size, the change parameters of the main body occupancy ratio corresponding to the main body mask, and the relative movement speed of the camera. Therefore, when shooting a video based on the adjusted camera zoom ratio, the main body size remains basically unchanged, and there will be no unnatural background changes and visual focus jumps due to the abrupt change of the zoom ratio, thereby improving the smoothness and overall effect of the video. For example, the main frame size change parameter can be input into an industrial model that determines the control signal value based on the parameter change value to obtain a first reference zoom ratio; the original zoom ratio, the change value of the main body occupancy ratio, and the relative movement speed of the camera are input into an industrial model that predicts a new control signal based on speed, parameter change, and known control signal to obtain a second reference zoom ratio; then the first reference zoom ratio and the second reference zoom ratio are weighted and summed to obtain the final adjusted zoom ratio.

[0074] In this way, when the photographer moves the camera, the electronic device can automatically adjust the zoom ratio of the camera, so that the captured video presents a visual effect that the size of the subject remains basically unchanged. In addition, since the relative movement speed of the camera and other parameters are referenced when adjusting the zoom ratio, the adjustment of the zoom ratio is smoother and smoother, and there will be no jerky video images, unnatural background changes, or visual focus jumps, thereby improving the smoothness and overall effect of the video.

[0075] For the sake of convenience in description, the zoom ratio before adjustment may be recorded as the "first zoom ratio"; the zoom ratio after adjustment may be recorded as the "second zoom ratio". In addition, the target subject frame corresponding to the subject may be recorded as the "first subject frame", the target subject mask corresponding to the subject may be recorded as the "first subject mask", and the proportion of the subject in the video screen calculated based on the first subject mask may be recorded as the "first occupancy ratio". For example, the subject frame determined when shooting is just started may be recorded as the first subject frame, the subject mask determined when shooting is just started may be recorded as the first subject mask, and the proportion of the subject in the video screen calculated based on the first subject mask may be recorded as the first occupancy ratio. In this way, in the subsequent video shooting process, if the size of the subject frame is not much different from the size of the first subject frame, and the proportion of the subject in the video screen is not much different from the first occupancy ratio, it can be determined that the subject presents a visual effect of basically unchanged size in the video.

[0076] Therefore, in the embodiment of the present application, when the camera is shooting, if the electronic device detects that the zoom ratio of the camera is the first zoom ratio at the first shooting moment, and determines that the main frame corresponding to the first shooting moment is the first main frame, the main body mask is the first main body mask, and the main body occupies the first occupancy ratio in the video screen, the camera can be controlled to continue shooting based on the first zoom ratio. Next, in the subsequent shooting process, the electronic device will detect the main body size in real time, that is, calculate the main body frame size and the main body occupancy ratio in real time. Among them, if the electronic device detects that there is a second shooting moment, the size of the captured main body changes significantly. For example, the second main body frame size corresponding to the second shooting moment is greatly different from the first main body frame size, and / or the second occupancy ratio corresponding to the second shooting moment is greatly different from the first occupancy ratio, the electronic device will control the camera to adjust from the first zoom ratio to the second zoom ratio. At this time, the third main body frame size corresponding to the second zoom ratio is slightly different from the first main body frame size, and the third occupancy ratio corresponding to the second zoom ratio is slightly different from the first occupancy ratio. In this way, the captured video can present a visual effect that the main body size is basically unchanged.

[0077] In some embodiments, as described above, when adjusting the zoom ratio of the camera from the first zoom ratio to the second zoom ratio, the electronic device can determine the camera zoom ratio to be adjusted by combining the change parameter of the main frame size, the change parameter of the main body occupancy ratio corresponding to the main body mask, and the relative movement speed of the camera. For example, the main frame size change parameter can be input into an industrial model that determines the control signal value based on the parameter change value to obtain the first reference zoom ratio; the original first zoom ratio, the change value of the main body occupancy ratio, and the relative movement speed of the camera are input into the industrial model that predicts the new control signal based on the speed, parameter change, and known control signal to obtain the second reference zoom ratio; finally, based on the first reference zoom ratio and the second reference zoom ratio, the final adjusted second zoom ratio is obtained.

[0078] In some embodiments, when determining the first reference zoom ratio, the first reference zoom ratio can be obtained by a proportional integral differential (PID) algorithm. The PID algorithm is often used in industrial control, and the difference between the target output parameter and the actual output parameter is input into the PID algorithm to obtain a system control signal. By controlling the industrial system through the obtained system control signal, the output parameter can be made to be less different from the target output parameter, so that the industrial system is more stable, and the output parameter can be maintained at the target output parameter. Therefore, in the present application, the size of the first main body frame can be used as the target output parameter in the industrial system; the size difference between the first main body frame and the second main body frame can be used as the difference between the target output parameter and the actual output parameter in the industrial system; and the first reference zoom ratio can be used as the system control signal in the industrial system. Therefore, the PID algorithm can be applied to the zoom ratio calculation process of the camera, and the size difference between the first main body frame and the second main body frame can be input into the PID algorithm to obtain the first reference zoom ratio, so that the difference between the main body frame size obtained by the first reference zoom ratio and the first main body frame size is less than the difference threshold.

[0079] In addition, in some other embodiments, when determining the second reference zoom ratio, the second reference zoom ratio can be obtained through a Kalman filter model. Among them, the Kalman filter is a mathematical model for estimating the state of a dynamic system, which is usually used in fields such as machine vision. The Kalman filter model can predict a new control signal based on a known control signal and multiple measurement values ​​(such as sensor data obtained by a sensor) in the presence of noise or error. Therefore, in the present application, the first zoom ratio can be used as a known control signal; the difference between the first subject occupancy ratio and the second subject occupancy ratio, the relative motion speed of the camera and other parameters can be used as multiple measurement values; and the second reference zoom ratio can be used as a new control signal. Therefore, the Kalman filter model can be applied to the zoom ratio calculation process of the camera, and the first zoom ratio and multiple parameters can be input into the Kalman filter model to obtain the second reference zoom ratio, so that the process of changing from the first zoom ratio to the second reference zoom ratio is smoother and smoother, and there will be no stiff video picture, unnatural background changes, visual focus jumps and other phenomena, thereby improving the fluency and overall effect of the video.

[0080] Next, after the first reference zoom ratio is obtained through the PID algorithm and the second reference zoom ratio is obtained through the Kalman filter model, it is necessary to perform a weighted sum calculation on the first reference zoom ratio and the second reference zoom ratio to obtain the final second zoom ratio. Specifically, refer to formula (1).

[0081] (one)

[0082] Among them, R can be the calculated second zoom ratio; R1 can be the first reference zoom ratio obtained by the PID algorithm; R2 can be the second reference zoom ratio obtained by the Kalman filter model; w1 is the weight corresponding to the first reference zoom ratio, w2 is the weight corresponding to the second reference zoom ratio, and the sum of w1 and w2 is 1.

[0083] Among them, in some embodiments, the weight w1 and weight w2 can be determined by the developer after multiple tests. For example, the developer can control the camera to move a certain distance, and then obtain the first reference zoom ratio R1 and the second reference zoom ratio R2; next, the developer can manually adjust the camera zoom ratio R so that the subject size does not change much and the video screen does not appear stiff. In this way, the developer can complete a test and obtain a set of test data R, R1, and R2. Similarly, the developer can complete multiple tests and obtain multiple sets of test data. Substituting each set of test data into the above formula (I) can obtain the weight w1 and weight w2.

[0084] Thus, through the above shooting method, when the photographer moves the camera, the electronic device can automatically adjust the zoom ratio of the camera, so that the captured video presents a visual effect of the subject size remaining unchanged. In addition, since the relative movement speed of the camera and other parameters are referenced when adjusting the zoom ratio, the adjustment of the zoom ratio is smoother and more stable, and there will be no stiff video screen, unnatural background changes, visual focus jumps, etc., which improves the smoothness and overall effect of the video.

[0085] It can be understood that the above-mentioned shooting method of the present application can be applied to third-party applications such as electronic devices or cameras. Among them, the applicable electronic device may include one or more cameras (such as wide-angle cameras, telephoto cameras, etc.); or, the electronic device can also be connected to other cameras to adjust the zoom ratio of the connected camera (such as the electronic device and the external camera can be connected via Bluetooth for data transmission). Specifically, the electronic device can be any electronic device such as a mobile phone, computer, tablet computer, augmented reality (AR) device, wearable device, laptop computer, vehicle-mounted smart terminal, etc., and the present application does not limit this.

[0086] It is understood that the shooting method provided in this application can be applied to any camera that can shoot videos and adjust the zoom ratio. For example, it can include but is not limited to: internal cameras of electronic devices, digital cameras, medical cameras, robot vision cameras, security surveillance cameras, and any other cameras, and this application does not limit this.

[0087] In addition, it can be understood that in the embodiments of the present application, the parameters, models, sizes, uses, etc. of the camera are not limited. For example, the camera can be a zoom camera, a mid-focus camera, an ultra-wide-angle camera, a macro camera, etc. The present application does not limit this. In this way, any camera that can shoot videos and change the zoom ratio can be used to shoot videos through the shooting method provided by the present application, thereby achieving the visual effect of the Hitchcock zoom method.

[0088] It can be understood that in some embodiments, the camera can be located inside the electronic device, and the zoom ratio of the camera can be directly controlled by the electronic device. In addition, the camera can also be an external camera located outside the electronic device, and the external camera can be connected to the electronic device through a wired connection or wirelessly (such as Bluetooth, etc.), so that the electronic device can obtain the operation information and video shooting status of the external camera, thereby adjusting the zoom ratio of the external camera. Therefore, this application does not limit the location of the camera.

[0089] For ease of description, the following uses the camera of an electronic device as an example to introduce the shooting method provided by this application.

[0090] The following is based on Figure 6 The flowchart shown in FIG. 1 introduces the specific implementation process of the shooting method provided in the embodiment of the present application. The shooting method can be applied to electronic devices, such as the above Figure 3 The mobile phone 10 shown in FIG. Figure 6 Specifically, the method comprises the following steps:

[0091] S601: A first shooting instruction is detected, where the first shooting instruction is used to instruct a camera of an electronic device to enter a first shooting mode, and in the first shooting mode, a change in size of a subject captured during movement of the camera satisfies a shooting condition.

[0092] In the embodiment of the present application, the first shooting instruction may be an operation instruction for the photographer to start the Hitchcock zoom; the first shooting mode may be the Hitchcock zoom shooting mode.

[0093] In some embodiments, the photographer can manually aim at the subject when shooting, and the electronic device can identify the subject based on the photographer's aiming operation. Figure 7 As shown, when the photographer performs Hitchcock zoom shooting through the mobile phone 10, when the mobile phone 10 detects the photographer's click operation on the screen, an alignment frame 701 is generated at the corresponding position. Therefore, the photographer can click on the subject in the video screen so that the alignment frame 701 is located at the subject position. In addition, the photographer can also directly drag the alignment frame 701 to the subject position for alignment. The present application does not limit the specific operation process of the photographer's alignment operation.

[0094] Next, after the electronic device detects that the photographer's finger has left the screen (for example, it detects that there is no pressing operation on the screen), it can identify the photographed subject based on the photographer's alignment operation. Among them, when the electronic device performs subject recognition, it can detect the subject from the video screen through an artificial intelligence (AI) algorithm. For example, the AI ​​algorithm can extract image features (such as edges, textures, colors, etc.) from the video screen; then perform instance segmentation on the video screen based on the image features, such as segmenting multiple entities such as people and animals according to the features of each pixel point; finally, the individual corresponding to the photographer's focus position is used as the subject, thereby identifying the photographed subject in the video screen. It can be understood that the subject can be identified by any algorithm that can perform image target detection, and this application does not limit this.

[0095] S602: At a first shooting moment, determining that a subject photographed by a camera has a first subject size, and the camera has a first zoom ratio.

[0096] In the embodiment of the present application, the first subject size may be a target subject size, which may include the first subject frame size mentioned above and the first occupancy ratio of the subject in the video screen. The first shooting moment may be any shooting moment at which the subject size captured is the first subject size. The first zoom ratio may be the camera zoom ratio before adjustment mentioned above, and at the first shooting moment, the subject captured by the camera at the first zoom ratio has the first subject size.

[0097] In some embodiments, when the camera is shooting, the electronic device can identify the subject to be shot, thereby determining the size of the subject in real time. The subject size may include at least one of the above-mentioned subject frame size (length and width of the subject frame) and the proportion of the subject in the video screen. For example, the first subject size may include the first subject frame size corresponding to the subject at the first shooting moment, and the first proportion of the subject corresponding to the first shooting moment. Figure 4 As shown in the main body frame 4011 in the figure, the main body frame can be a rectangular frame; in addition, the main body frame can also be any shape such as a diamond frame, an elliptical frame, etc., which is not limited in this application. In addition, the specific occupancy ratio of the subject in the video screen can be calculated based on the different data in the main body mask. For example, the electronic device can obtain the first main body mask corresponding to the first screen captured by the camera at the first shooting moment. In addition, the electronic device can set the data corresponding to the pixel points of the main part of the first screen in the first main body mask as the first data, and set the data corresponding to the pixel points of the non-main part in the first main body mask as the second data, so that each pixel point in the first screen corresponds to each data in the first main body mask. In this way, the first occupancy ratio of the subject in the first screen can be determined based on the number of the first data and the total number of data in the first main body mask.

[0098] S603: At a second shooting moment, it is detected that the subject shot by the camera has a second subject size, and a difference between the first subject size and the second subject size does not satisfy a shooting condition.

[0099] In an embodiment of the present application, the second shooting moment may be a shooting moment after the first shooting moment. The second subject size may be a subject size that is significantly different from the first subject size and is captured by the camera at the first zoom ratio. The second subject size may include a second subject frame size and a second occupancy ratio of the subject in the video screen.

[0100] In some embodiments, when the photographer moves the camera, the electronic device detects the subject size in real time and determines whether the change in the subject size meets the shooting conditions. For example, the electronic device can determine whether the first subject size and the second subject size meet the following shooting conditions: the absolute value of the difference between the second subject frame size and the first subject frame size corresponding to the second shooting moment is less than the difference threshold, and the absolute value of the difference between the second occupancy ratio and the first occupancy ratio corresponding to the second shooting moment is less than the ratio threshold. When the electronic device determines that the difference between the first subject size and the second subject size does not meet the above shooting conditions, the electronic device can execute step S604 to adjust the zoom ratio of the camera. Among them, the difference threshold and the ratio threshold can be set arbitrarily. For example, the maximum subject frame size error that the video viewer cannot see the change in the subject size can be used as the difference threshold, and the maximum subject occupancy ratio error that the video viewer cannot see the change in the subject size can be used as the ratio threshold. The specific settings of the difference threshold and the ratio threshold are not limited in this application.

[0101] S604: adjusting the zoom ratio of the camera from the first zoom ratio to the second zoom ratio, wherein the subject photographed by the camera at the second zoom ratio has a third subject size, and a difference between the third subject size and the first subject size satisfies a photographing condition.

[0102] In the embodiment of the present application, the second zoom ratio may be the adjusted camera zoom ratio. The third subject size is the subject size of the subject photographed by the second zoom ratio, and may include a third subject frame size and a third occupancy ratio corresponding to the second zoom ratio.

[0103] In some embodiments, after determining through the above step S603 that the difference between the first subject size and the second subject size does not meet the shooting condition, the electronic device can control the camera to adjust the first zoom ratio to the second zoom ratio, so that the difference between the third subject size captured by the camera through the second zoom ratio and the first subject size meets the shooting condition.

[0104] Among them, the difference between the third body size and the first body size that satisfies the shooting conditions may include: the absolute value of the difference between the third body frame size corresponding to the second zoom ratio and the first body frame size is less than the difference threshold; the absolute value of the difference between the third occupancy ratio and the first occupancy ratio corresponding to the second zoom ratio is less than the ratio threshold.

[0105] In some embodiments, as described above, when adjusting the zoom ratio of the camera from the first zoom ratio to the second zoom ratio, the electronic device may first determine the first reference zoom ratio based on the difference between the first main frame size and the second main frame size. In addition, the electronic device may also determine the second reference zoom ratio based on the relative movement speed between the first shooting moment and the second shooting moment of the camera. Finally, the final second zoom ratio is determined based on the first reference zoom ratio and the second reference zoom ratio. The specific process of obtaining the second zoom ratio can be referred to as follows Fig.10 It is described in , and will not be described in detail here.

[0106] Thus, through the above shooting method, when the photographer moves the camera, the electronic device can automatically adjust the zoom ratio of the camera, so that the captured video presents a visual effect of the subject size remaining unchanged. In addition, since the relative movement speed of the camera and other parameters are referenced when adjusting the zoom ratio, the adjustment of the zoom ratio is smoother and more stable, and there will be no stiff video screen, unnatural background changes, visual focus jumps, etc., which improves the smoothness and overall effect of the video.

[0107] It is understandable that during the video shooting process, due to factors such as the photographer's shaking, the movement of the camera, and the adjustment of the focal length, the captured video subject may be shaken. Figure 8 As shown in the figure, when the photographer shoots a video, due to the shaking of the camera, the main frame 801 may move from the upper left corner position in (a) to the lower left corner position in (b), so that the video will show the visual effect of the main body shaking. Therefore, in the process of shooting a video, the electronic device can also perform anti-shake processing on the main body. For example, Figure 8 In the example, the main frame 801 is moved from the lower left corner position in (b) to the upper left corner position in (c) by anti-shake processing, so that the captured subject is always in the same position in the video screen, thereby avoiding the visual effect of subject shaking in the video.

[0108] Among them, when the subject is anti-shake processed, the anti-shake processing can be performed by performing sparse optical flow tracking on each subject pixel point in the subject frame. Among them, sparse optical flow tracking means that some key feature points (for example, human eyes, nose, etc.) in the subject frame are first selected, and then the same key feature points are determined in two consecutive frames, and the relative motion information of the matched same key feature points in the two frames is calculated (for example, the displacement between the edge position and the center position), so as to estimate the relative position difference of the subject in the video picture based on the displacement of some key feature points. Next, the relative position difference of the subject in the video picture obtained by sparse optical flow tracking, as well as the acceleration data collected by the acceleration sensor in the electronic device, the angular velocity data collected by the gyroscope sensor and other data can be used to perform anti-shake processing on the video picture. For example, if it is determined through various data that the video picture moves 2 cm to the upper right corner, the electronic device can control the video picture to move 2 cm in the opposite direction (lower left corner). Furthermore, if the position difference of the subject in the video screen is "7 mm from the center position to the edge position", the subject can be adjusted to move in the opposite direction "7 mm from the edge position to the center position" based on the position difference. After the subject moves in the opposite direction, blank parts may appear in the video screen. For example, after the subject moves from the edge position to the center position, blank parts may appear at the edge position. The blank parts can be filled by copying the pixels adjacent to the blank parts to make the video screen look more complete. In this way, the subject can be adjusted to the initial position in the video screen, so that the captured subject is always in the same position in the video screen, thereby avoiding the visual effect of screen shaking in the video.

[0109] Thus, in the shooting method provided by the present application, by combining the anti-shake processing technology and the camera zoom ratio adjustment technology, the shot video can present the following visual effects: the background of the video screen will change, while the size of the shooting subject remains basically unchanged, and the subject has a high image stability. Thus, by enlarging or reducing the background, the spatial sense of the video and the impact of the lens are enhanced, and the action or psychology of the shooting subject can be strengthened, so that the video viewers can better experience the three-dimensional effect of the scene and the psychological changes of the shooting subject.

[0110] The following is based on Fig. 9 The flowchart shown in the figure describes the specific execution process of the shooting method combined with zoom ratio adjustment and anti-shake processing provided by the present application. Among them, the shooting method can be applied to electronic devices, such as the above Figure 3 The mobile phone 10 is shown in FIG.

[0111] As described above, in some embodiments, when adjusting the zoom ratio of the camera, a first reference zoom ratio can be first obtained through the main frame size difference and the PID algorithm, and a second reference zoom ratio can be obtained through the relative movement speed of the camera, the change in the proportion of the main body in the video screen and the Kalman filter. The first reference zoom ratio and the second reference zoom ratio are then weightedly summed to obtain the second zoom ratio.

[0112] In some embodiments, the relative movement speed of the camera may be determined based on the relative distance between the subject and the camera, and the relative distance between the subject and the camera may be represented by subject depth information.

[0113] It can be understood that the depth information of an image refers to the depth value of each pixel in the image, which is used to indicate the distance between the real object corresponding to the image and the camera. Therefore, the change in the subject depth information can indicate the change in the distance between the real object corresponding to the subject and the camera from the first shooting moment to the second shooting moment, and can also be used to indicate the movement distance of the camera.

[0114] In some embodiments, the subject depth information can be determined by binocular stereo vision technology. Binocular stereo vision technology is a machine vision technology that determines the image depth through two cameras. Specifically, the electronic device can control the first camera and the second camera to shoot the subject at the same shooting time to obtain the first picture and the second picture, and the first camera and the second camera have the same zoom ratio at the same shooting time. Since the two cameras do not overlap and there is a certain distance, there is a certain coordinate position difference (parallax) between each pixel in the first picture and the pixel corresponding to the second picture. Then, based on the focal length of the first camera, the spatial distance between the two cameras, the parallax corresponding to each pixel in the first picture, and the parameters of the two cameras, the depth information corresponding to each pixel in the first picture taken by the first camera is determined. Then, the electronic device can filter out the depth information of the background in combination with the first data corresponding to the subject in the subject mask corresponding to the first picture, and only retain the depth information corresponding to each pixel of the subject, so as to determine the relative distance between the subject and the first camera based on the depth information corresponding to each pixel of the subject. In addition, the depth information of the subject can also be determined by any other machine vision technology, which is not limited in this application. Among them, since electronic devices such as mobile phones usually have multiple cameras inside, when the electronic device uses a first camera to shoot a video, any other camera can be used to assist the first camera in obtaining the depth information of the subject using binocular stereo vision technology.

[0115] Therefore, in Fig. 9In the flowchart shown, camera 1 can be used for video shooting, and camera 2 can be used to assist camera 1 in obtaining depth information of the subject, so that camera 1 and camera 2 are used in combination to complete the shooting method provided in this application.

[0116] Specifically, in Fig. 9 In the example, when the photographer starts shooting through camera 1 (901), he can first perform instance segmentation of the video screen through the above-mentioned AI algorithm (902), such as segmenting multiple individuals such as people and animals according to the characteristics of each pixel point. Then, combined with the photographer's manual alignment operation (903), the subject of the shooting is identified. Thereby, the subject frame (904) and subject mask (905) corresponding to the subject are obtained, and the proportion of the subject in the video screen at the current moment is determined based on the subject mask (906). In this way, in the subsequent shooting process, the subject frame, subject mask, and occupancy ratio at the start of shooting are used as the adjustment basis, so that the size of the subject remains basically unchanged during the entire shooting process.

[0117] Furthermore, during the shooting process, the electronic device can also extract image feature points from the video screen (907). Then, the subject feature points within the subject frame are selected for sparse optical flow tracking (908), thereby obtaining the relative motion information of the subject in the video screen. Then, based on the relative motion information of the subject in the video screen, the acceleration data collected by the acceleration sensor, and the angular velocity data collected by the gyroscope sensor, the video screen is subjected to anti-shake processing (909). This avoids the shaking of the subject caused by factors such as the photographer's hand shaking and movement, making the captured subject image more stable, and realizing 5D anti-shake for the subject, that is, anti-shake operation can be achieved in the five axes of up, down, front, back, and horizontal.

[0118] In addition, when the photographer is moving the electronic device, if the photographer needs to adjust the camera 1 from the first zoom ratio to the second zoom ratio at the second shooting moment, the electronic device needs to first calculate the first reference zoom ratio and the second reference zoom ratio, and then perform weighted sum calculation on the first reference zoom ratio and the second reference zoom ratio to obtain the second zoom ratio. Specifically, the electronic device can calculate the first reference zoom ratio through the size change of the main frame and the PID algorithm. In addition, the depth information of each pixel in the video screen is obtained by using the binocular stereo vision technology through the camera 1 and the camera 2 (910), and then the depth information of the pixel of the non-main part is filtered out in combination with the main body mask, so as to obtain the depth information of each pixel of the main part (911). In addition, the relative movement distance of the camera can be determined based on the change of the depth information of each pixel of the main part at the second shooting moment and the depth information corresponding to the first shooting moment (912). Then, based on the moving speed of the electronic device, the difference in the proportion of the subject in the video screen, the relative movement distance of the camera, and the first zoom ratio of the camera, a Kalman filter dynamic modeling is performed to determine the second reference zoom ratio of the camera at the second shooting moment. Finally, the first reference zoom ratio and the second reference zoom ratio are weighted and calculated to obtain the second zoom ratio, so as to adjust the zoom ratio of the camera 1 (913), and shoot the video at the adjusted zoom ratio (901).

[0119] In this way, the captured video can present the following visual effects: the background of the video screen will change, while the size of the captured subject remains basically unchanged, and the subject has a high image stability.

[0120] Combine the following Fig.10 The flowchart shown in FIG. 1 describes in detail the specific implementation process of the shooting method provided by the present application. The shooting method can be applied to electronic devices, such as the above Figure 3 The mobile phone 10 shown in FIG. Fig.10 Specifically, the method comprises the following steps:

[0121] S1001: A first shooting instruction is detected, where the first shooting instruction is used to instruct a camera to enter a first shooting mode.

[0122] In the embodiment of the present application, the first shooting instruction may be an operation instruction for the photographer to start the Hitchcock zoom; the first shooting mode may be the Hitchcock zoom shooting mode.

[0123] In some embodiments, the electronic device can respond to the photographer's Hitchcock zoom shooting instruction to shoot a video. In addition, during the process of the camera shooting, the electronic device can also identify the subject of the shooting. The specific process can be referred to above Figure 6The process is described in step S601 in the process and will not be described again here.

[0124] In addition, in some embodiments, when the camera is shooting in the first shooting mode, the electronic device can also perform anti-shake processing on the captured subject in real time. For example, the electronic device can determine the position information of the same key feature point in the two subject frames corresponding to two consecutive frames to calculate the position difference of the matched same key feature point in the two frames (such as the displacement from the edge position to the center position). Finally, the electronic device can perform anti-shake processing on the latter frame of the two frames based on the position difference and the sensor data collected by the gyroscope sensor. For example, the position of the subject in the latter frame can be moved based on the position difference; or, if a blank part appears after the subject position is moved, the pixels adjacent to the blank part can be copied and pasted to achieve picture compensation, or the latter frame can be rotated in reverse based on the sensor data. In this way, the subject can be adjusted to the initial position in the video picture, so that the captured subject is always in the same position in the video picture, thereby avoiding the visual effect of the video showing a shaking picture.

[0125] S1002: At a first shooting moment, it is detected that the camera corresponds to a first zoom ratio, and it is determined that the subject corresponds to a first subject frame and a first subject mask.

[0126] In an embodiment of the present application, the first shooting moment can be any moment, and the first shooting moment needs to correspond to the first main body frame and the first main body mask; the first zoom ratio is a camera zoom ratio that enables the first shooting moment to correspond to the first main body frame and the first main body mask.

[0127] In some embodiments, since the Hitchcock zoom shooting method requires that the size of the subject remains basically unchanged, in the embodiments of the present application, the electronic device can use the subject frame when the camera just starts shooting the subject as the first subject frame; and use the subject mask when the photographer just starts shooting the subject as the first subject mask. As a result, in the subsequent shooting process, the first subject frame and the first subject mask are used as adjustment references, so that the size of the subject remains basically unchanged during the entire shooting process. The method for obtaining the first subject frame and the first subject mask can refer to the above Figure 6 The step S602 in FIG. 1 is described above and will not be repeated here.

[0128] S1003: When the camera is at a second shooting moment with a first zoom ratio, a second subject frame and a second subject mask corresponding to the subject are detected.

[0129] In the present application, the second shooting moment may be any moment after the first shooting moment when shooting is performed using the first zoom factor. The second main body frame is the main body frame corresponding to the second shooting moment, and the second main body mask is the main body mask corresponding to the second shooting moment.

[0130] In some embodiments, after the first shooting moment, during the shooting process, the electronic device will acquire the corresponding subject frame and subject mask in real time.

[0131] S1004: Determine whether the second main body frame is significantly different from the first main body frame, or whether the second occupation ratio corresponding to the second main body mask is significantly different from the first occupation ratio corresponding to the first main body mask. If yes, proceed to step S1006; if no, proceed to step S1005.

[0132] In the present application, the first occupancy ratio may be the occupancy ratio of the subject in the video screen determined by the first subject mask, and the second occupancy ratio may be the occupancy ratio of the subject in the video screen determined by the second subject mask.

[0133] In some embodiments, after determining the subject mask, the proportion of the subject in the video screen can be determined based on the subject mask. Specifically, the number of pixel data corresponding to the subject pixel points and the total number of pixels of the subject mask can be determined, and the ratio of the two can be used as the proportion of the subject in the video screen. Figure 5 As shown, since the data corresponding to the main pixel points (pixel points A to D) in mask 502 is 1, and the data corresponding to the non-main pixel points in mask 502 is 0, it can be determined based on mask 502 that the proportion of the main body in the picture 501 is 4 / 9.

[0134] In some embodiments, if the size of the first main body frame differs greatly from the size of the second main body frame, and / or the first occupancy ratio differs greatly from the second occupancy ratio, then the process goes to step S1006 to determine whether the zoom ratio of the camera needs to be adjusted. If the size of the first main body frame differs slightly from the size of the second main body frame, and / or the first occupancy ratio differs slightly from the second occupancy ratio, then the process goes to step S1005 to control the camera to continue shooting video at the first zoom ratio.

[0135] S1005: Control the camera to continue shooting video at the first zoom ratio.

[0136] In an embodiment of the present application, if it is determined through the above step S1004 that the size of the first main body frame is slightly different from the size of the second main body frame, and the first occupancy ratio is slightly different from the second occupancy ratio, it can be determined that the size of the subject photographed at the second shooting moment using the first zoom ratio remains unchanged, then the electronic device can control the camera to continue shooting video using the first zoom ratio.

[0137] In addition, in some embodiments, the electronic device will continue to repeat steps S1003 and S1004, and continue to monitor the size of the main frame and the proportion of the subject in the video screen in real time, so that when the size of the main frame or the proportion of the subject in the video screen changes significantly, the zoom ratio of the camera can be adjusted in time.

[0138] S1006: Determine whether the zoom ratio of the camera needs to be adjusted to keep the size of the subject frame unchanged and the proportion of the subject in the video screen unchanged.

[0139] In an embodiment of the present application, when the electronic device determines through the above step S1004 that the size of the first main body frame and the size of the second main body frame are significantly different, or the first occupancy ratio and the second occupancy ratio are significantly different, it means that the size of the subject has changed significantly and does not comply with the Hitchcock zoom rule. In this case, it is necessary to adjust the zoom ratio of the camera to keep the size of the main body frame unchanged and the occupancy ratio of the subject in the video screen unchanged.

[0140] When determining the target zoom ratio to be adjusted, the first reference zoom ratio can be obtained through the following steps S61a and S62a, and the second reference zoom ratio can be obtained through the following steps S61b to S64b. Then, through step S1007, based on the first reference zoom ratio and the second reference zoom ratio, the second zoom ratio that needs to be adjusted is determined.

[0141] S61a: Obtain the difference between the size of the first main body frame and the size of the second main body frame.

[0142] In some embodiments, when determining the first reference zoom ratio, the difference between the first main frame size and the second main frame size may be calculated first. The main frame size refers to the length and width of the main frame. Figure 4 As shown in , after determining the main body frame 4011, the length and width of the main body frame 4011 can be obtained. In this way, after determining the size of the first main body frame and the size of the second main body frame, the difference between the two can be determined. For example, if the size of the first main body frame is 100×100 and the size of the second main body frame is 80×80, the difference between the size of the first main body frame and the size of the second main body frame is 20×20.

[0143] S62a: Substitute the difference between the first main body frame size and the second main body frame size into the formula of the PID algorithm to obtain a first reference zoom ratio.

[0144] In the embodiment of the present application, after the difference between the first main body frame size and the second main body frame size is determined in the above step S61a, the difference is substituted into the formula of the PID algorithm to obtain the first reference zoom ratio. Fig.11 As shown in FIG, the PID algorithm is to determine the control signal of the system through the proportion P, the integral I and the differential D. Specifically, refer to Fig.11 As shown, first, the deviation value between the target parameter r(t) (eg, the size of the first main frame) and the actual output parameter y(t) (eg, the size of the second main frame) can be obtained, for example, Fig.11 The symbol "-" shown in the figure can be used to indicate that the deviation value is obtained by subtracting the target parameter r(t) from the actual output parameter y(t). Next, the deviation value can be output to the formulas corresponding to the error proportion P, error integral I, and error differential D, respectively, to obtain the calculation results corresponding to each formula. Finally, the calculation results of each formula can be summed to obtain the output result, for example, Fig.11 The symbol "∑" shown can be used to indicate that the error proportion P, error integral I, and error differential D are summed to obtain the output result. In addition, when the actual output parameter corresponding to the output result is still significantly different from the target parameter, the steps of the PID algorithm such as calculating the deviation value, calculating the results of each formula separately, and summing the results of each formula can be repeated to obtain the final output result that meets the target parameter.

[0145] Specifically, the calculation formula of the PID algorithm can refer to the following formulas (II) to (VI).

[0146] (two)

[0147] (three)

[0148] (Four)

[0149] (five)

[0150] (six)

[0151] Among them, in the above formula (ii), e(t) is the calculated difference between the size of the first main body frame and the size of the second main body frame; r(t) is the size of the first main body frame; y(t) is the size of the second main body frame.

[0152] In the above formula (III), P is the calculated error ratio, K pis the error proportional coefficient, and e(t) is the difference between the size of the first main frame and the size of the second main frame calculated by formula (ii).

[0153] In the above formula (IV), I is the calculated error integral, K i is the error integral coefficient, t is the shooting time corresponding to the continuous multi-frame video images, It refers to integrating the difference between the size of the main frame corresponding to each frame of the video image and the size of the first main frame in the time period t before the second shooting moment.

[0154] In the above formula (V), D is the calculated error differential, K d is the error differential coefficient, It refers to the rate of change of the error value between the main body frame size corresponding to each frame of the video image and the first main body frame size in the time period t before the second shooting moment.

[0155] In the above formula (VI), Z is the calculated first reference zoom ratio, P is the error ratio determined by formula (III), I is the error integral determined by formula (IV), and D is the error differential determined by formula (V). e(t) is the difference between the first main body frame size and the second main body frame size, K p is the error proportionality coefficient, K p e(t) is the error ratio between the size of the first main frame and the size of the second main frame, K i is the error integral coefficient, t is each shooting moment between the first shooting moment and the second shooting moment, K is the error integral between the size of each main frame corresponding to each shooting moment and the size of the first main frame, d is the error differential coefficient, is the differential error between each main body frame size corresponding to each shooting time and the first main body frame size.

[0156] In addition, it can be understood that in the above formulas (III) to (V), the error proportionality coefficient K p , Error integral coefficient K i , and the error differential coefficient K d When any coefficient in changes, the first reference zoom ratio will change. Therefore, developers can set the above three coefficients through multiple debugging. For example, Fig.12 As shown, the error integral coefficient K i and error differential coefficient K d When the error proportionality coefficient K is kept fixed, p Set different values ​​multiple times to find a suitable value as the final error proportional coefficient K p .

[0157] It is understandable that when the photographer moves the camera a small distance to shoot, the first reference zoom ratio will not change significantly over time. Fig.12 As shown, when the error integral coefficient K p Too large or too small (e.g., K p When K is the reference value (0.5, 1.1, 1.6, etc.), the electronic device will react to the slight difference, causing the first reference zoom ratio to fluctuate over time. When K is the reference value (Reference), the first reference zoom ratio will remain stable after reaching a certain value. Therefore, the developer can set the error ratio K multiple times. p , then calculate the error ratio P, and select the value of the first reference zoom ratio that will not fluctuate greatly over time as the final error ratio coefficient K p (like Fig.12 K p = reference value). In this way, the error proportionality coefficient K can be determined p Similarly, developers can use the same method to determine the error integral coefficient K i and error differential coefficient K d , which will not be described here. In addition, the error proportional coefficient K can also be determined by any other method: p , Error integral coefficient K i , and the error differential coefficient K d , this application does not limit this.

[0158] In this way, the electronic device can determine the first reference zoom ratio by executing the method corresponding to the above steps S61a and S62a through the above formulas (ii) to (vi), so that the size of the main body frame corresponding to the first reference zoom ratio is slightly different from the size of the first main body frame.

[0159] Next, the electronic device can also perform subsequent steps S61b to S64b, first determining the relative movement speed of the camera between the first shooting moment and the second shooting moment based on the change value of the spatial relative distance between the subject and the camera; and then generating a second reference zoom ratio through the Kalman filter model based on the first zoom ratio, the difference between the first occupancy ratio and the second occupancy ratio, and the relative movement speed corresponding to the camera. In this way, the second zoom ratio can be obtained through the first reference zoom ratio and the second reference zoom ratio.

[0160] S61b: Determine the depth information corresponding to the subject in the continuous multiple frames of video.

[0161] In some embodiments, when determining the second reference zoom ratio, the corresponding depth information in a plurality of consecutive video frames before the second shooting moment may be determined first.

[0162] S62b: Determine the relative movement speed of the camera based on the multiple depth information.

[0163] In some embodiments, after determining the depth information corresponding to the subject at different shooting times through the above step S61b, the relative distance between the subject and the camera at different shooting times can be determined. In this way, the electronic device can determine the relative movement distance of the camera within a certain period of time based on the relative distance between the subject and the camera at different shooting times, and then determine the relative movement speed of the camera based on the relative movement distance and the shooting time interval.

[0164] S63b: Obtain the moving speed of the electronic device through a sensor.

[0165] In some embodiments, the movement data of the electronic device may be collected by sensors such as an acceleration sensor and a gyroscope sensor to obtain the movement speed of the electronic device.

[0166] S64b: Perform Kalman filter modeling on parameters such as the movement speed of the electronic device, the relative movement speed of the camera, and the first zoom ratio, and obtain a second reference zoom ratio through the Kalman filter model.

[0167] In some embodiments, after the electronic device determines the relative movement speed of the camera through the above step S62b and determines the moving speed of the electronic device through step S63b, it can perform Kalman filter modeling on parameters such as the moving speed of the electronic device, the relative movement speed of the camera, the first zoom ratio, the first occupancy ratio and the second occupancy ratio, and obtain the second reference zoom ratio through the Kalman filter model.

[0168] Among them, the Kalman filter is a mathematical model for estimating the state of a dynamic system, and is often used in fields such as machine vision. The Kalman filter model can predict new control signals based on known control signals and multiple measurements (for example, sensor data obtained by sensors) in the presence of noise or errors. For example, Fig.13A schematic diagram of a Kalman filter model is shown. Among them, k-1 refers to the previous state (such as the previous moment, the last time, etc.), k refers to the current state, and k+1 refers to the next state (such as the next moment, the next time, etc.). The x in the k-1 state refers to the control signal of the previous state (such as the first zoom factor), and the x in the k state refers to the control signal that needs to be calculated currently (such as the second reference zoom factor). The circle represents the vector that needs to be input by the electronic device, such as u and z can be parameters such as the moving speed of the electronic device and the relative movement speed of the camera. It should be understood that for the sake of convenience, Fig.13 Only two input vectors are shown, and more or fewer input vectors may be included in practice, and this application does not limit this. The boxes represent coefficients in the Kalman model, such as B, H, F, R, and Q are all coefficients in the Kalman model; the asterisks represent Gaussian noise, such as w and v are both interference noise. In this way, the parameters such as the moving speed of the electronic device, the relative movement speed of the camera, the difference between the first occupancy ratio and the second occupancy ratio are input into the Kalman filter, and combined with the first zoom ratio, the second reference zoom ratio can be obtained.

[0169] In addition, if Fig.14 As shown in Figure , the calculation flow chart of the Kalman filter is shown. Fig.14 As shown, when calculating the information corresponding to the current k state, the information of the previous state (k-1 state) (P k-1|k-1 , X k-1|k-1 ), predict the information of the current state (k state) (P k|k-1 , X k|k-1 Then, the output parameter corresponding to the predicted information is compared with the target parameter to achieve error evaluation. If the evaluation result is good, such as the difference between the output parameter and the target parameter is small, the information is updated and the information prediction result is output. That is, the predicted information is updated to the known information (P k|k , X k|k ) and output. Next, the Kalman filter model can predict the subsequent state information again, for example, assigning k+1 to k (e.g., Fig.14 k←k+1) is shown in the figure, so as to repeat the prediction step to predict the information of the next state (k+1 state). k-1|k-1 , X k-1|k-1 ) may be a first zoom ratio, a first occupancy ratio; (P k|k-1 , X k|k-1 ) can be the predicted zoom ratio and the corresponding occupancy ratio; (P k|k , X k|k) can be the zoom ratio of the next state and the corresponding occupancy ratio. In this way, the second reference zoom ratio can be determined by the Kalman filter model, so that the second reference zoom ratio is adjusted smoothly as the camera moves, avoiding the video picture being too stiff due to the fast and slow zoom speed.

[0170] S1007: Obtain a second zoom magnification based on the first reference zoom magnification and the second reference zoom magnification.

[0171] In some embodiments, after obtaining the first reference zoom ratio through the above steps S61a and S62a, and obtaining the second reference zoom ratio through steps S61b to S64b, the first reference zoom ratio and the second reference zoom ratio can be weighted and summed to obtain the second zoom ratio as shown in the above formula (I). The electronic device can first execute the above steps S61a and S62a to obtain the first reference zoom ratio, and then execute steps S61b to S64b to obtain the second reference zoom ratio; or the electronic device can also first execute steps S61b to S64b to obtain the second reference zoom ratio, and then execute the above steps S61a and S62a to obtain the first reference zoom ratio, and the present application does not limit this.

[0172] S1008: Control the camera to adjust from the first zoom ratio to the second zoom ratio, and shoot a video at the second zoom ratio.

[0173] In an embodiment of the present application, after the second zoom ratio is calculated through the above step S1007, the camera can be controlled to adjust from the first zoom ratio to the second zoom ratio, and continue to shoot video at the second zoom ratio.

[0174] Thus, through the above shooting method, when the photographer moves the camera, the electronic device can automatically adjust the zoom ratio of the camera, so that the captured video presents a visual effect of the subject size remaining unchanged, and the video image has a high image stability. In addition, since the relative movement speed of the camera and other parameters are referenced when adjusting the zoom ratio, the adjustment of the zoom ratio is smoother and smoother, and there will be no jerky video images, unnatural background changes, or visual focus jumps, thereby improving the smoothness and overall effect of the video.

[0175] An embodiment of the present application provides a readable storage medium, on which instructions are stored. When the instructions are executed on an electronic device, the electronic device executes the shooting method mentioned in the present application.

[0176] An embodiment of the present application provides a computer program product, including: computer instructions, when the computer instructions are executed on an electronic device, the electronic device executes the shooting method mentioned in the present application.

[0177] An embodiment of the present application also provides an electronic device, which includes: a memory and a processor, the memory is used to store instructions executed by one or more processors of the electronic device, and the processor is one of the one or more processors of the electronic device, and is used to execute the shooting method mentioned in the present application.

[0178] The above-mentioned shooting method of the present application can be performed by an electronic device, wherein the electronic device can be any electronic device such as a mobile phone, a computer, a tablet computer, an augmented reality (AR) device, a laptop computer, etc. The form of the electronic device is not specifically limited in the embodiments of the present application.

[0179] like Fig.15 As shown, taking the above-mentioned mobile phone 10 as an example, a hardware structure diagram of an electronic device of an embodiment of the present application is exemplified.

[0180] like Fig.15 As shown, the mobile phone 10 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a gyroscope sensor 180A, a magnetic sensor 180B, an acceleration sensor 180C, an infrared sensor 180D, a temperature sensor 180E, a touch sensor 180F, an ambient light sensor 180G, a light sensor 180H, etc.

[0181] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the mobile phone 10. In other embodiments of the present application, the mobile phone 10 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0182] The processor 110 may include one or more processing units, for example, the processor 110 may include a central processing unit (CPU), a graphics processing unit (GPU), an embedded neural processing unit (NPU), an application processor (AP), a modem processor, a controller, a video codec, a digital signal processor (DSP), etc. Different processing units may be independent devices or integrated into one or more processors.

[0183] In an embodiment of the present application, the processor 110 may be used to execute the shooting method provided by the present application. For example, the second zoom ratio that needs to be adjusted may be calculated.

[0184] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0185] In the embodiment of the present application, the processor 110 can control the camera to adjust the zoom ratio so as to perform Hitchcock zoom video shooting.

[0186] The USB interface 130 can be used to connect a charger to charge the mobile phone 10, or to transfer data between the mobile phone 10 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0187] The charging management module 140 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. While the charging management module 140 charges the battery 142, it can also power the electronic device through the power management module 141.

[0188] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the screen 194, the camera 193, and the wireless communication module 160. In some other embodiments, the power management module 141 may also be disposed in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may also be disposed in the same device.

[0189] The wireless communication function of the mobile phone 10 can be realized by antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, and a modulation and demodulation processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the mobile phone 10 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antenna. For example, antenna 1 can be reused as a diversity antenna of a wireless local area network. In some other embodiments, antenna 1 can be used in combination with a tuning switch.

[0190] The mobile communication module 150 can provide wireless communication solutions including 2G / 3G / 4G / 5G etc. applied on the mobile phone 10 .

[0191] The screen 194 is used to display images, videos, etc. The screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), etc. In some embodiments, the mobile phone 10 may include 1 or N screens 194, where N is a positive integer greater than 1.

[0192] In some embodiments, screen 194 may display the video captured by the photographer.

[0193] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. In some embodiments, the mobile phone 10 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0194] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the mobile phone 10 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the mobile phone 10 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0195] The mobile phone 10 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor.

[0196] The gyro sensor 180A may be used to determine the motion posture of the mobile phone 10. In some embodiments, the angular velocity of the mobile phone 10 around three axes (ie, x, y, and z axes) may be determined by the gyro sensor 180A.

[0197] The magnetic sensor 180B can detect the strength of the magnetic field.

[0198] The acceleration sensor 180C can detect the magnitude of the acceleration of the mobile phone 10 in various directions (generally three axes). When the mobile phone 10 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching, pedometers, etc.

[0199] The infrared sensor 180D is used to detect, receive and analyze infrared radiation signals.

[0200] The temperature sensor 180E is used to detect temperature. In some embodiments, the mobile phone 10 uses the temperature detected by the temperature sensor 180E to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180E exceeds a threshold, the mobile phone 10 reduces the performance of a processor located near the temperature sensor 180E to reduce power consumption and implement thermal protection.

[0201] The touch sensor 180F is also called a "touch control device". The touch sensor 180F can be set on the screen 194, and the touch sensor 180F and the screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180F is used to detect touch operations acting on or near it.

[0202] The ambient light sensor 180G can be used to sense the brightness of ambient light.

[0203] The light sensor 180H can be used to detect light intensity. It has high sensitivity and fast response speed. It can be used in various lighting environments and provide real-time feedback on light conditions.

[0204] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The mobile phone 10 may receive key input and generate key signal input related to user settings and function control of the mobile phone 10.

[0205] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback.

[0206] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0207] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or disconnected from the mobile phone 10 by inserting the SIM card interface 195 or removing the SIM card interface 195. The mobile phone 10 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1.

[0208] The embodiments disclosed in the present application may be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application may be implemented as a computer program or program code executed on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0209] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor, a microcontroller, an application specific integrated circuit, or a microprocessor.

[0210] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0211] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0212] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented in a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0213] It should be noted that, in the examples and description of the present application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the statement "include one" do not exclude the existence of other identical elements in the process, method, article or equipment including the elements.

[0214] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the present application.

Claims

1. A shooting method, applied to an electronic device, characterized in that: The method comprises: A first shooting instruction is detected, wherein the first shooting instruction is used to instruct a first camera of the electronic device to enter a first shooting mode, and in the first shooting mode, a change in the size of a subject captured by the first camera during movement satisfies a shooting condition; At a first shooting moment, the subject photographed by the first camera has a first subject size, and the first camera has a first zoom ratio, and the first subject size includes a first subject frame size; At a second shooting moment, it is detected that the subject photographed by the first camera has a second subject size, and a difference between the first subject size and the second subject size does not satisfy the shooting condition, and the first zoom ratio is adjusted to a second zoom ratio, and the second subject size includes a second subject frame size; wherein the second zoom ratio is determined by: determining a first reference zoom ratio based on a difference between the first main body frame size and the second main body frame size; determining a second reference zoom ratio based on a relative movement speed of the first camera between the first shooting moment and the second shooting moment; determining the second zoom magnification based on the first reference zoom magnification and the second reference zoom magnification; The subject photographed by the first camera at the second zoom ratio has a third subject size, and a difference between the third subject size and the first subject size satisfies the shooting condition.

2. The method according to claim 1, characterized in that The main body dimensions include at least one of the following: The size of the subject frame and the proportion of the subject in the picture captured by the first camera; The main frame size includes the length and width of the main frame.

3. The method according to claim 2, characterized in that The subject size includes the first subject size and the second subject size, the first subject size also includes a first occupation ratio, and the second subject size also includes a second occupation ratio; and the shooting conditions include: An absolute value of a difference between the first main body frame size and the second main body frame size is less than a difference threshold; An absolute value of a difference between the first occupancy ratio and the second occupancy ratio is smaller than a ratio threshold.

4. The method according to claim 3, characterized in that The determining a first reference zoom ratio based on a difference between the first main body frame size and the second main body frame size includes: The difference between the first main body frame size and the second main body frame size is input into the PID algorithm function to obtain the first reference zoom ratio; wherein the PID algorithm function is constructed in the following formula: ， Wherein, Z is the first reference zoom ratio, e(t) is the difference between the first main body frame size and the second main body frame size, K p is the error proportionality coefficient, K p e(t) is the error ratio between the first main frame size and the second main frame size, K i is the error integral coefficient, t is each shooting moment between the first shooting moment and the second shooting moment, is the error integral between each main frame size corresponding to each shooting moment and the first main frame size, K d is the error differential coefficient, is the differential of the error between each main body frame size corresponding to each shooting moment and the first main body frame size.

5. The method according to claim 3, characterized in that: The determining a second reference zoom ratio based on the relative movement speed of the first camera between the first shooting moment and the second shooting moment includes: determining a relative movement speed of the first camera between the first shooting moment and the second shooting moment based on a change value of a spatial relative distance between the subject and the first camera; The second reference zoom ratio is generated through a Kalman filter model based on the first zoom ratio, the difference between the first occupancy ratio and the second occupancy ratio, and the relative movement speed corresponding to the first camera.

6. The method according to claim 5, characterized in that The determining, based on the change value of the spatial relative distance between the subject and the first camera, the relative movement speed of the first camera between the first shooting moment and the second shooting moment includes: At the first shooting moment, obtaining a first relative distance between the subject and the first camera; At the second shooting moment, obtaining a second relative distance between the subject and the first camera; Based on the first relative distance and the second relative distance, a relative movement speed of the first camera between the first shooting moment and the second shooting moment is determined.

7. The method according to claim 6, characterized in that The method further comprises: At the first shooting moment, a first subject mask corresponding to a first picture shot by the first camera is obtained, wherein each pixel in the first picture corresponds to each data in the first subject mask one by one, a subject pixel in the first picture corresponds to first data in the first subject mask, and a non-subject pixel in the first picture corresponds to second data in the first subject mask; The first occupancy ratio is determined based on the number of the first data and the number of data in the first body mask.

8. The method according to claim 7, characterized in that The electronic device further includes a second camera, and acquiring a first relative distance between the subject and the first camera at the first shooting moment includes: At the first shooting moment, controlling the second camera to shoot the subject to generate a second picture, wherein at the same shooting moment, the zoom ratio of the second camera is the same as the zoom ratio of the first camera; Determine a disparity corresponding to each pixel point in the first picture captured by the first camera, wherein the disparity corresponding to each pixel point in the first picture is a coordinate position difference between each pixel point in the first picture and a corresponding pixel point in the second picture; Determine depth information corresponding to each pixel in the first picture based on the parallax corresponding to each pixel in the first picture, the first zoom factor, first camera parameters, second camera parameters, and a spatial distance between the first camera and the second camera; Based on the pixel points corresponding to the first data in the first subject mask in the first picture, filtering out the depth information corresponding to each pixel point of the subject from the depth information corresponding to each pixel point of the first picture; A first relative distance between the subject and the first camera is determined based on depth information corresponding to each pixel of the subject.

9. The method according to claim 3, characterized in that: The determining the second zoom magnification based on the first reference zoom magnification and the second reference zoom magnification includes: Based on the weighted sum function R, the second zoom ratio is determined; wherein the weighted sum function R is constructed in the form of the following formula: ， Among them, R is the second zoom ratio, R1 is the first reference zoom ratio, w1 is the weight corresponding to the first reference zoom ratio, R2 is the second reference zoom ratio, and w2 is the weight corresponding to the second reference zoom ratio.

10. The method according to claim 1, characterized in that During the process in which the first camera adopts the first shooting mode to shoot, the method further includes: Obtaining the subject frames corresponding to the subject in two consecutive frames; Based on the position information of the matching feature points in the two subject frames in the two frames, determining the position difference of the subject captured by the first camera in the two frames; Based on the position difference and the sensor data collected by the gyroscope sensor of the electronic device, anti-shake processing is performed on the latter frame of the two frames.

11. The method according to claim 10, characterized in that The anti-shake processing includes one or more of moving the subject position, image compensation, and rotating the image.

12. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store instructions executed by one or more processors of the electronic device, and the processor is one of the one or more processors of the electronic device, and is used to execute the shooting method described in any one of claims 1 to 11.

13. A readable storage medium, characterized in that: The readable storage medium stores instructions, and when the instructions are executed on an electronic device, the electronic device executes the shooting method according to any one of claims 1 to 11.

14. A computer program product, characterized in that include: A computer instruction, when the computer instruction is executed on an electronic device, causes the electronic device to execute the shooting method according to any one of claims 1 to 11.

Images