Photographing method and device, electronic device and computer-readable storage medium

Abstract

The present application discloses a shooting method, including controlling the image sensor to move parallel to the imaging plane when the electronic device is in a stationary state, and collecting multiple frames of first images; and generating a target image based on the multiple frames of the first images. The shooting method, shooting device, electronic device and non-volatile computer-readable storage medium of the embodiments of the present application control the image sensor to move parallel to the imaging plane when the electronic device is in a stationary state, so that there will be an offset between the collected multiple frames of first images, and the collection point of the same target object (such as corresponding to a pixel) corresponds to filters of different colors in different frames, so that the first images of different frames can collect light of different color components of the same collection point, so that when generating the target image based on the multiple frames of the first image, there is no need to perform demosaicing processing through a demosaicing algorithm, and the accuracy of the pixel information is higher than the poor accuracy of the pixel information obtained by the demosaicing algorithm.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically, to a photographing method, a photographing device, an electronic device, and a non-volatile computer-readable storage medium. Background Art

[0002] At present, image sensors used for color imaging are generally equipped with filters, and each pixel can only receive light of one color. For example, the more common Bayer array filter, the pixel can only receive the red component, green component or blue component of the light. The missing pixel information needs to be supplemented by a demosaicing algorithm. However, the accuracy of the pixel information calculated by the demosaicing algorithm is low. Summary of the invention

[0003] Embodiments of the present application provide a photographing method, a photographing device, an electronic device, and a non-volatile computer-readable storage medium.

[0004] The shooting method of the embodiment of the present application includes controlling the image sensor to move parallel to the imaging plane and collecting multiple frames of first images when the electronic device is in a stationary state; and generating a target image based on the multiple frames of the first images.

[0005] The shooting device of the embodiment of the present application includes a control module and a generation module. The control module is used to control the image sensor to move parallel to the imaging plane and collect multiple frames of first images when the electronic device is in a stationary state; the generation module is used to generate a target image based on the multiple frames of the first images.

[0006] The electronic device of the embodiment of the present application includes a camera, a processor and a driving device, the camera includes an image sensor, when the electronic device is in a stationary state, the driving device drives the image sensor to move parallel to the imaging plane, and the image sensor captures multiple frames of first images; the processor is used to generate a target image based on the multiple frames of the first images.

[0007] The non-volatile computer-readable storage medium of the embodiment of the present application includes a computer program. When the computer program is executed by one or more processors, the processors execute the following shooting method: when the electronic device is in a stationary state, control the image sensor to move parallel to the imaging plane and capture multiple frames of first images; and generate a target image based on the multiple frames of the first images.

[0008] The shooting method, shooting device, electronic device and non-volatile computer-readable storage medium of the embodiments of the present application control the image sensor to move parallel to the imaging plane when the electronic device is in a stationary state, so that an offset will appear between the collected multiple frames of first images, and the collection point of the same target object (such as corresponding to a pixel) corresponds to filters of different colors in different frames, so that the first images of different frames can collect light of different color components of the same collection point. Therefore, when generating a target image based on the multiple frames of first images, there is no need to perform demosaicing through a demosaicing algorithm, but directly obtain the light of different color components of each collection point of the target object from the first images of different frames, and directly generate the demosaiced target image. Compared with the poor accuracy of pixel information obtained by the demosaicing algorithm, there is no need to perform demosaicing algorithm processing, which is beneficial to reducing pseudo color and zipper effect, and the accuracy of pixel information is higher.

[0009] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0011] Figure 1 is a flowchart of a shooting method in some embodiments of the present application;

[0012] Figure 2 is a schematic diagram of a photographing device according to some embodiments of the present application;

[0013] Figure 3 is a schematic plan view of an electronic device according to some embodiments of the present application;

[0014] Figure 4 and Figure 5 is a schematic diagram of a scene of a shooting method in some embodiments of the present application;

[0015] Figures 6 to 9 is a flowchart of a shooting method in some embodiments of the present application;

[0016] Fig.10 is a schematic diagram of a scene of a shooting method in some embodiments of the present application;

[0017] Fig.11 and Fig.12 is a flowchart of a shooting method in some embodiments of the present application;

[0018] Figures 13 to 15 is a schematic diagram of a scene of a shooting method in some embodiments of the present application;

[0019] Fig.16 It is a schematic diagram of the connection status of a non-volatile computer-readable storage medium and a processor in certain embodiments of the present application. DETAILED DESCRIPTION

[0020] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions from beginning to end. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of the present application, and cannot be understood as limiting the embodiments of the present application.

[0021] See also Figure 1 The present application provides a shooting method. The shooting method comprises the steps of:

[0022] 011: When the electronic device is in a stationary state, control the image sensor to move parallel to the imaging plane and collect multiple frames of first images; and

[0023] 012: Generate a target image according to multiple frames of first images.

[0024] See also Figure 2 , the embodiment of the present application provides a shooting device 10. The shooting device 10 includes a control module 11 and a generation module 12. The shooting method of the embodiment of the present application can be applied to the shooting device 10. The control module 11 and the generation module 12 are used to execute step 011 and step 012 respectively. That is, the control module 11 is used to control the image sensor to move parallel to the imaging surface when the electronic device is in a stationary state, and to collect multiple frames of first images, and the generation module 12 is used to generate a target image according to the multiple frames of first images.

[0025] See also Figure 3 , the embodiment of the present application also provides an electronic device 100. The electronic device 100 includes a processor 20, a camera 30 and a driving device 40, and the camera 30 includes an image sensor 31. The shooting method of the embodiment of the present application can be applied to the electronic device 100. The driving device 40 can cooperate with the image sensor 31 to perform step 011, and the processor 20 is used to perform step 012. That is, when the electronic device 100 is in a stationary state, the driving device 40 drives the image sensor 31 to move parallel to the imaging plane, and the image sensor 31 collects multiple frames of first images; the processor 20 is used to generate a target image based on the multiple frames of first images.

[0026] Specifically, the electronic device 100 further includes a housing 50. The electronic device 100 may be a mobile phone, a tablet computer, a display device, a laptop computer, a teller machine, a gate machine, a smart watch, a head-mounted display device, a game console, etc. Figure 3As shown, the embodiment of the present application is described by taking the electronic device 100 as a mobile phone as an example. It can be understood that the specific form of the electronic device 100 is not limited to a mobile phone. The housing 50 can also be used to install functional modules such as a display device, an imaging device, a power supply device, and a communication device of the electronic device 100, so that the housing 50 provides dustproof, drop-proof, waterproof and other protections for the functional modules.

[0027] The electronic device 100 also includes a posture sensor, which is used to collect posture data of the electronic device 100. For example, the posture sensor can be a gyroscope, which can collect the angular velocity of the electronic device 100 around the pitch axis, roll axis and yaw axis, or the posture sensor can be an accelerometer, which can collect the acceleration of the electronic device 100 around the pitch axis, roll axis and yaw axis.

[0028] Taking the attitude sensor as a gyroscope as an example, the processor 20 can determine whether the electronic device 100 is in a stationary state through the attitude data. It can be understood that the stationary state refers to the situation where the position of the electronic device 100 relative to the earth remains unchanged. At this time, the preset attitude threshold is 0. When the attitude data is equal to the preset attitude threshold, it can be determined that the electronic device 100 is in a stationary state; and the stationary state refers to the situation where the change in the position of the electronic device 100 relative to the earth is less than the preset difference threshold, which means that the speed of the electronic device 100 moving is extremely slow, and the scene ranges captured by the first images of different frames are basically the same, and there is basically no offset. For example, the preset attitude threshold can be 0.5, 1, 2, etc., and the attitude data can be less than the preset attitude threshold to determine that the electronic device 100 is in a stationary state. That is to say, when the pitch angular velocity, the roll angular velocity and the yaw angular velocity (because the pitch angular velocity, the roll angular velocity and the yaw angular velocity are directional and may be negative values, the absolute values ​​of the pitch angular velocity, the roll angular velocity and the yaw angular velocity are taken) are all less than the preset attitude threshold, it can be determined that the electronic device 100 is in a stationary state.

[0029] When the electronic device 100 is in a stationary state, for example, the electronic device 100 is in a tripod mode for shooting, the scene range of the multiple frames of first images shot by the image sensor 31 remains unchanged, there is no offset between the multiple frames of first images, and it is impossible to perform de-mosaic processing without interpolation through a de-mosaic algorithm. Therefore, when the electronic device 100 is in a stationary state, the processor 20 drives the image sensor 31 to move parallel to the imaging plane through the driving device 40, so that there is an offset between the multiple frames of first images captured by the image sensor 31. The offset between the multiple frames of first images means that, for the same acquisition point of the photographed scene, the pixel information of the acquisition point in the first images of different frames is different, and the pixel information of the acquisition point in the first images of different frames is generated by different color components in the light reflected by the acquisition point.

[0030] The driving device 40 can drive the image sensor 31 to move in multiple directions. For example, the driving device 40 can drive the image sensor 31 to move in a direction parallel to the imaging plane, or the driving device 40 can drive the image sensor 31 to move in a direction close to or away from the imaging plane to achieve focusing. It can be understood that in order to ensure that the scene range of the first image captured by the image sensor 31 is offset, the multiple directions of movement at least include the direction parallel to the imaging plane. When moving parallel to the imaging plane, the moving direction and moving distance of the image sensor 31 can be accurately controlled by the driving device 40.

[0031] The driving device 40 can be a voice coil motor or a micro pan-tilt head. The voice coil motor and the micro pan-tilt head can achieve high-precision movement of the image sensor 31, thereby achieving pixel-level offset of the image sensor 31, and the offset distance is less than the predetermined pixel distance, such as the offset distance is within 2 pixel distances.

[0032] For example, see Figure 4 , is a partial schematic diagram of the image sensor 31, and the Bayer array of the image sensor 31 is used as an example for explanation. The image sensor 31 includes an R channel, a G channel, and a B channel. When the image sensor 31 does not move, a frame of the first image P1 is captured. The first image P1 includes a sub-image R1, a sub-image G1, and a sub-image B1. The sub-image R1 includes R pixels and blank pixels, the sub-image G1 includes G pixels and blank pixels, and the sub-image B1 includes B pixels and blank pixels. When the driving device 40 drives the image sensor 31 to move in the horizontal direction (such as Figure 4 When the image sensor 31 moves one pixel to the right (horizontally as shown), at this time, the image sensor 31 collects another frame of the first image P1, and the first image P1 at this time includes the sub-image R2, the sub-image G2 and the sub-image B2; then the driving device 40 drives the image sensor 31 again along the vertical direction perpendicular to the horizontal direction (as shown in FIG. Figure 4 When the image sensor 31 moves one pixel in the horizontal direction (as shown in the figure), the image sensor 31 captures another frame of the first image P1. The first image P1 at this time includes the sub-image R3, the sub-image G3 and the sub-image B3. Finally, the driving device 40 drives the image sensor 31 to move one pixel in the horizontal direction (as shown in the figure). Figure 4 When the image sensor 31 moves one pixel horizontally to the left as shown, at this time, the image sensor 31 collects another frame of the first image P1. At this time, the first image P1 includes sub-image R4, sub-image G4 and sub-image B4; thereby, 4 frames of the first image P1 are obtained.

[0033] The first image may be a RAW image that has not been processed, that is, image data that has not been processed by any algorithm after being output by the image sensor 31 of the camera 30. Alternatively, the first image may be an image that has been processed by black level correction (BLC) and lens shading correction (LSC), thereby obtaining a first image with better image quality. Of course, the first image may also be processed by other algorithms other than the above-mentioned BLC and LSC to achieve better image quality.

[0034] After obtaining multiple frames of offset first images, the target image can be synthesized based on the multiple frames of first images. For example, see Figure 5 , in the 4 frames of the first image P1, the sub-images of the same channel (such as sub-images R1 to R4, sub-images G1 to G4, and sub-images B1 to B4) can respectively form a single-channel R channel image, G channel image, and B channel image without blank pixels, and then synthesize the R channel image, G channel image, and B channel image to generate the target image P0. Since the image sensor 31 moves along the parallel imaging plane when collecting multiple frames of the first image, when multiple pixels corresponding to the same acquisition point in the multiple frames of the first image receive the light reflected by the acquisition point, the colors of the filters corresponding to the multiple pixels are different, so that the multiple pixels can obtain different color components of the light reflected by the acquisition point. Therefore, after the aligned multiple frames of the first image are synthesized, each pixel contains pixel information of different color components, so that the target image after demosaicing can be generated without the need for demosaicing.

[0035] The shooting method, shooting device 10 and electronic device 100 of the embodiment of the present application control the image sensor 31 to move parallel to the imaging plane when the electronic device 100 is in a stationary state, so that an offset will appear between the collected multiple frames of first images, and the collection point of the same target object (such as corresponding to a pixel) corresponds to filters of different colors in different frames, so that the first images of different frames can collect light of different color components of the same collection point. Therefore, when generating a target image based on the multiple frames of first images, there is no need to perform demosaicing through a demosaicing algorithm, but directly obtain the light of different color components of each collection point of the target object from the first images of different frames, and directly generate the demosaiced target image. Compared with the poor accuracy of pixel information obtained by the demosaicing algorithm, there is no need to perform demosaicing algorithm processing, which is beneficial to reducing pseudo color and zipper effect, and the accuracy of pixel information is higher.

[0036] See also Figure 2 , Figure 3 and Figure 6 In some embodiments, the shooting method further comprises:

[0037] 013: When the electronic device 100 is in motion, multiple frames of first images are captured.

[0038] In some embodiments, the photographing device 10 further includes a collection module 13, and the collection module 13 is used to perform step 013. That is, the collection module 13 is used to collect multiple frames of first images when the electronic device 100 is in motion.

[0039] In some implementations, the image sensor 31 is further configured to execute step 013. That is, the image sensor 31 is further configured to capture multiple frames of first images when the electronic device 100 is in motion.

[0040] Specifically, the processor 20 can determine whether the electronic device 100 is in motion through the posture data. For example, the processor 20 determines whether the posture data (such as pitch angular velocity, roll angular velocity or yaw angular velocity) is greater than a preset posture threshold value. The preset posture threshold value can be 0, 0.5, 1, 2, etc. When the posture data is greater than the preset posture threshold value, it is determined that the electronic device 100 is in motion. In other embodiments, in order to ensure the diversity of the offsets of the first images of different frames, when determining that the electronic device 100 is in motion, it can be determined whether the pitch angular velocity, roll angular velocity and yaw angular velocity are all greater than the preset posture threshold value, or at least two of the pitch angular velocity, roll angular velocity and yaw angular velocity are greater than the preset posture threshold value, thereby ensuring that there are offsets in multiple directions in multiple frames of first images, and corresponding to the filter color arrangement of the image sensor, the light of each acquisition point can be received by filters of at least three colors, thereby ensuring that an accurate demosaiced target image can be generated subsequently.

[0041] When the electronic device 100 is in motion, there is an offset between the multiple frames of first images captured by the image sensor 31. Therefore, at this time, there is no need to control the driving device 40 to drive the image sensor 31 to move parallel to the imaging plane, so that multiple frames of first images with offsets can be obtained. Then, the processor 20 can synthesize the target image based on the multiple frames of images with offsets. The principle of synthesizing the target image from the multiple frames of first images acquired in the motion state is similar to the principle of synthesizing the target image from the multiple frames of first images acquired in the static state, and will not be repeated here.

[0042] See also Figure 2 , Figure 3 , Figure 7 and Figure 8 In some embodiments, the shooting method further comprises:

[0043] 014: when the posture data is greater than a preset posture threshold, determining that the electronic device 100 is in motion; or

[0044] 015: When the posture data is greater than the preset posture threshold, the inter-frame information of the multiple-frame captured images is obtained;

[0045] 016: When the inter-frame information is within a preset range, it is determined that the electronic device 100 is in motion.

[0046] In some embodiments, the photographing device 10 further includes a determination module 14, and the first determination module 14 is used to execute step 014, step 015, and step 016. That is, the determination module 14 is used to determine that the electronic device 100 is in motion when the posture data is greater than a preset posture threshold; or to obtain inter-frame information of multiple frames of captured images when the posture data is greater than a preset posture threshold; and to determine that the electronic device 100 is in motion when the inter-frame information is within a preset range.

[0047] In some embodiments, the processor 20 is further configured to execute step 014, step 015, and step 016. That is, the processor 20 is further configured to determine that the electronic device 100 is in motion when the posture data is greater than a preset posture threshold; or to obtain inter-frame information of multiple frames of captured images when the posture data is greater than a preset posture threshold; and to determine that the electronic device 100 is in motion when the inter-frame information is within a preset range.

[0048] Specifically, the processor 20 can directly determine whether the electronic device 100 is in motion through posture data. For example, the processor 20 determines whether the posture data (such as pitch angular velocity, roll angular velocity or yaw angular velocity) is greater than a preset posture threshold. The preset posture threshold can be 0, 0.5, 1, 2, etc. When the posture data is greater than the preset posture threshold, it is determined that the electronic device 100 is in motion.

[0049] However, it is possible that the electronic device 100 and the subject are on the same moving object at the same time (for example, in a smoothly moving subway, the electronic device 100 is set on a tripod and the subject is photographed in tripod mode), and the relative positions of the electronic device 100 and the subject have not changed. At this time, the electronic device 100 has moved relative to the earth and is therefore in a moving state, but there is no offset between the multiple frames of first images captured by the image sensor 31. Therefore, it is impossible to generate a demosaiced target image without interpolation through a demosaicing algorithm.

[0050] Therefore, when it is determined that the posture data is greater than the preset posture threshold, it is necessary to further determine whether the relative position of the electronic device 100 and the subject is unchanged. The processor 20 can obtain multiple frames of captured images, and then calculate the inter-frame information of the multiple frames of captured images. The inter-frame information can be determined by any two frames of images in the multiple frames of captured images. For example, the sum of the difference in pixel values ​​at corresponding positions of any two frames of images in the multiple frames is calculated as the inter-frame information. When the relative position of the electronic device 100 and the subject is unchanged, the two frames of images are basically the same, so the inter-frame information is basically 0 or the inter-frame information is within a preset range, such as the inter-frame information is less than the preset difference threshold (such as the preset difference threshold is 5, 10, etc.). Therefore, by determining whether the inter-frame information is within the preset range, it is possible to accurately determine whether the relative position of the electronic device 100 and the subject has changed, so that when the inter-frame information is within the preset range, it is determined that the electronic device 100 is in a moving state. And the judgment of the inter-frame information is only performed when the posture data that can be directly obtained is greater than the preset posture threshold, so as to prevent the calculation of the inter-frame information in the static state, thereby reducing the time required for the state judgment of the electronic device 100 and improving the shooting efficiency.

[0051] See also Figure 2 , Figure 3 and Fig. 9 In some embodiments, step 012 includes:

[0052] 0121: selecting a first image frame from multiple first image frames according to a preset strategy as a reference frame, and defining the remaining first images as non-reference frames;

[0053] 0122: Calculate the pixel deviation and similarity of the reference frame and each non-reference frame respectively;

[0054] 0123: generating a deviation function according to the pixel deviation and the similarity, determining the maximum similarity in the deviation function based on the deviation function, and aligning the non-reference frame with the reference frame in the plurality of first image frames according to the pixel deviation corresponding to the maximum similarity;

[0055] 0124: Synthesize the aligned multiple frames of the first image to generate the target image.

[0056] In some embodiments, the generation module 12 is further used to execute step 0121, step 0122, step 0123, and step 0124. That is, the generation module 12 is used to select a first image frame from multiple first image frames according to a preset strategy and define the remaining first images as non-reference frames, respectively calculate the pixel deviation and similarity of the reference frame and each non-reference frame, and generate a deviation function according to the pixel deviation and similarity, and determine the maximum similarity in the deviation function based on the deviation function, and align the non-reference frame with the reference frame in the multiple first image frames according to the pixel deviation corresponding to the maximum similarity; synthesize the aligned multiple first image frames to generate the target image.

[0057] In some embodiments, the processor 20 is further configured to execute step 0121, step 0122, step 0123, and step 0124. That is, the processor 20 is configured to select a first image frame from multiple first image frames according to a preset strategy and define the remaining first images as non-reference frames; respectively calculate the pixel deviation and similarity of the reference frame and each non-reference frame; and generate a deviation function according to the pixel deviation and similarity, and determine the maximum similarity in the deviation function based on the deviation function, and align the non-reference frame with the reference frame in the multiple first image frames according to the pixel deviation corresponding to the maximum similarity; synthesize the aligned multiple first image frames to generate the target image.

[0058] Specifically, before aligning multiple frames of first images, the processor 20 needs to first select one of the multiple frames of first images as a reference frame according to a preset strategy, and the remaining first images as non-reference frames, so as to determine the reference frame and non-reference frame in the multiple frames of first images, and then align them based on the reference frame. The preset strategy can be: first calculate the sharpness of each first image, and the sharpness can be determined based on the gradient of the first image in the horizontal and vertical directions, and then select the first image with the largest sharpness as the reference frame, and use the other first images in the multiple frames of first images except the reference frame as non-reference frames, and then use the reference frame as the reference to align the non-reference frames with the reference frame.

[0059] More specifically, the processor 20 first calculates the pixel deviation and similarity of the reference frame and each non-reference frame respectively. Since there are multiple frames of the first image, multiple sets of corresponding relationships between pixel deviations and similarities can be obtained.

[0060] like Fig.10As shown, taking two frames of the first image as an example, the processor 20 can first identify the first pixel L1 of the reference frame P11, then identify the second pixel L2 corresponding to the first pixel L1 in the non-reference frame, and obtain the pixel deviation between the first pixel L1 and the second pixel L2, and then obtain the similarity between the first feature point L1 and the second feature point L2 by the following formula (1). The pixel deviation is the position deviation between the reference frame and the non-reference frame. Since the reference frame and the non-reference frame in the multiple frames of the first image have been determined, the processor 20 can obtain the pixel deviation between the reference frame and the non-reference frame by obtaining the position deviation between the first pixel L1 and the second pixel L2.

[0061]

[0062] Where D(u, v) represents the pixel deviation between the reference frame and the non-reference frame, and δ is an artificially set parameter used to adjust the similarity. From formula (1), it can be seen that when the pixel deviation D(u, v) is larger, the similarity Wij is smaller.

[0063] In summary, the processor 20 can obtain the pixel deviation D(u, v) between the reference frame and the non-reference frame through the position deviation between the first pixel L1 on the reference frame P11 and the second pixel L2 on the non-reference frame P12, and substitute D(u, v) into formula (1) to obtain the similarity W between the reference frame and the non-reference frame.

[0064] Next, the processor 20 obtains the pixel deviation and similarity between the reference frame and each non-reference frame through formula (1).

[0065] Therefore, the processor 20 can construct a deviation function as shown in the following formula (2) through multiple sets of pixel deviations and similarities.

[0066]

[0067] Wherein, I(x, y) is the pixel position in the reference frame, u and v represent the horizontal and vertical offsets of the non-reference frame relative to the reference frame, respectively, and W represents the similarity between the position of I(x, y) in the reference frame and the corresponding position in the non-reference frame.

[0068] Therefore, the maximum value of W (similarity) in the deviation function can be obtained through formula (2), so as to obtain the pixel deviation D (u, v) corresponding to the maximum value W, and according to the pixel deviation D (u, v) corresponding to the maximum value W, the non-reference frame in the multiple frames of the first image is aligned with the reference frame.

[0069] Currently, when aligning multiple image frames, only pixel-by-pixel or region-by-region traversal is performed to obtain the position offset between the non-reference frame and the reference frame at the corresponding position, thereby aligning the multiple image frames. Therefore, the alignment strategy based on this method can only achieve pixel-level alignment accuracy (i.e., integer value).

[0070] In this embodiment, an offset function is constructed regarding the pixel offset and similarity between the reference frame and the non-reference frame, and the pixel offset corresponding to the maximum similarity is used to align the multiple frames of the first image. When the similarity of the offset function is the largest, the accuracy of the pixel offset calculated according to the maximum similarity of the offset function can be accurate to less than 1 pixel, that is, sub-pixel level, which improves the alignment accuracy. Therefore, by aligning the multiple frames of the first image with the pixel offset when the similarity is the largest, the alignment error can be reduced.

[0071] See also Figure 2 , Figure 3 and Fig.11 In some embodiments, step 0121: selecting a first image frame from multiple first image frames according to a preset strategy as a reference frame, and defining the remaining first images as non-reference frames, includes the steps of:

[0072] 01211: Calculate the sharpness of the first images of the plurality of frames respectively, wherein the sharpness is determined according to the gradients of the first images in the horizontal direction and the vertical direction;

[0073] 01212: Determine the first image with the greatest sharpness as the reference frame; and

[0074] 01213: Determine the first image other than the reference frame as a non-reference frame.

[0075] In some embodiments, the generating module 12 is further configured to execute step 01211, step 01212, and step 01213. That is, the generating module 12 is configured to respectively calculate the sharpness of the multiple frames of the first image, the sharpness being determined according to the gradients of the first image in the horizontal direction and the vertical direction; determine the first image with the largest sharpness as the reference frame; and determine the first images other than the reference frame as non-reference frames.

[0076] In some embodiments, the processor 20 is used to perform step 01211, step 01212, and step 01213. That is, the processor 20 is used to respectively calculate the sharpness of multiple frames of first images, where the sharpness is determined according to the gradients of the first images in the horizontal and vertical directions; determine the first image with the largest sharpness as the reference frame; and determine the first images other than the reference frame as non-reference frames.

[0077] Specifically, the processor 20 can calculate the sharpness of the first images of the multiple frames respectively, and the sharpness is determined according to the horizontal and vertical gradients of the first image. The sharpness can directly reflect the clarity of the image. When the sharpness is within a certain range, the greater the sharpness, the better the image quality.

[0078] Therefore, the sharpness of multiple frames of first images can be calculated, and the first image with the largest sharpness can be used as a reference frame, and the first images other than the reference frame can be non-reference frames.

[0079] More specifically, the sharpness of the first image can be calculated according to the following formula (3):

[0080]

[0081] Wherein, Ix is the gradient of the first image in the horizontal direction, Iy is the gradient of the first image in the vertical direction, and S is the sharpness of the first image.

[0082] See also Figure 2 , Figure 3 and Fig.12 , the shooting method of the embodiment of the present application, step 012 also includes the steps of:

[0083] 0125: extracting image data of channels of the same preset type in the first image to generate a plurality of second images with single channels; and

[0084] 0126: Processing the single-channel second image to generate a third image, the size of the third image is larger than the second image;

[0085] 0127: synthesize multiple third images to generate a target image, the third image includes channel pixels and filling pixels, the filling pixels are obtained by blank pixels according to the channel pixels, including:

[0086] 01271: polling channel pixels and blank pixels through a polling frame of a preset size, and calculating the weight of the channel pixels according to the position offset of the channel pixels and the blank pixels in the polling frame and the gradient information of the polling frame; and

[0087] 01272: Fill blank pixels with fill pixels based on the pixel values ​​and weights of the channel pixels in the polling box.

[0088] In some embodiments, the generation module 12 is used to perform step 0125, step 0126, step 0127, step 01271 and step 01272. That is, the generation module 12 is used to extract image data of the same preset type of channel in the first image to generate multiple single-channel second images; and process the single-channel second images to generate a third image, the size of the third image is larger than the second image; synthesize multiple third images to generate a target image, the third image includes channel pixels and filling pixels, and the filling pixels are obtained from blank pixels according to the channel pixels, including: polling channel pixels and blank pixels through a polling frame of a preset size, and calculating the weight of the channel pixels according to the position offset of the channel pixels and the blank pixels in the polling frame and the gradient information of the polling frame; and filling the blank pixels with filling pixels according to the pixel values ​​and weights of the channel pixels in the polling frame.

[0089] In some embodiments, the processor 20 is used to perform step 0125, step 0126, step 0127, step 01271 and step 01272. That is, the processor 20 is used to extract image data of the same preset type of channel in the first image to generate multiple single-channel second images; and process the single-channel second images to generate a third image, the size of the third image is larger than the second image; synthesize multiple third images to generate a target image, the third image includes channel pixels and filling pixels, and the filling pixels are obtained from blank pixels according to the channel pixels, including: polling the channel pixels and blank pixels through a polling frame of a preset size, and calculating the weight of the channel pixels according to the position offset of the channel pixels and the blank pixels in the polling frame and the gradient information of the polling frame; and filling the blank pixels with filling pixels according to the pixel values ​​and weights of the channel pixels in the polling frame.

[0090] Specifically, the processor 20 may further extract image data of the same preset type of channel in the multiple frames of first images, and fuse the extracted image data to generate multiple single-channel second images.

[0091] Among them, the preset type channel of the first image includes image data of multiple channels. For example, for a filter arranged in a conventional Bayer array, the number of preset type channels can be determined according to the color, such as the preset type channels of the first image include R channel, G channel and B channel, and the number of pixels of R channel, G channel and B channel is 1:2:1; or the number of preset type channels can be determined according to the pixels contained in the basic unit constituting the filter. The basic unit of the Bayer array includes 4 pixels, that is, the first image includes R channel, G1 channel, G2 channel and B channel, wherein the bands of G1 and G2 may be the same or partially overlapped; or, the first image includes R channel, G channel, B channel and W channel, and the number of pixels of R channel, G channel, B channel and W channel is the same; of course, the channel distribution of the first image is not limited to the above two methods, and is not limited here. In the embodiment of the present application, the first image includes R channel, G1 channel, G2 channel and B channel.

[0092] Thus, the processor 20 can determine the number of extracted second images according to the number of channels of the smallest repeating unit in the first image. For example, if the smallest repeating unit in the first image includes R channel, G1 channel, G2 channel and B channel, a total of 4 channels, then the number of second images is 4. The processor 20 can also determine the number of second images according to the number of colors included in the first image. For example, if the first image includes three colors of R, G, and B, then the number of second images is 3. For another example, if the third image includes four colors of R, G, B, and W, then the number of second images is 4.

[0093] It should be noted that if the minimum repeating unit of the first image contains 4 channels, 4 single-channel data images can be extracted from each frame of the first image, and 4*N single-channel second images can be extracted from N first images.

[0094] When processing the second image of a single channel, the second image may be magnified by a predetermined multiple, so that the resolution of the generated third image is magnified by a predetermined multiple, such as 2 times, 3 times, 4 times or more. Fig.13 As shown, the second image is magnified by 2 times to obtain the following Fig.13 The third image shown in the right figure includes channel pixels having the same pixel values ​​as each pixel in the second image, and filling pixels that are not filled with pixel values, i.e., blank pixels. The pixel values ​​of the blank pixels can be obtained according to the pixel values ​​of the channel pixels, thereby obtaining the pixel values ​​of the filling pixels.

[0095] Among them, the single-channel second image includes an image of the R (red) channel, an image of the G (green) channel, an image of the B (blue) channel, an image of the W (white) channel, etc. When the second image of the R channel is processed, the generated third image is an enlarged image of the second image of the R channel; when the second image of the G channel is processed, the generated third image is an enlarged image of the second image of the G channel; when the second image of the B channel is processed, the generated third image is an enlarged image of the second image of the B channel. Therefore, when the second image contains a plurality of single-channel images of different colors, the generated third image is a plurality of single-channel images of different colors.

[0096] Next, the processor 20 may poll the channel pixels and blank pixels in the third image through the polling frame, so as to calculate the weight of the channel pixels according to the position offsets of the channel pixels and blank pixels in the same polling frame and the gradient information of the polling frame.

[0097] Specifically, Fig.14 As shown, a coordinate system is established with the upper left corner of the third image as the origin, and S is a polling frame. When the polling frame S polls to pixel N, the channel pixels located in the polling frame S are A1, A2, A4 and A5 respectively. According to the coordinate system, the position offset of the channel pixels and the blank pixels can be obtained. Taking one pixel as a unit on the coordinate axis as an example, the position change of the channel pixel A1 relative to the blank pixel is (1, 1), the position change of the channel pixel A2 relative to the blank pixel is (-1, 1), the position change of the channel pixel A4 relative to the blank pixel is (1, -1), and the position change of the channel pixel A5 relative to the blank pixel is (-1, -1). The gradient information of the polling frame is the gradient information of the third image corresponding to the polling frame, which can be obtained according to the gradient value of the channel pixels in the polling frame. Therefore, the weights corresponding to all channel pixels can be obtained according to the following formula (4). Among them, the position offset of the channel pixel and the blank pixel can be obtained according to the following formula (5). For example, taking the polling frame S as an example, the position offset d1 between the channel pixel A1 and the blank pixel is [1, 1]T, which is substituted into formula (4) to obtain the weight W1 of the channel pixel A1. Thus, the weight W1 of the channel pixel A1, the weight W2 of the channel pixel A2, the weight W4 of the channel pixel A4, and the weight W5 of the channel pixel A2 can be obtained respectively, that is, the weights of all the channel pixels in the polling frame S are obtained.

[0098]

[0099] d i =[x i -x 0 ,y i -y 0 ] T (5)

[0100] Among them, Wi is the weight of the channel pixel at the i-th pixel position, di is the position offset between the channel pixel at the i-th pixel position and the blank pixel, C is the covariance matrix constructed according to the gradient information of the third image corresponding to the polling frame, T is the transposition sign, R(x, y) is the pixel value of the blank pixel polled by the polling frame, and Vi is the pixel value of the channel pixel at the i-th pixel position.

[0101] After aligning M frames of images to extract M*N second images, the M*N second images can generate multiple third images. Therefore, the second images of the same preset type of channels will correspond to multiple third images. For example, two frames of third images containing R channel, G1 channel, G2 channel and B channel will generate two R channel second images, two G1 channel second images, two G2 channel second images and two B channel second images. The second images of the same channel are the second images of the same preset type.

[0102] Taking two R-channel second images as an example, there are two corresponding R-channel third images. Therefore, the processor 20 can calculate the pixel value of the blank pixel according to the pixel value and weight of the channel pixel in the polling frame corresponding to the position in the two third images, such as Fig.15 As shown, P3 and P4 represent a third image respectively. When the polling frame S2 polls the blank pixel N1 in the third image P3, it is necessary to calculate the pixel value of the blank pixel N1. Then, it is necessary to calculate the pixel value of the blank pixel N1 according to the pixel values ​​and corresponding weights of the channel pixels R1, R2, R4 and R5 in the polling frame S2 of the third image P3 (calculated according to the above formula (4)), and the pixel values ​​and corresponding weights of the channel pixels R1', R2', R4' and R5' in the polling frame S3 of the third image P4, and according to the following formula (6).

[0103]

[0104] Here, m is the number of frames M of the third image (as can be seen from the above, the number of frames M of the third image is consistent with the number of third images).

[0105] It should be noted that the polling frame S2 in the third image P3 is at the same position as the polling frame S3 in the third image P4, and the positions of the blank pixel N1 and the blank pixel N2 are the same. It can be seen from formula (6) that the calculation parameters of the blank pixels at the same position in multiple third images are the same, and therefore, the pixel values ​​of the blank pixels at the same position in multiple third images are the same. The processor 20 only needs to calculate the pixel value of the blank pixel in one third image, and then the pixel value of the blank pixel in all the third images can be obtained, so as to fill the pixel value of the blank pixel with the filling pixel to obtain a third image with complete pixel filling.

[0106] It can be understood that according to the blank pixels obtained in the above manner, the pixel values ​​of the blank pixels at the same position in the multiple third images are the same, so the pixel values ​​of the filled pixels in the multiple third images are the same. The processor 20 can generate a fourth image based on the pixel values ​​of the filled pixels and the pixel values ​​of the channel pixels in the third image corresponding to the reference frame. In this way, a fourth image of the R channel, a fourth image of the G1 channel, a fourth image of the G2 channel, and a fourth image of the B channel can be obtained.

[0107] The reference frame is an image with the greatest sharpness, and the fourth image generated according to the pixel values ​​of the filling pixels and the pixel values ​​of the channel pixels in the third image corresponding to the reference frame is closer to the gradient information of the first image.

[0108] Since the filling of blank pixels refers to the image information of a single channel corresponding to multiple frames of the offset first image, the gradient information of multiple frames of the third image is integrated, so that the pixel information of the third image is more accurate. Therefore, there is no need to perform demosaicing through a demosaicing algorithm. Instead, the fourth image of the R channel, the fourth image of the G1 channel, the fourth image of the G2 channel and the fourth image of the B channel are directly integrated to generate the demosaiced target image.

[0109] In some embodiments, after obtaining the target image, the processor 20 may further perform image processing operations on the target image to further improve the image quality of the target image, wherein the image processing operations include at least one of gamma correction, color correction and sharpening operations.

[0110] Then, the processor 20 may perform at least one of gamma correction, color correction and sharpening operations on the target image. For example, if the processor 20 performs gamma correction on the target image, the brightness of the target image can be improved. For example, if the processor 20 performs color correction on the target image, the image color of the target image can be made more consistent with the actual situation. For example, if the processor 20 performs sharpening operations on the target image, the clarity of the target image can be improved. The processor 20 may perform image processing for de-mosaicing according to the actual situation of the generated target image. For example, if the brightness of the target image is low, gamma correction is performed. For example, if the color of the target image is distorted, color correction is performed. If the clarity of the target image is low, sharpening operations are performed. In this way, a target image with better image quality can be obtained to obtain a good visual experience.

[0111] See also Fig.16 The embodiment of the present application further provides a non-volatile computer-readable storage medium 200 including a computer program 201. When the computer program 201 is executed by one or more processors 20, the one or more processors 20 execute the shooting method of any of the above embodiments.

[0112] For example, when the computer program 201 is executed by one or more processors 20, the processor 20 performs the following shooting method:

[0113] 011: When the electronic device is in a stationary state, control the image sensor to move parallel to the imaging plane and collect multiple frames of first images; and

[0114] 012: Generate a target image according to multiple frames of first images.

[0115] For another example, when the computer program 201 is executed by one or more processors 20, the processor 20 performs the following shooting method:

[0116] 013: When the electronic device 100 is in motion, multiple frames of first images are captured.

[0117] In the description of this specification, the descriptions with reference to the terms "certain embodiments", "in an example", "exemplarily", etc., mean that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0118] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0119] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A shooting method, characterized in that: include: When the electronic device is in a stationary state, controlling the image sensor to move parallel to the imaging plane and collecting multiple frames of the first image; and According to a preset strategy, one frame of the first image is selected from multiple frames of the first image to be defined as a reference frame, and the remaining first images are defined as non-reference frames; Calculating pixel deviation and similarity of the reference frame and each non-reference frame respectively; generating a deviation function according to the pixel deviation and the similarity, determining the maximum similarity in the deviation function based on the deviation function, and aligning the non-reference frame with the reference frame in the plurality of frames of the first image according to the pixel deviation corresponding to the maximum similarity, The pixel deviation is expressed as follows: , D(u, v) represents the pixel deviation between the reference frame and the non-reference frame, and δ is used to adjust the similarity. The deviation function is expressed as follows: I (x, y) is the pixel position in the reference frame, u and v represent the horizontal and vertical offsets of the non-reference frame relative to the reference frame, and W represents the similarity between the position I (x, y) in the reference frame and the corresponding position in the non-reference frame; The aligned multiple frames of the first images are synthesized to generate a target image.

2. The shooting method according to claim 1, characterized in that: Also includes: When the electronic device is in motion, a plurality of frames of the first image are collected.

3. The shooting method according to claim 1, characterized in that: The electronic device further includes a posture sensor, and the posture sensor is used to collect posture data. The shooting method further includes: Acquiring the posture data; and When the posture data is less than a preset posture threshold, it is determined that the electronic device is in the stationary state.

4. The shooting method according to claim 3, characterized in that: The posture data includes a pitch angular velocity, a roll angular velocity, and a yaw angular velocity, and when the posture data is less than a preset posture threshold, determining that the electronic device is in the stationary state includes: When the pitch angular velocity, the roll angular velocity, and the yaw angular velocity are all less than the preset attitude threshold, it is determined that the electronic device is in the stationary state.

5. The shooting method according to claim 2, characterized in that: The electronic device further includes a posture sensor, and the posture sensor is used to collect posture data. The shooting method further includes: When the posture data is greater than a preset posture threshold, determining that the electronic device is in the motion state; or When the posture data is greater than the preset posture threshold, obtaining inter-frame information of multiple frames of acquired images; When the inter-frame information is within a preset range, it is determined that the electronic device is in the motion state.

6. The shooting method according to claim 1, characterized in that: The controlling the image sensor to move parallel to the imaging plane includes: controlling the image sensor to move along a plurality of movement directions, wherein the plurality of movement directions at least include a direction parallel to the imaging plane.

7. The shooting method according to claim 1, characterized in that: The electronic device further comprises a driving device, wherein the driving device comprises a voice coil motor or a micro pan-tilt head, and the controlling the image sensor to move parallel to the imaging plane comprises: The driving device is controlled to drive the image sensor to move parallel to the imaging surface.

8. The shooting method according to claim 1, characterized in that: The selecting one first image frame from multiple first image frames according to a preset strategy as the reference frame and defining the remaining first images as non-reference frames includes: Calculating the sharpness of the first image of multiple frames respectively, wherein the sharpness is determined according to the gradients of the first image in the horizontal direction and the vertical direction; Determine the first image with the largest sharpness as the reference frame; and The first images other than the reference frame are determined as the non-reference frames.

9. The shooting method according to claim 1, characterized in that: The method further comprises: Extracting image data of channels of the same preset type in the first image to generate a plurality of single-channel second images; and Processing the single-channel second image to generate a third image, wherein the third image has a larger size than the second image; Synthesizing a plurality of the third images to generate the target image, wherein the third image includes channel pixels and filling pixels, and the filling pixels are obtained from blank pixels according to the channel pixels, comprising: Polling the channel pixels and the blank pixels through a polling frame of a preset size, and calculating the weights of the channel pixels according to the position offsets of the channel pixels and the blank pixels in the polling frame and the gradient information of the polling frame; and According to the pixel values ​​and weights of the channel pixels in multiple polling boxes in multiple third images corresponding to the second image of the same preset type channel, the pixel values ​​of the blank pixels are calculated, and the pixel values ​​of the blank pixels are filled with the filling pixels. The multiple polling boxes correspond to multiple third images respectively and have the same position in the multiple second images.

10. An image acquisition device, characterized in that: include: A first acquisition module, used to control the image sensor to move parallel to the imaging plane and acquire multiple frames of first images when the electronic device is in a stationary state; A generation module is used to select one frame of the first image from multiple frames of the first image according to a preset strategy and define it as a reference frame, and define the remaining first images as non-reference frames; respectively calculate the pixel deviation and similarity of the reference frame and each frame of the non-reference frame; generate a deviation function according to the pixel deviation and the similarity, and determine the maximum similarity in the deviation function based on the deviation function, and align the non-reference frame in the multiple frames of the first image with the reference frame according to the pixel deviation corresponding to the maximum similarity, wherein the pixel deviation is expressed by the following formula: , D(u, v) represents the pixel deviation between the reference frame and the non-reference frame, δ is used to adjust the similarity, and the deviation function is expressed as follows: I (x, y) is the pixel position in the reference frame, u and v represent the horizontal and vertical offsets of the non-reference frame relative to the reference frame, and W represents the similarity between the position I (x, y) in the reference frame and the corresponding position in the non-reference frame; The aligned multiple frames of the first images are synthesized to generate a target image.

11. An electronic device, characterized in that: The electronic device comprises a camera, a processor and a driving device, wherein the camera comprises an image sensor, and when the electronic device is in a stationary state, the driving device drives the image sensor to move parallel to an imaging surface, and the image sensor collects a plurality of frames of first images; The processor is used to select one frame of the first image from multiple frames of the first image according to a preset strategy and define it as a reference frame, and define the remaining first images as non-reference frames; respectively calculate the pixel deviation and similarity of the reference frame and each frame of the non-reference frame; generate a deviation function according to the pixel deviation and the similarity, and determine the maximum similarity in the deviation function based on the deviation function, and align the non-reference frame in the multiple frames of the first image with the reference frame according to the pixel deviation corresponding to the maximum similarity, wherein the pixel deviation is expressed by the following formula: , D(u, v) represents the pixel deviation between the reference frame and the non-reference frame, δ is used to adjust the similarity, and the deviation function is expressed as follows: I (x, y) is the pixel position in the reference frame, u and v represent the horizontal and vertical offsets of the non-reference frame relative to the reference frame, and W represents the similarity between the position I (x, y) in the reference frame and the corresponding position in the non-reference frame; The aligned multiple frames of the first images are synthesized to generate a target image.

12. A non-volatile computer-readable storage medium containing a computer program, wherein when the computer program is executed by a processor, the processor executes the shooting method according to any one of claims 1 to 9.

Images