Image processing method and device, storage medium and electronic equipment

Abstract

The invention relates to the technical field of images, in particular to an image processing method and device, a computer readable storage medium and electronic equipment, and the method comprises the steps: obtaining a preview image in a moon shooting scene, and shortening the exposure time during shooting when detecting that the preview image comprises a target object; acquiring a motion state of the electronic equipment, and determining a reference frame image from the preview image when the state of the electronic equipment is responded to be a stable state; and determining motion information of the current frame image relative to the reference frame image, and performing image deformation on the current frame image based on the motion information to obtain a target image. According to the technical scheme provided by the embodiment of the invention, the problem that the obtained image is not clear when the moon is shot is solved.

Description

Background Art

[0002] Electronic devices usually shake during the shooting process, which can cause unclear images. Therefore, electronic devices use EIS (Electronic Image Stabilizing) to better suppress the impact of shaking.

[0003] However, at a telephoto magnification, the same small amplitude shake also has a huge impact on the picture. When shooting the moon, a telephoto magnification is usually used. Even if EIS technology is used for image processing, there will still be problems with the captured image not being clear enough, and there will also be unclear bright spots on the edge of the moon in the image, resulting in poor image effect. Summary of the invention

[0004] The purpose of the present disclosure is to provide an image processing method, an image processing device, an electronic device and a computer-readable medium, which solve the problem of unclear images obtained when photographing the moon.

[0005] According to a first aspect of the present disclosure, there is provided an image processing method, comprising: obtaining a preview image in a moon shooting scene, and shortening a shooting exposure time when it is detected that the preview image includes a target object; obtaining a state of the electronic device, and in response to the state of the electronic device being a stable state, determining a reference frame image from the preview image; determining motion information of a current frame image relative to the reference frame image, and based on the motion information, performing image deformation on the current frame image to obtain a target image.

[0006] According to a second aspect of the present disclosure, there is provided an image processing device, comprising: a data acquisition module, for acquiring a preview image in a moon shooting scene, and shortening the exposure time during shooting when it is detected that the preview image includes a target object; a state determination module, for acquiring the state of the electronic device, and in response to the state of the electronic device being a stable state, determining a reference frame image from the preview image; and an image displacement module, for determining motion information of a current frame image relative to the reference frame image, and performing image deformation on the current frame image based on the motion information to obtain a target image.

[0007] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented.

[0008] According to a fourth aspect of the present disclosure, an electronic device is provided, characterized in that it includes: one or more processors; and a memory for storing one or more programs, which enables the one or more processors to implement the above method when the one or more programs are executed by the one or more processors.

[0009] An image processing method provided by an embodiment of the present disclosure, on the one hand, when an electronic device uses the camera to shoot the moon, if a target object appears in the preview image, the exposure time during shooting will be shortened, so that the moon part in the obtained target image will not be overexposed, and the texture of the moon itself can be retained. On the other hand, the state of the electronic device is obtained. If the state of the electronic device is a stable state, a reference frame image is determined from the shot image, and the motion information of the current frame image relative to the reference frame image is determined. Then, based on the motion information of the current frame image relative to the reference frame image, the current frame image is displaced, so that the image drift of the current frame image can be eliminated, and a clearer target image can be obtained when shooting the moon.

[0010] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0012] Figure 1 A schematic diagram of imaging comparison when a camera shakes, to which an embodiment of the present disclosure can be applied, is shown.

[0013] Figure 2 The flowchart schematically shows an image processing method in an exemplary embodiment of the present disclosure.

[0014] Figure 3 A comparison diagram of the moon photographed at different exposure times in an exemplary embodiment of the present disclosure is schematically shown.

[0015] Figure 4 Another comparison diagram of the moon photographed at different exposure times in an exemplary embodiment of the present disclosure is schematically shown.

[0016] Figure 5 A framework diagram schematically illustrates an electronic image stabilization technology in an exemplary embodiment of the present disclosure.

[0017] Figure 6 A data flow diagram of an image processing method according to an exemplary embodiment of the present disclosure is schematically shown.

[0018] Figure 7 The following is a schematic diagram schematically showing the composition of an image processing device in an exemplary embodiment of the present disclosure.

[0019] Figure 8 A schematic diagram of an electronic device to which the embodiments of the present disclosure can be applied is shown. DETAILED DESCRIPTION

[0020] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0021] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0022] In the related art, the existing technology can effectively suppress the influence of shaking under normal circumstances, but under the condition of long focal length and high magnification, the same shaking amplitude has a huge impact on the picture. Figure 1 As shown, taking a telephoto lens with a focal length of 1000mm as an example, when the device shakes 0.1°, the shaking amplitude at the wide-angle end accounts for 0.45% of the entire screen, and the screen shaking is not obvious; but when the lens is in the telephoto section, the shaking amplitude accounts for 22.00% of the entire screen. Furthermore, due to the limitations of the camera module on the mobile phone, telephoto lenses are usually digitally zoomed when the magnification exceeds 10X. Therefore, the real posture and reference posture calculated by the original anti-shake algorithm cannot well reflect the posture of the image block, and slight shaking will also occur. Secondly, EIS using gyroscope information relies on gyroscope accuracy. Consumer-grade gyroscopes usually have continuous and random drift, but when using telephoto high-magnification focal lengths for framing, the framing cannot be maintained.

[0023] When photographing the moon, the moon scene is first a low-light night scene. At this time, the camera will automatically brighten the overall brightness of the picture by increasing the exposure time, but when photographing the moon, the moon will become a bright ball, and the texture on the moon cannot be seen. At the same time, the exposure time is too long, resulting in serious motion blur. When the camera shakes, the moon in the image becomes an elliptical bright spot with unclear edges, and the effect is poor.

[0024] Based on the above shortcomings, the present disclosure provides an image processing method, which is applied to an electronic device including a camera and a gyroscope, and the electronic device may be a smart phone, a tablet computer, a camera, etc. The above scheme can be applied to a scene when a user uses the above electronic device to take a photo of the moon.

[0025] Combine the following Figure 2 The image processing method in this exemplary embodiment is described. Figure 2 An exemplary process of the image processing method is shown, which may include steps S210 to S240.

[0026] In step S210, a preview image of the moon shooting scene is obtained, and when it is detected that the preview image includes a target object, the shooting exposure time is shortened.

[0027] In an example implementation of the present disclosure, the electronic device may be an electronic device including a camera module and a gyroscope, including but not limited to a mobile phone and a tablet computer. When a user uses the camera in the camera module of the electronic device to take a picture of the moon, a preview image may be obtained in real time, and it may be determined whether the preview image includes a target object, wherein the target object may be a luminous object.

[0028] It should be noted that the above-mentioned luminous objects include but are not limited to the moon, street lights, fireworks, etc. Specifically, the presence of luminous objects can be determined by brightness detection. If the above-mentioned preview image includes luminous objects, refer to Figure 3 and Figure 4 As shown, shortening the exposure time during shooting can prevent the moon portion in the target image from being overexposed, and can preserve the texture of the moon itself.

[0029] Specifically, a neural network can be used to perform binary classification on the preview image to obtain the target confidence that the preview image includes the moon. The binary classification categories may include moon scenes and non-moon scenes, and a first confidence and a second confidence are set. If the target confidence is greater than the first confidence, it can be determined that the preview image includes a luminous object, that is, a moon-like scene, but it cannot be determined as a moon scene. If the target confidence is greater than the second confidence, it is determined that the preview image includes the moon. If the target confidence is less than the first confidence, it is determined that the preview image does not include a luminous object.

[0030] The first confidence level may be 60%, and the second confidence level may be 90%. The specific values ​​of the first confidence level and the second confidence level may be customized according to user requirements and are not specifically limited in this exemplary implementation.

[0031] In an example implementation, if the preview image includes the moon, that is, the target confidence is greater than the second confidence, then after the moon disappears from the preview image for a predetermined time, the exposure time is called back to the initial value, that is, when the user is shooting the moon, if the moon is accidentally moved out of the preview image, the exposure time is called back within the preset time, which can improve the user experience, reduce the number of exposure time adjustments, and reduce power consumption. The predetermined time can be 5 seconds, 6 seconds, etc., and can also be customized according to user needs, which is not specifically limited in this example implementation.

[0032] If the target confidence is greater than the first confidence and less than or equal to the second confidence, the exposure time is immediately called back to the initial value when the luminous object disappears from the preview image.

[0033] In an example implementation, optical image stabilization technology may be used during exposure to ensure image stability, wherein the optical image stabilization may be lens-based optical image stabilization, image sensor-based optical image stabilization, or both the lens and the image sensor may be optically stabilized simultaneously.

[0034] In step S220, the state of the electronic device is obtained, and in response to the state of the electronic device being a stable state, a reference frame image is determined from the preview image.

[0035] In this example implementation, if the exposure time is shortened, when photographing the moon, the camera is usually in a telephoto state, and objects farther away from the camera can be photographed. The telephoto ratio is greater than the preset magnification ratio. The preset magnification ratio can be set as needed. Exemplarily, the telephoto ratio is a high magnification ratio (zoom) of 10x. It should be noted that the camera has a telephoto ratio, and the camera can be a telephoto camera or other cameras with a telephoto ratio, which is not limited here.

[0036] Optionally, the electronic device starts the camera, zooms in on the camera screen, enters the telephoto magnification, and uses the telephoto magnification camera to shoot; during the process of shooting with the telephoto magnification camera, gyroscope data is obtained in real time through the gyroscope, and the state of the electronic device is determined based on the gyroscope data.

[0037] In this example implementation, the state of the electronic device may include a motion state and a stable state, and the stable state may include a stable moving state and a stable stationary state. It is understandable that when the electronic device is in a motion state, it means that the electronic device has a large jitter, and when the electronic device is in a stable state, it means that the electronic device has a small jitter, the jitter amplitude of the stable state is smaller than the jitter amplitude of the motion state, and the jitter amplitude of the stable stationary state is smaller than the jitter amplitude of the stable moving state.

[0038] In one embodiment, during the process of the electronic device using a camera with a long focal length, the state of the electronic device is determined by using gyroscope data obtained by the gyroscope, including: during the process of the electronic device using a camera with a long focal length, the state of the electronic device is determined based on the current instantaneous value in the gyroscope data obtained by the gyroscope.

[0039] Optionally, if the current instantaneous value is greater than a first preset instantaneous value, the electronic device is determined to be in a moving state; if the current instantaneous value is less than or equal to the first preset instantaneous value, the electronic device is determined to be in a stable state. The first preset instantaneous value can be set as needed.

[0040] Optionally, if the current instantaneous value is less than or equal to the first preset instantaneous value and greater than the second preset instantaneous value, or the current instantaneous value is less than or equal to the second preset instantaneous value and the cumulative value of the preset time length is greater than the preset cumulative value, the state of the electronic device is determined to be a stable moving state; if the current instantaneous value is less than or equal to the first preset instantaneous value, and the cumulative value of the preset time length is less than or equal to the preset cumulative value, the state of the electronic device is determined to be a stable stationary state.

[0041] It can be understood that if the current instantaneous value is less than or equal to the first preset instantaneous value and greater than the second preset instantaneous value, and the cumulative value of the preset time length is less than or equal to the preset cumulative value, then the state of the electronic device is determined to be a stable moving state; if the current instantaneous value is less than or equal to the first preset instantaneous value and greater than the second preset instantaneous value, and the cumulative value of the preset time length is greater than the preset cumulative value, then the state of the electronic device is determined to be a stable moving state; if the current instantaneous value is less than or equal to the second preset instantaneous value, and the cumulative value of the preset time length is greater than the preset cumulative value, then the state of the electronic device is determined to be a stable moving state.

[0042] In an example implementation of the present disclosure, when the electronic device is shooting using a telephoto camera, determining the state of the electronic device using gyroscope data acquired by the gyroscope may include steps S310 to S320.

[0043] In step S310, when the electronic device uses a telephoto camera to shoot, various gyroscope data obtained by the gyroscope are used to determine the current instantaneous value of the gyroscope and the accumulated value of the preset time.

[0044] In this exemplary embodiment, the preset duration can be set as required. For example, the preset duration can be 10 frames.

[0045] Optionally, the current instantaneous value of the gyroscope is the latest gyroscope data acquired by the gyroscope, and the accumulated value of the preset time period is the sum of each gyroscope data acquired by the gyroscope within the preset time period.

[0046] Step S320, determining the state of the electronic device based on the current instantaneous value of the gyroscope and the accumulated value of the preset time length.

[0047] In an example implementation, if the current instantaneous value of the gyroscope is greater than a first preset instantaneous value, and the cumulative value of the preset time length is greater than the preset cumulative value, indicating that the jitter of the electronic device is large, the state of the electronic device is determined to be a motion state, that is, the electronic device is in a motion scene; if the current instantaneous value of the gyroscope is less than or equal to the first preset instantaneous value, and the cumulative value of the preset time length is less than or equal to the preset cumulative value, indicating that the jitter of the electronic device is small, the state of the electronic device is determined to be a stable state.

[0048] Among them, the first preset instantaneous value and the preset cumulative value can be set as needed.

[0049] In an example implementation, if the current instantaneous value of the gyroscope is less than or equal to the first preset instantaneous value, and the cumulative value of the preset time length is less than or equal to the preset cumulative value, then the state of the electronic device is determined to be a stable state, including: if the current instantaneous value of the gyroscope is less than or equal to the first preset instantaneous value and greater than the second preset instantaneous value, and the cumulative value of the preset time length is less than or equal to the preset cumulative value, then the state of the electronic device is determined to be a stable moving state, that is, the electronic device is in a stable moving scene; the first preset instantaneous value is greater than the second preset instantaneous value; if the current instantaneous value of the gyroscope is less than or equal to the second preset instantaneous value, and the cumulative value of the preset time length is less than or equal to the preset cumulative value, then the state of the electronic device is determined to be a stable static state, that is, the electronic device is in a stable static scene. Among them, the second preset instantaneous value can be set as needed.

[0050] It can be understood that if the current instantaneous value of the gyroscope is less than or equal to the first preset instantaneous value and greater than the second preset instantaneous value, and the cumulative value of the preset time length is less than or equal to the preset cumulative value, it means that the electronic device has a certain degree of jitter, and the state of the electronic device is determined to be a stable moving state; if the current instantaneous value of the gyroscope is less than or equal to the second preset instantaneous value, and the cumulative value of the preset time length is less than or equal to the preset cumulative value, it means that the electronic device has a small jitter and is basically in a stationary state, and the state of the electronic device is determined to be a stable stationary state.

[0051] It should be noted that when an electronic device uses a telephoto camera to shoot, it is usually necessary to aim at the subject, adjust the shooting angle, and finally press the shutter to acquire the image. Slight movement when adjusting the angle will often cause the subject to exceed the shooting screen. Therefore, the electronic device is in a stable moving state, which can eliminate the drift of the stable moving state and provide the ability to slightly move the shooting screen. After the electronic device adjusts the shooting angle, when taking a photo when the shooting screen is completely stable, inadvertent shaking may cause the shooting screen to shift again. Therefore, the electronic device is in a stable still state, which can eliminate the drift of the stable still state and provide the ability to still the picture when stable and still, thereby meeting the two major needs of stable fine-tuning of the angle and stable framing.

[0052] In this embodiment, the electronic device obtains various gyroscope data through the gyroscope to determine the current instantaneous value of the gyroscope and the accumulated value of the preset time. The state of the electronic device can be judged more accurately through the two dimensions of the current instantaneous value and the accumulated value of the preset time.

[0053] The reference frame image is an image used as a reference in the process of eliminating the drift of the current frame image. The drift is the movement of the current frame image.

[0054] Optionally, the reference frame image may be a preceding frame image of the current frame image or a designated key frame image. Exemplarily, the images generated during the shooting process of the electronic device are image 1, image 2, and image 3 in sequence. When image 3 is the current frame image, the preceding frame image of the current frame image is the previous frame image 2, or image 1, and the key frame image may be the first frame image 1; when image 4 is the current frame image, the preceding frame image of the current frame image is the previous frame image 3, or image 2, and the key frame image may be the first frame image 1.

[0055] In step S230, the motion information of the current frame image relative to the reference frame image is determined, and based on the motion information, the current frame image is deformed to obtain a target image.

[0056] It can be understood that if the reference frame image is the previous frame image of the current frame image, the change between the current frame image and the previous frame image is small, and the motion information of the current frame image relative to the reference frame image can be determined more accurately and quickly.

[0057] If the reference frame image is a designated key frame image, a new reference frame image is determined at intervals of a designated duration, and the motion information of the current frame image relative to the new reference frame image is determined. This can avoid the accumulated drift error of iterative calculation of adjacent image frames, and can accurately determine the motion information of the current frame image relative to the reference frame image during long-term shooting.

[0058] Optionally, the motion information of the current frame image relative to the reference frame image includes displacement information and may also include rotation information. The displacement information may be a displacement vector.

[0059] Optionally, if the state of the electronic device is a moving state, EIS (Electronic Image Stabilizing, electronic image stabilization technology) is used to process the current frame image to obtain a target image.

[0060] Specifically, refer to Figure 5 As shown, EIS (Electronic Image Stabilizing) can usually complete video stabilization in three steps, which can specifically include motion estimation, motion compensation (non-subjective motion removal) and image deformation. Among them, the motion estimation module is usually regarded as a gyroscope to detect the angular velocity of the camera at a specific moment, and integrates the gyroscope data between each frame to build a camera motion model. Then the real motion path of the camera is obtained, and then the real path is filtered and fitted to obtain the camera reference posture of each frame, and then the real posture of the camera corresponding to each frame of the image is obtained. After obtaining the real posture and reference posture of each frame, the image needs to be transformed so that the perspective of the picture is transferred from the real posture to the reference posture. In this process, the picture needs to be transformed and cropped.

[0061] In an example implementation of the present disclosure, determining the motion information of the current frame image relative to the reference frame image may include steps S410 to S430.

[0062] In step S410, based on the posture difference between the current frame image and the reference frame image, a first displacement value of the current frame image relative to the reference frame image is determined.

[0063] Optionally, the electronic device determines the posture information Orientation_cur of the current frame image and the posture information Orientation_key of the reference frame image, performs difference processing on the posture information Orientation_cur of the current frame image and the posture information Orientation_key of the reference frame image, and obtains the posture difference Orientation_diff (expressed in quaternion) between the current frame image and the reference frame image; selects a preset reference feature point in the reference frame image, calculates the reference feature point corresponding to the current feature point in the current frame image according to the posture difference Orientation_diff, and determines the drift value between the position of the reference feature point and the position of the current feature point as the first displacement value of the current frame image relative to the reference frame image.

[0064] In step S420, the current frame image is shifted according to the first shift value to obtain an intermediate image, and a second shift value of the intermediate image relative to the reference frame image is determined.

[0065] Optionally, the electronic device performs displacement compensation on the current frame image by a first displacement value to obtain an intermediate image, and determines a second displacement value of the intermediate image relative to the reference frame image.

[0066] In one embodiment, the electronic device can use key points to determine the second displacement value of the intermediate image relative to the reference frame image, specifically including: the electronic device uses a feature extraction algorithm to extract reference feature points in the reference frame image, and uses a KLT (Kanade-Lucas-Tomasi, sparse optical flow tracking) algorithm in the current frame image to track the reference feature points to obtain the current feature points in the current frame image; the reference feature points and the current feature points form a point pair, and determine the relative displacement value of the point pair as the second displacement value of the intermediate image relative to the reference frame image.

[0067] In another implementation, the electronic device may also use a grayscale projection method to determine the second displacement value of the intermediate image relative to the reference frame image.

[0068] In step S430, motion information of the current frame image relative to the reference frame image is determined based on the first displacement value and the second displacement value.

[0069] Optionally, the electronic device adds the first displacement value to the second displacement value to obtain a target displacement value as motion information of the current frame image relative to the reference frame image. Exemplarily, the first displacement value d of the current frame image relative to the reference frame image x1 =12pixel (pixel), the current frame image is displaced according to the first displacement value to obtain an intermediate image, and a second displacement value d of the intermediate image relative to the reference frame image is determined x2 =2.2pixel, then the target displacement value d x =d x1 +d x2 =14.2pixel.

[0070] Optionally, the electronic device may average the first displacement value and the second displacement value, and use the average value as motion information of the current frame image relative to the reference frame image.

[0071] In other implementations, the electronic device may also determine the motion information of the current frame image relative to the reference frame image by weighted summation, weighted average, or random selection, etc., which is not limited here.

[0072] In this embodiment, the electronic device determines a first displacement value of the current frame image relative to the reference frame image, displaces the current frame image according to the first displacement value to obtain an intermediate image, and then determines a second displacement value of the intermediate image relative to the reference frame image; that is, a coarse displacement value (first displacement value) is first determined for the current frame image and the reference frame image, and a fine displacement value (second displacement value) is then determined based on the coarse displacement value, thereby avoiding the need for a large amount of calculation to determine the displacement value of the reference frame image and the current frame image, reducing the amount of calculation, improving the operation speed, and more quickly determining the motion information of the current frame image relative to the reference frame image.

[0073] In one embodiment, determining a second displacement value of an intermediate image relative to a reference frame image includes: projecting the intermediate image and the reference frame image respectively to obtain a projection curve of the intermediate image and a projection curve of the reference frame image; after shifting the projection curve of the intermediate image by different displacement values, determining a similarity between the projection curve of the intermediate image and the projection curve of the reference frame image after each displacement; determining a target similarity from each similarity, and using the displacement value corresponding to the target similarity as the second displacement value of the intermediate image relative to the reference frame image.

[0074] Optionally, the electronic device performs grayscale projection on the intermediate image and the reference frame image respectively to obtain a projection curve of the intermediate image and a projection curve of the reference frame image; after shifting the projection curve of the intermediate image by different displacement values ​​for a preset number of times, an NCC (Normalized cross-correlation) value between the projection curve of the intermediate image after each displacement and the projection curve of the reference frame image is determined, and the NCC value represents the similarity between the projection curve of the intermediate image after the displacement and the projection curve of the reference frame image; a target NCC value is determined from each NCC value, and the displacement value corresponding to the target NCC value is used as a second displacement value of the intermediate image relative to the reference frame image.

[0075] In an optional embodiment, the electronic device determines the maximum NCC value from each NCC value as the target NCC value. The maximum NCC value indicates that the similarity between the projection curve of the intermediate image after displacement and the projection curve of the reference frame image is the highest. The displacement value corresponding to the maximum NCC value is the second displacement value of the intermediate image relative to the reference frame image.

[0076] In other optional implementations, the electronic device may also determine the next largest NCC value from various NCC values ​​as the target NCC value, or randomly determine the target NCC value, which is not limited here.

[0077] Optionally, the intermediate image and the reference frame image are projected respectively to obtain a projection curve of the intermediate image and a projection curve of the reference frame image, including: projecting the intermediate image in a first direction and a second direction respectively to obtain a first-direction projection curve and a second-direction projection curve of the intermediate image; the first direction and the second direction are perpendicular to each other; projecting the reference frame image in the first direction and the second direction respectively to obtain a first-direction projection curve and a second-direction projection curve of the reference frame image; after shifting the projection curve of the intermediate image by different displacement values, determining the similarity between the projection curve of the intermediate image after each displacement and the projection curve of the reference frame image, including: after shifting the first-direction projection curve and the second-direction projection curve of the intermediate image multiple times and each time by the same displacement value, determining a first similarity between the first-direction projection curve of the intermediate image after each displacement and the first-direction projection curve of the reference frame image, and a second similarity between the second-direction projection curve of the intermediate image and the second-direction projection curve of the reference frame image; based on the first similarity and the second similarity after each displacement, determining the similarity between the projection curve of the intermediate image after each displacement and the projection curve of the reference frame image.

[0078] Optionally, the first direction and the second direction may be the horizontal direction and the vertical direction, ie, the X direction and the Y direction in the image coordinate system.

[0079] It can be understood that the electronic device displaces the first direction projection curve and the second direction projection curve of the intermediate image multiple times and each time the displacement value is the same, that is, each time the electronic device displaces the projection curve of the intermediate image, the first direction projection curve and the second direction projection curve are displaced by the same displacement value, and the displacement values ​​of different displacements are different, so that the first direction projection curve and the second direction projection curve of the intermediate image after multiple displacements can be obtained.

[0080] Optionally, the electronic device adds the first similarity after each displacement to the second similarity to obtain the similarity between the projection curve of the intermediate image after each displacement and the projection curve of the reference frame image.

[0081] Optionally, the electronic device may also determine the target similarity by performing weighted summation or weighted average on the first similarity and the second similarity, which is not limited here.

[0082] In this embodiment, the electronic device projects the intermediate image and the reference frame image respectively to obtain a projection curve of the intermediate image and a projection curve of the reference frame image, and after shifting the projection curve of the intermediate image by different displacement values, determines the similarity between the projection curve of the intermediate image and the projection curve of the reference frame image after each displacement, and determines a target similarity from each similarity. The target similarity represents the similarity between the intermediate image and the reference frame image, so that the displacement value corresponding to the target similarity can be accurately used as a second displacement value of the intermediate image relative to the reference frame image.

[0083] Furthermore, the electronic device may project the intermediate image in the first direction and the second direction respectively, and project the reference frame image in the first direction and the second direction respectively, so as to accurately determine the similarity between the intermediate image and the reference frame image.

[0084] Optionally, the electronic device determines a target displacement value of the current frame image based on the motion information, and performs image changes on the current frame image based on the target displacement value. Specifically, the electronic device performs displacement compensation on the current frame image to obtain a target image.

[0085] The motion information may include a first displacement value of the current frame image relative to the reference frame image and a second displacement value of the intermediate image relative to the reference frame image, and the intermediate image is an image obtained by displacing the current frame image according to the first displacement value.

[0086] In an exemplary implementation, the electronic device adds the first displacement value and the second displacement value to obtain a target displacement value. For example, the first displacement value is d x1 ,d y1 , the second displacement value is d x1 ,d y2 , then the target displacement value is d x =d x1 +d x2 , d y =d y1 +d y2 .

[0087] In another optional implementation, the electronic device performs weighted summation on the first displacement value and the second displacement value to obtain the target displacement value. In other implementations, the electronic device may also use other methods to determine the target displacement value, which is not limited here.

[0088] The stable state of the electronic device may include a stable moving state and a stable stationary state. Optionally, if the state of the electronic device is a stable moving state, the first displacement value and the second displacement value are filtered, and a target displacement value is determined based on the filtered first displacement value and the second displacement value, and the current frame image is displaced based on the target displacement value to obtain a target image; if the state of the electronic device is a stable stationary state, the current frame image is displacement compensated (i.e., image affine transformation) based on the target displacement value to obtain a target image.

[0089] The way in which the electronic device filters the motion information may include at least one of smoothing filtering, mean filtering, etc., which is not limited here.

[0090] Exemplarily, the motion information of the current frame image relative to the reference frame image includes a first displacement value d of the current frame image relative to the reference frame image. x1 and the second displacement value d of the intermediate image relative to the reference frame image y2 ; Electronic equipment uses the filtering formula d′ xn = k(d xn -d′ x(n-1) ) for the first displacement value d x1 and the second displacement value d y2 Filtering: d x1 = k(d x1 -0), d′ x2 = k(d x2 -d′ x1 ).

[0091] Where d′ xn is the displacement value after filtering, d′ x1 is the first displacement value after filtering, d′ x2 is the second displacement value after filtering, and k is a preset coefficient, such as k=0.1.

[0092] It can be understood that if the state of the electronic device is a stable moving state, the motion information is filtered, and the filtered motion information can retain the original motion trend and make the filtered data smoother, and the obtained target image and adjacent images have higher coherence.

[0093] Refer to the following Figure 6The above image processing method is described as a whole. First, the preview image can be subjected to moon scene recognition to determine whether it is in the moon scene. The specific judgment method has been defined in detail above and is not defined here. Then, exposure control can be performed based on the recognition result to shorten the exposure time when the preview image includes a luminous object. During exposure, optical image stabilization technology can be used to ensure the stability of the image. The optical image stabilization can be based on the lens, based on the image sensor, or both the lens and the image sensor.

[0094] Then, the image and EIS processing can be performed according to the gyroscope data of the electronic device, which may specifically include motion estimation, attitude filtering and image deformation. Before the image is deformed, the above attitude information can be gridded, and then the above image grids can be updated based on the displacement calculated by the motion information in the above image processing method. When the above image grids are updated, smoothing filtering can be performed to retain the motion characteristics of the image. After the above grids are updated based on the above displacement information, image deformation is performed to obtain the target image.

[0095] The above-mentioned image processing method, on the one hand, when the electronic device uses a camera to shoot the moon, if a luminous object appears in the preview image, the exposure time during shooting will be shortened, so that the moon part in the obtained target image will not be overexposed, and the texture of the moon itself can be retained. On the other hand, the state of the electronic device is determined by the gyroscope data obtained by the gyroscope. If the state of the electronic device is a stable state, a reference frame image is determined from the captured image, and the motion information of the current frame image relative to the reference frame image is determined. Then, based on the motion information of the current frame image relative to the reference frame image, the current frame image is displaced, and the image drift of the current frame image can be eliminated, so that a clearer target image can be obtained when shooting the moon.

[0096] When an electronic device uses a telephoto camera to take pictures, it is usually necessary to press the shutter button in a stable state to obtain an image. The above image processing method can eliminate the shaking of the electronic device in a stable state and take clearer images.

[0097] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0098] For further reference, Figure 7As shown, in the embodiment of this example, an image processing device 700 is also provided, including a data acquisition module 710, a state determination module 720 and an image displacement module 730. Among them:

[0099] The data acquisition module 710 can be used to obtain a preview image in a moon shooting scene. When it is detected that the preview image includes a target object, the exposure time during shooting is shortened; the state determination module 720 can be used to obtain the state of the electronic device, and in response to the state of the electronic device being a stable state, a reference frame image is determined from the preview image; the image displacement module 730 can determine the motion information of the current frame image relative to the reference frame image, and based on the motion information, perform image deformation on the current frame image to obtain a target image.

[0100] In an example implementation, the data acquisition module 710 may be configured to perform binary classification on the preview image to determine a target confidence level that includes the moon in the preview image; in response to the target confidence level being greater than a first confidence level, it is determined that the preview image includes the target object, and the shooting exposure time is shortened.

[0101] The above-mentioned data acquisition module 710 can be configured to respond to the target confidence being greater than the second confidence, determine that the preview image includes the moon, wherein the second confidence is greater than the first confidence; in response to the target object in the preview image including the moon, when the length of time during which the moon is not detected in the preview image exceeds a threshold, call back the exposure time to the initial value.

[0102] In an example implementation, the state determination module 720 may be configured to determine the current instantaneous value of the gyroscope and the cumulative value of a preset time period through various gyroscope data acquired by the gyroscope; and determine the state of the electronic device based on the current instantaneous value of the gyroscope and the cumulative value of the preset time period.

[0103] Specifically, if the current instantaneous value of the gyroscope is greater than the first preset instantaneous value, and the cumulative value of the preset time length is greater than the preset cumulative value, then the state of the electronic device is determined to be a motion state; if the current instantaneous value of the gyroscope is less than or equal to the first preset instantaneous value, and the cumulative value of the preset time length is less than or equal to the preset cumulative value, then the state of the electronic device is determined to be a stable state.

[0104] When judging whether the above-mentioned electronic device is in a stable state, the above-mentioned state determination module can be configured to determine that the state of the electronic device is a stable moving state if the current instantaneous value of the gyroscope is less than or equal to the first preset instantaneous value and greater than the second preset instantaneous value, and the cumulative value of the preset time length is less than or equal to the preset cumulative value; the first preset instantaneous value is greater than the second preset instantaneous value; if the current instantaneous value of the gyroscope is less than or equal to the second preset instantaneous value, and the cumulative value of the preset time length is less than or equal to the preset cumulative value, then determine that the state of the electronic device is a stable static state.

[0105] The image displacement module 730 can be configured to determine a first displacement value of the current frame image relative to the reference frame image based on the posture difference between the current frame image and the reference frame image; perform image changes on the current frame image according to the first displacement value to obtain an intermediate image, and determine a second displacement value of the intermediate image relative to the reference frame image; and determine motion information of the current frame image relative to the reference frame image based on the first displacement value and the second displacement value.

[0106] When determining the second displacement value of the intermediate image relative to the reference frame image, the image displacement module 730 can be configured to project the intermediate image and the reference frame image respectively to obtain a projection curve of the intermediate image and a projection curve of the reference frame image; after displacing the projection curve of the intermediate image by different displacement values, determine the similarity between the projection curve of the intermediate image after each displacement and the projection curve of the reference frame image; determine the target similarity from each similarity, and use the displacement value corresponding to the target similarity as the second displacement value of the intermediate image relative to the reference frame image.

[0107] Specifically, the intermediate image and the reference frame image are projected respectively to obtain a projection curve of the intermediate image and a projection curve of the reference frame image, including:

[0108] The intermediate image is projected in a first direction and a second direction respectively to obtain a first direction projection curve and a second direction projection curve of the intermediate image; the first direction and the second direction are perpendicular to each other; the reference frame image is projected in the first direction and the second direction respectively to obtain a first direction projection curve and a second direction projection curve of the reference frame image; after the projection curve of the intermediate image is shifted by different displacement values, the similarity between the projection curve of the intermediate image after each shift and the projection curve of the reference frame image is determined, including: after the first direction projection curve and the second direction projection curve of the intermediate image are shifted multiple times and the displacement is the same each time, the first similarity between the first direction projection curve of the intermediate image after each shift and the first direction projection curve of the reference frame image, and the second similarity between the second direction projection curve of the intermediate image and the second direction projection curve of the reference frame image are determined; based on the first similarity and the second similarity after each shift, the similarity between the projection curve of the intermediate image after each shift and the projection curve of the reference frame image is determined.

[0109] The specific details of each module in the above device have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0110] Below Figure 8 The mobile terminal 800 in FIG. 1 is taken as an example to exemplify the structure of the electronic device. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Figure 8 The construction in can also be applied to fixed type equipment.

[0111] like Figure 8 As shown, the mobile terminal 800 may specifically include: a processor 801, a memory 802, a bus 803, a mobile communication module 804, an antenna 1, a wireless communication module 805, an antenna 2, a display screen 806, a camera module 807, an audio module 808, a power module 809 and a sensor module 810.

[0112] The processor 801 may include one or more processing units, for example, the processor 801 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor and / or an NPU (Neural-Network Processing Unit), etc. The image processing method in this exemplary embodiment may be executed by an AP, a GPU or a DSP, and when the method involves processing related to a neural network, it may be executed by an NPU.

[0113] The processor 801 may be connected to the memory 802 or other components via a bus 803 .

[0114] The memory 802 may be used to store computer executable program codes, which include instructions. The processor 801 executes various functional applications and data processing of the mobile terminal 800 by running the instructions stored in the memory 802. The memory 802 may also store application data, such as images, videos and other files.

[0115] The communication function of the mobile terminal 800 can be implemented by the mobile communication module 804, antenna 1, wireless communication module 805, antenna 2, modulation and demodulation processor and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 804 can provide 2G, 3G, 4G, 5G and other mobile communication solutions applied to the mobile terminal 800. The wireless communication module 805 can provide wireless communication solutions such as wireless LAN, Bluetooth, near field communication, etc. applied to the mobile terminal 800.

[0116] The display screen 806 is used to implement display functions, such as displaying user interfaces, images, and videos. The camera module 807 is used to implement shooting functions, such as shooting images and videos. The audio module 808 is used to implement audio functions, such as playing audio and collecting voice. The power module 809 is used to implement power management functions, such as charging the battery, powering the device, and monitoring the battery status. The sensor module 810 may include a depth sensor 8101, a pressure sensor 8102, a gyroscope sensor 8103, an air pressure sensor 8104, etc., to implement corresponding sensing detection functions.

[0117] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0118] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a program product capable of implementing the above-mentioned method of the present specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of the present specification.

[0119] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0120] In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0121] In addition, program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0122] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0123] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: Applied to electronic equipment, the method comprises: Acquire a preview image in a moon shooting scene, and when it is detected that the preview image includes a target object, shorten the shooting exposure time; Acquiring the state of the electronic device, and in response to the state of the electronic device being a stable state, determining a reference frame image from the preview image; The motion information of the current frame image relative to the reference frame image is determined, and based on the motion information, the current frame image is deformed to obtain a target image.

2. The method according to claim 1, characterized in that: When it is detected that the preview image includes a target object, shortening the shooting exposure time includes: Performing binary classification on the preview image to determine the object confidence that the preview image includes the moon; In response to the target confidence being greater than the first confidence, determining that the preview image includes the target object, and shortening the shooting exposure time; Wherein, the target object in the preview image includes a luminous object.

3. The method according to claim 2, characterized in that The method further comprises: In response to the target confidence being greater than a second confidence, determining that the preview image includes the moon, wherein the second confidence is greater than the first confidence; In response to the target object in the preview image including the moon, when the duration during which the moon is not detected in the preview image exceeds a threshold, the exposure time is called back to an initial value.

4. The method according to claim 1, characterized in that: The determining motion information of the current frame image relative to the reference frame image includes: Determining a first displacement value of the current frame image relative to the reference frame image based on a posture difference between the current frame image and the reference frame image; Performing an image change on the current frame image according to the first displacement value to obtain an intermediate image, and determining a second displacement value of the intermediate image relative to the reference frame image; Based on the first displacement value and the second displacement value, motion information of the current frame image relative to the reference frame image is determined.

5. The method according to claim 4, characterized in that The determining a second displacement value of the intermediate image relative to the reference frame image comprises: Projecting the intermediate image and the reference frame image respectively to obtain a projection curve of the intermediate image and a projection curve of the reference frame image; After shifting the projection curve of the intermediate image by different displacement values, determining the similarity between the projection curve of the intermediate image after each displacement and the projection curve of the reference frame image; A target similarity is determined from the various similarities, and a displacement value corresponding to the target similarity is used as a second displacement value of the intermediate image relative to the reference frame image.

6. The method according to claim 5, characterized in that The step of projecting the intermediate image and the reference frame image respectively to obtain a projection curve of the intermediate image and a projection curve of the reference frame image comprises: Projecting the intermediate image in a first direction and a second direction respectively to obtain a first-direction projection curve and a second-direction projection curve of the intermediate image; the first direction and the second direction are perpendicular to each other; Projecting the reference frame image in a first direction and a second direction respectively to obtain a first direction projection curve and a second direction projection curve of the reference frame image; After the projection curve of the intermediate image is shifted by different shift values, determining the similarity between the projection curve of the intermediate image after each shift and the projection curve of the reference frame image comprises: After displacing the first-direction projection curve and the second-direction projection curve of the intermediate image multiple times and each time by the same displacement value, determine a first similarity between the first-direction projection curve of the intermediate image after each displacement and the first-direction projection curve of the reference frame image, and a second similarity between the second-direction projection curve of the intermediate image and the second-direction projection curve of the reference frame image; Based on the first similarity and the second similarity after each displacement, the similarity between the projection curve of the intermediate image after each displacement and the projection curve of the reference frame image is determined.

7. The method according to claim 5, characterized in that Determining the target similarity from the various similarities includes: The maximum value among the similarities is taken as the target similarity.

8. The method according to claim 1, characterized in that The obtaining the state of the electronic device comprises: Determine the current instantaneous value of the gyroscope and the accumulated value of the preset time period through each gyroscope data obtained by the gyroscope; The state of the electronic device is determined based on the current instantaneous value of the gyroscope and the accumulated value of a preset time period.

9. The method according to claim 8, characterized in that The determining the state of the electronic device based on the current instantaneous value of the gyroscope and the accumulated value of the preset time period includes: If the current instantaneous value of the gyroscope is greater than the first preset instantaneous value, and the cumulative value of the preset time length is greater than the preset cumulative value, determining that the state of the electronic device is a motion state; If the current instantaneous value of the gyroscope is less than or equal to the first preset instantaneous value, and the cumulative value of the preset time length is less than or equal to the preset cumulative value, it is determined that the state of the electronic device is a stable state.

10. The method according to claim 9, characterized in that If the current instantaneous value of the gyroscope is less than or equal to a first preset instantaneous value, and the cumulative value of the preset time length is less than or equal to a preset cumulative value, determining that the state of the electronic device is a stable state includes: If the current instantaneous value of the gyroscope is less than or equal to the first preset instantaneous value and greater than the second preset instantaneous value, and the cumulative value of the preset time length is less than or equal to the preset cumulative value, it is determined that the state of the electronic device is a stable moving state; the first preset instantaneous value is greater than the second preset instantaneous value; If the current instantaneous value of the gyroscope is less than or equal to the second preset instantaneous value, and the cumulative value of the preset time length is less than or equal to the preset cumulative value, it is determined that the state of the electronic device is a stable static state.

11. An image processing device, characterized in that: Applied to, the device comprises: A data acquisition module, used for acquiring a preview image in a moon shooting scene, and shortening the exposure time during shooting when it is detected that the preview image includes a target object; a state determination module, configured to obtain the state of the electronic device, and in response to the state of the electronic device being a stable state, determine a reference frame image from the preview image; The image displacement module is used to determine the motion information of the current frame image relative to the reference frame image, and based on the motion information, perform image deformation on the current frame image to obtain a target image.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image processing method according to any one of claims 1 to 10 is implemented.

13. An electronic device, characterized in that: include: one or more processors; as well as A memory for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the image processing method according to any one of claims 1 to 10.

Images