Image processing method and device, electronic equipment and computer readable storage medium

Abstract

The invention relates to an image processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring a first image frame and a second image frame in response to a first target signal; wherein in the process of obtaining the first image frame, the anti-shake assembly is controlled to be enabled; in the process of acquiring the second image frame, controlling the anti-shake assembly to be disabled, and controlling the lens assembly to move from the current position to the optical center of the lens assembly; and performing deblurring processing on the second image frame to obtain a target image frame corresponding to the second image frame. By adopting the method, sufficient time can be provided for the lens to move from the current position to the optical center direction, and a larger anti-shake angle is provided for anti-shake, so that the electronic equipment can generate a clear anti-shake image under the condition of large vibration.

Description

[0001] Cross-references

[0002] This application claims priority to Chinese patent application number 202311129982X, filed on September 1, 2023, and entitled “Image processing method, device, electronic device and computer-readable storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of image processing technology, and in particular to an image processing method, device, electronic device and computer-readable storage medium. Background Art

[0004] With the development of electronic devices such as smart phones, shooting technology has become increasingly popular. People can use cameras to take photos or videos anytime and anywhere. However, in the process of using the camera, the electronic device often shakes due to external influences, resulting in blurred images. Therefore, it is necessary to perform anti-shake processing on the electronic device.

[0005] At present, there are three main methods of anti-shake processing, namely electronic image stabilization (EIS, Electronics Image Stabilization), optical image stabilization (OIS, Optical Image Stabilization) and AI image stabilization (AIS, AI Image Stabilization).

[0006] However, with current anti-shake methods, when the vibration of the electronic device is large, the image generated is still not clear. Summary of the invention

[0007] The embodiments of the present application provide an image processing method, device, electronic device, and computer-readable storage medium, which can generate clearer images when the electronic device vibrates too much.

[0008] In a first aspect, the present application provides an image processing method. The method comprises:

[0009] In response to a first target signal, acquiring a first image frame and a second image frame; wherein, in the process of acquiring the first image frame, controlling the anti-shake component to be enabled;

[0010] In the process of acquiring the second image frame, controlling the anti-shake component to be disabled, and controlling the lens assembly to move from a current position toward an optical center direction of the lens assembly;

[0011] The second image frame is deblurred to obtain a target image frame corresponding to the second image frame.

[0012] In a second aspect, the present application also provides an image processing device. The device comprises:

[0013] An image acquisition module, configured to acquire a first image frame and a second image frame in response to a first target signal; wherein, in the process of acquiring the first image frame, the anti-shake component is controlled to be enabled;

[0014] a lens control module, used for controlling the anti-shake component to be disabled and controlling the lens component to move from a current position toward an optical center direction of the lens component during the process of acquiring the second image frame;

[0015] The blur processing module is used to perform a deblurring process on the second image frame to obtain a target image frame corresponding to the second image frame.

[0016] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0017] In response to a first target signal, acquiring a first image frame and a second image frame; wherein, in the process of acquiring the first image frame, controlling the anti-shake component to be enabled;

[0018] In the process of acquiring the second image frame, controlling the anti-shake component to be disabled, and controlling the lens assembly to move from a current position toward an optical center direction of the lens assembly;

[0019] The second image frame is deblurred to obtain a target image frame corresponding to the second image frame.

[0020] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0021] In response to a first target signal, acquiring a first image frame and a second image frame; wherein, in the process of acquiring the first image frame, controlling the anti-shake component to be enabled;

[0022] In the process of acquiring the second image frame, controlling the anti-shake component to be disabled, and controlling the lens assembly to move from a current position toward an optical center direction of the lens assembly;

[0023] The second image frame is deblurred to obtain a target image frame corresponding to the second image frame.

[0024] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0025] In response to a first target signal, acquiring a first image frame and a second image frame; wherein, in the process of acquiring the first image frame, controlling the anti-shake component to be enabled;

[0026] In the process of acquiring the second image frame, controlling the anti-shake component to be disabled, and controlling the lens assembly to move from a current position toward an optical center direction of the lens assembly;

[0027] The second image frame is deblurred to obtain a target image frame corresponding to the second image frame.

[0028] The above-mentioned image processing method, device, electronic device, computer-readable storage medium and computer program product can provide a collection time corresponding to at least one frame of the second image frame by controlling the anti-shake component to be disabled and controlling the lens component to move from the current position toward the optical center of the lens component during the process of acquiring the second image frame, so as to allow the lens component to move from the current position toward the optical center of the lens component, thereby improving the success rate of restoring the lens component to the optical center, and can meet the requirements of moving the lens component from the current position to the optical center in more scenarios, so that when the anti-shake component is restored and enabled, the lens is located at the optical center as much as possible, thereby having a larger anti-shake angle to stabilize the lens component, improving the anti-shake effect of the anti-shake component, and the electronic device can generate a clear anti-shake image even in the case of large vibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 A diagram showing an application environment of an image processing method in an embodiment;

[0031] Figure 2 is a flowchart of an image processing method in one embodiment;

[0032] Figure 3 A schematic diagram of capturing a first image frame and a second image frame in one embodiment;

[0033] Figure 4 is a schematic diagram of an image frame sequence in which a first image frame and a second image frame alternately appear in one embodiment;

[0034] Figure 5 is a schematic diagram of a lens assembly moving toward an optical center in one embodiment;

[0035] Figure 6 is a schematic diagram of an image frame sequence in which a first image frame and a second image frame alternately appear in another embodiment;

[0036] Figure 7 A flowchart of performing a deblurring process on a second image frame according to a first image frame to obtain a target image frame in one embodiment;

[0037] Figure 8 A flowchart of performing a deblurring process on a second image frame according to a first image frame to obtain a target image frame in another embodiment;

[0038] Fig. 9 A flowchart of performing a deblurring process on a second image frame according to a first image frame to obtain a target image frame in another embodiment;

[0039] Fig.10 is a flowchart of an image processing method in another embodiment;

[0040] Fig.11 is a flowchart of an image processing method in another embodiment;

[0041] Fig.12 A schematic diagram of controlling enabling and disabling of an anti-shake component in one embodiment;

[0042] Fig.13 is a flowchart of an image processing method in another embodiment;

[0043] Fig.14 is a structural block diagram of an image processing device in one embodiment;

[0044] Fig.15 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0046] The image processing method provided in the embodiment of the present application can be applied to Figure 1 The electronic device 100 includes a camera module 20, a processor 30 and a housing 40. The camera module 20 and the processor 30 are both arranged in the housing 40, and the housing 40 can also be used to install functional modules such as a power supply device and a communication device of the electronic device 100, so that the housing 40 provides dustproof, drop-proof, waterproof and other protections for the functional modules.

[0047] The camera module 20 can be a front camera module, a rear camera module, a side camera module, an under-screen camera module, etc., and is not limited here. The camera module 20 includes a lens assembly, an anti-shake assembly, and an image sensor 21. When the camera module 20 captures an image, light passes through the lens assembly and reaches the image sensor 21. The image sensor 21 is used to convert the light signal irradiated on the image sensor 21 into an electrical signal.

[0048] The electronic device 100 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. It is understandable that the electronic device 100 may also be any other device or equipment with an image processing function.

[0049] In one embodiment, Figure 2 As shown, an image processing method is provided, which is applied to Figure 1 Taking the electronic device in as an example, the method includes the following steps 202 to 206.

[0050] Step 202, in response to the first target signal, acquiring a first image frame and a second image frame; wherein, in the process of acquiring the first image frame, controlling the anti-shake component to be enabled.

[0051] The electronic device sequentially acquires a first image frame and a second image frame in response to a first target signal, wherein, in the process of acquiring the first image frame, the anti-shake component is controlled to be enabled, that is, the first image frame is acquired when the anti-shake component is in a working state, and therefore, the first image is also an anti-shake image. The first target signal indicates that the image sensor has completed the acquisition of the previous frame of image and has started to acquire the next frame of image. When each frame of image acquisition is completed, a first target signal is generated, and the acquisition of the next frame of image is started at the same time. The first target signal may be a pulse signal. It can be understood that the first target signal does not occupy the acquisition time of the previous frame of image or the next frame of image, and may be generated by hardware or software. For example, the first target signal is a Vsync (Vertical Synchronization) signal output by a GPO (General-purpose output) pin of the image sensor. The Vsync signal is used to indicate the state of a frame of image data, that is, the acquisition of the previous frame of image is completed and the acquisition of the next frame of image is started. The anti-shake component includes an optical anti-shake component (OIS), and the anti-shake corresponding to the optical anti-shake component is optical anti-shake.

[0052] It can be understood that when the electronic device generates the first target signal, it indicates that one frame of image has been acquired and the acquisition of the next frame of image has begun. For example, the electronic device generates the first target signal, responds to the first target signal, acquires the first image frame, and starts to acquire the second image frame; or, the electronic device responds to the first target signal, acquires the second image frame, and starts to acquire the first image frame, that is, the acquisition order of the first image frame and the second image frame is not specifically limited. However, usually, when the camera module is initialized, the anti-shake component is enabled, and the first image frame is acquired in response to the first first target signal, that is, the first image frame acquired by the electronic device is the first image frame.

[0053] Step 204: during the process of acquiring the second image frame, the anti-shake component is controlled to be disabled, and the lens assembly is controlled to move from the current position toward the optical center of the lens assembly.

[0054] In this embodiment, the electronic device controls the anti-shake component to be disabled during the process of acquiring the second image frame, and controls the lens component to move from the current position toward the optical center of the lens component to the target position. The target position includes the optical center or any position within a preset range centered on the optical center, for example, a preset range of 20%-30% of the lens movement stroke centered on the optical center, and the target position can be any position within the preset range. It is easy to understand that controlling the anti-shake component to be disabled means that the anti-shake component is in a non-working state, that is, the lens component is not anti-shake. The current position refers to the position of the lens component corresponding to the start of acquiring the second image frame. It can be understood that in the process of acquiring the image, when the lens component shakes due to external factors, the position of the lens component will change. Therefore, the lens component can be anti-shake by the anti-shake component to generate an anti-shake image.

[0055] Optionally, the electronic device controls the anti-shake component to be enabled, and in response to the first target signal, obtains the first image frame; controls the anti-shake component to be disabled, and in response to the first target signal and when the target condition is met, obtains the second image frame. The target condition is a condition that needs to be met to control the lens assembly to move from the current position to the optical center. For example, the target condition may be that the number of first image frames acquired continuously meets the target number of frames, or the first offset distance between the lens position and the optical center is greater than the first preset distance, or the second offset distance between the lens position and the stroke boundary is less than the second preset distance, etc. In response to the first target signal and when the target condition is met, the electronic device controls the anti-shake component to be disabled, obtains the second image frame corresponding to the first target signal to be collected, and controls the lens assembly to move from the current position to the optical center; in response to the first target signal but when the target condition is not met, the anti-shake component is controlled to be enabled, and obtains the first image frame corresponding to the first target signal to be collected. Furthermore, the anti-shake component is disabled at a low level, and the anti-shake component is enabled at a high level. That is to say, if the processor of the electronic device receives a low-level synchronization signal of the anti-shake component, the anti-shake component is controlled to be disabled; if the processor of the electronic device receives a high-level synchronization signal of the anti-shake component, the anti-shake component is controlled to be enabled.

[0056] In one example, a schematic diagram of obtaining a first image frame and a second image frame is shown in FIG. Figure 3 As shown, the target condition is that in response to the first target signal and the number of first image frames continuously acquired meets one frame, the electronic device turns on the camera module to start acquiring images, and when the camera module is initialized, the electronic device controls the anti-shake component to be enabled, and acquires the first frame of image, which is the first image frame; in response to the first first target signal, the first frame of image, that is, the first image frame acquisition is completed, and the second frame of image is acquired, the anti-shake component is controlled to be disabled, and the second image frame corresponding to the first first target signal to be acquired is acquired, that is, the second frame of image is the second image frame, and the lens assembly is controlled to move from the current position toward the optical center; in response to the second first target signal, the first frame of image, that is, the first image frame acquisition is completed, and the second frame of image is acquired. In response to the target signal, the second frame image acquisition is completed, the third frame image acquisition is started, the anti-shake component is controlled to be enabled, and the third frame image acquisition corresponding to the second first target signal is obtained, that is, the third frame image is the first image frame; in response to the third preset trigger signal, the third frame image acquisition is completed, the fourth frame image acquisition is started, the anti-shake component is controlled to be disabled, and the second frame image acquisition corresponding to the third preset trigger signal is obtained, that is, the fourth frame image is the second image frame, and the lens assembly is controlled to move from the current position toward the optical center, and so on, until the image sensor stops acquiring, and an image frame sequence alternating between the first image frame and the second graphic frame can be obtained. In an example, the image frame sequence alternating between the first image frame and the second graphic frame is as follows: Figure 4 shown.

[0057] When the camera module in the electronic device leaves the factory, the hardware parameters of the lens assembly, anti-shake assembly and image sensor have been set, such as the size of the lens assembly, the moving range of the lens assembly, the number of lenses contained in the lens assembly, the size of the anti-shake assembly, the material of the image sensor, etc. The center point of the moving range of the lens assembly is the optical center.

[0058] In an optional embodiment, controlling the lens assembly to move from the current position toward the optical center may be controlling any position on the lens assembly to move toward the optical center, for example, controlling the center of the lens assembly to move toward the optical center. It is understandable that the anti-shake angle of the lens assembly is the largest when the lens assembly is at the optical center. If the lens assembly is controlled to move toward the optical center at intervals of one frame and is moved to the optical center, then when the image sensor enters the state corresponding to the start of collecting the first image frame when the anti-shake assembly is enabled, the anti-shake angle is the largest, and each first image frame can be optically anti-shake at the maximum angle.

[0059] Step 206: Deblur the second image frame to obtain a target image frame corresponding to the second image frame.

[0060] The electronic device may perform a deblurring process on the second image frame to obtain a target image frame corresponding to the second image frame, and the clarity of the target image frame is higher than that of the second image frame. It is understandable that the second image frame may be deblurred by a deblurring algorithm or a blurring processing model to obtain a target image frame corresponding to the second image frame, and the specific manner of performing the deblurring process by the deblurring algorithm or the blurring processing model is not specifically limited herein.

[0061] Optionally, the second image frame may be deblurred according to the first image frame to obtain a target image frame corresponding to the second image frame. Further, the second image frame may be deblurred according to a first image frame adjacent to the second image frame to obtain a target image frame corresponding to the second image frame. For example, a deblurring convolution kernel corresponding to the second image frame may be determined according to the first image frame adjacent to the second image frame and the second image frame, and the second image frame may be deblurred according to the deblurring convolution kernel to obtain a target image frame corresponding to the second image frame.

[0062] Optionally, interpolation processing may be performed based on the first image frame to obtain a target image frame corresponding to the second image frame. For example, interpolation processing may be performed based on two first image frames to obtain a target image frame corresponding to the second image frame between the two first image frames. Since the target image frame is obtained by interpolation of the first image frame with higher definition, the definition of the target image frame is also higher than that of the second image frame.

[0063] Optionally, an image frame sequence consisting of the first image frame and the target image frame may be used as a target image sequence, and the target image sequence may be output.

[0064] In the above-mentioned image processing method, the first image frame and the second image frame are obtained by responding to the first target signal, wherein, in the process of obtaining the first image frame, the anti-shake component is controlled to be enabled, and in the process of obtaining the second image frame, the anti-shake component is controlled to be disabled, and the lens component is controlled to move from the current position toward the optical center. It is possible to provide a collection time corresponding to at least one frame of the second image frame for the lens component to move from the current position toward the optical center, thereby improving the success rate of moving the lens component from the current position to the optical center, and meeting the requirements of moving the lens component from the current position to the optical center in more scenarios, avoiding the situation where the lens component cannot be successfully moved from the current position to the optical center due to insufficient time, so that when the anti-shake component is enabled, the anti-shake angle corresponding to the lens component is larger, thereby improving the anti-shake effect of the anti-shake component, and even when the vibration amplitude of the electronic device is large, a clearer anti-shake image can be obtained.

[0065] In one embodiment, in response to a first target signal, acquiring a second image frame includes: in response to the first target signal, acquiring a lens position, and calculating a first offset distance between the lens position and the optical center; if the first offset distance is greater than the first target distance, acquiring the second image frame.

[0066] In this embodiment, during the process of shooting an image, the electronic device responds to the first target signal, obtains the position of the lens assembly, and calculates the first offset distance between the position of the lens assembly and the optical center; if the first offset distance is greater than the first target distance, a second image frame is obtained. The position of the lens assembly can be obtained by a Hall sensor, and the first offset distance between the position of the lens assembly and the optical center can be calculated in a camera coordinate system or a world coordinate system. If the first offset distance is greater than the first target distance, it means that the lens assembly deviates far from the optical center, and it is necessary to control the anti-shake component to be disabled, and control the lens assembly to move from the current position to the optical center. By detecting the first offset distance between the position of the lens assembly and the optical center, the size relationship between the first offset distance and the first target distance can be quickly determined, thereby quickly and accurately determining whether to control the lens assembly to move from the current position to the optical center.

[0067] In one example, a schematic diagram of the movement of the lens assembly position toward the optical center is shown in FIG. Figure 5As shown, the upward movement of the lens assembly on the coordinate axis X is taken as an example for explanation, the movement stroke of the lens assembly is L, and point O is the center point within the movement range of the lens assembly, that is, the optical center. When the lens assembly is at position x1, the distance between position x1 and the optical center O is greater than the first target distance, the second image frame is acquired, the anti-shake assembly is controlled to be disabled, and the lens assembly is controlled to move from position x1 to the optical center O, that is, the lens assembly is pulled back to the optical center O; when the lens is at position x2, in response to the first target signal, since the distance between position x2 and the optical center O is less than the first target distance, the lens assembly is not controlled to move from position x2 to the optical center O, the anti-shake assembly is controlled to be enabled, and the first image frame is acquired. An example of an image frame sequence corresponding to the acquired first image frame and the second image frame is shown in FIG. Figure 6 shown.

[0068] In an optional embodiment, the first target distance can be determined according to the movement stroke of the lens assembly, the movement stroke of the lens assembly includes the maximum movement stroke of the lens, and further, the first target distance is 60% to 80% of half of the maximum movement stroke of the lens assembly. For example, the first target distance is 80% of half of the maximum movement stroke of the lens assembly. Assuming that the maximum movement stroke of the lens assembly is 10 mm (millimeter), then half of the maximum movement stroke of the lens assembly is 5 mm, and 80% of half of the maximum movement stroke of the lens assembly is 4 mm, that is, the first target distance can be 4 mm. If the first offset distance between the position of the lens assembly and the optical center is greater than 4 mm, the second image frame is acquired, and the anti-shake assembly is disabled, and the lens assembly is controlled to move from the current position toward the optical center.

[0069] In the above embodiment, the target condition is that the first offset distance between the position of the lens assembly and the optical center is greater than the first target distance. In response to the first target signal, if the first offset distance is greater than the first target distance, the second image frame is acquired, and the anti-shake assembly is disabled at the same time, and the lens assembly is controlled to move from the current position toward the optical center. That is, in response to the first target signal, the lens assembly is controlled to move from the current position toward the optical center only after the lens assembly deviates from the optical center by a long distance. The number of times the lens assembly moves toward the optical center can be reduced, but each time the lens assembly moves toward the optical center, it has enough time to move to the optical center. When the anti-shake assembly is enabled, the lens assembly has the maximum anti-shake angle, thereby reducing the power consumption of the electronic device while ensuring a larger anti-shake angle.

[0070] In some embodiments, if the first offset distance is less than or equal to the first preset distance, the anti-shake component continues to be controlled to be enabled to acquire the first image frame.

[0071] In response to the first target signal, the position of the lens assembly is obtained, and the first offset distance between the position of the lens assembly and the optical center is calculated. If the first offset distance is less than or equal to the first preset distance, the anti-shake assembly is continuously controlled to be enabled, and the first image frame is obtained. It can be understood that the anti-shake assembly is controlled to be enabled, and the anti-shake assembly performs optical image stabilization on the lens assembly, that is, no other operations are performed on the lens assembly except for the optical image stabilization of the anti-shake assembly. In other words, in response to the first target signal, if the first offset distance is less than or equal to the first preset distance, the anti-shake assembly remains enabled, and the first image frame is obtained, until in response to the first target signal and the first offset distance is greater than the first target distance, the anti-shake assembly is controlled to be disabled, and the lens assembly is controlled to move from the current position toward the optical center.

[0072] In the above embodiment, if the first offset distance is less than or equal to the first target distance, the anti-shake component continues to be controlled to be enabled and the first image frame is obtained. This can increase the number of first image frames captured when the anti-shake component is enabled, that is, increase the number of anti-shake image frames, thereby improving the image clarity and reducing the number of times the lens assembly moves from the current position toward the optical center, thereby reducing the power consumption of the electronic device.

[0073] In some embodiments, after acquiring the second image frame, the method further includes: in response to a second target signal, controlling the anti-shake component to be enabled, acquiring the first image frame, and then executing in response to a third target signal, acquiring the position of the lens component, and calculating the first offset distance between the position of the lens component and the optical center, if the first offset distance is greater than the first target distance, acquiring the second image frame, and if the first offset distance is less than or equal to the first target distance, continuing to control the anti-shake component to be enabled, and acquiring the first image frame, wherein the generation time of the first target signal is earlier than the generation time of the second target signal, and the generation time of the second target signal is earlier than the generation time of the third target signal.

[0074] In this embodiment, the electronic device acquires the second image frame in response to the first target signal. Since the anti-shake component is disabled and the lens component is controlled to move from the current position toward the optical center of the lens component in the process of acquiring the second image frame, when responding to the second target signal, the position of the lens component is located at the target position, for example, at the optical center. At this time, the first offset distance between the lens component and the optical center is less than the first target distance, and the anti-shake component is controlled to be enabled to acquire the first image frame. Then, when responding to the third target signal, the position of the lens component is acquired, and the first offset distance between the position of the lens component and the optical center is calculated, and the size between the first offset distance and the first target distance is compared. If the first offset distance is greater than the first target distance, the second image frame is acquired. If the first offset distance is less than or equal to the first target distance, the first image frame is acquired. By analogy, an image frame sequence alternating between the first image frame and the second image frame can be obtained. Among them, the generation time of the first target signal is earlier than the generation time of the second target signal, and the generation time of the second target signal is earlier than the generation time of the third target signal. The first target signal, the second target signal and the third target signal can be the same signal generated at different times. For example, the first target signal is a pulse signal generated at the first moment, the second target signal is a pulse signal generated at the second moment, and the third target signal is a pulse signal generated at the third moment. The first moment is earlier than the second moment, and the second moment is earlier than the third moment.

[0075] In the above embodiment, during the process of acquiring the second image frame, since at least one frame of time is provided for the lens assembly to move toward the optical center, there is sufficient time for the lens assembly to move from the current position to the optical center; in response to the second target signal, the anti-shake component is enabled to acquire the first image frame, and in response to the third target signal, the second image frame is acquired only when it is determined that the first offset distance is greater than the first target distance, thereby increasing the number of frames of the first image frame and improving the clarity of the captured image. At the same time, the number of times the lens assembly moves from the current position toward the optical center is reduced, thereby reducing the power consumption of the electronic device.

[0076] In one embodiment, in response to a first target signal, acquiring a second image frame includes: in response to the first target signal, acquiring a lens assembly position, and calculating a second offset distance between the lens assembly position and a travel boundary; if the second offset distance is less than the second target distance, acquiring a second image frame.

[0077] The travel boundary refers to the boundary of the movable range of the lens assembly. The travel boundary can be set in advance. In this embodiment, the travel boundary that specifically participates in the calculation can be a boundary target point selected on the travel boundary, that is, the travel boundary includes the boundary target point. Since the shape of the travel boundary of the movable range of the lens is not necessarily a regular shape, the inflection point on the travel boundary can be selected as the boundary target point.

[0078] If the second offset distance is less than the second target distance, it means that the distance between the lens position and the stroke boundary is close and may be about to reach the stroke boundary. Then, the second image frame is acquired, and the anti-shake component is disabled, and the lens component is controlled to move from the current position toward the optical center. The second target distance can be determined according to the movement stroke of the lens component, and the lens movement stroke includes the maximum movement stroke of the lens. Further, the second target distance is 20% to 30% of half of the maximum movement stroke of the lens, wherein the first target distance can be greater than the second target distance.

[0079] Optionally, the distance between the position of the lens assembly and the boundary target point closest to the position of the lens assembly can be calculated as the second offset distance. If the second offset distance is less than the second target distance, the anti-shake assembly is controlled to be disabled, the second image frame is acquired, and the lens assembly is controlled to move from the current position toward the optical center.

[0080] In an optional embodiment, in response to a first target signal, the position of the lens assembly is acquired, and a second offset distance between the position of the lens assembly and the travel boundary is calculated; if the second offset distance between the position of the lens assembly and the travel boundary is not less than a second preset distance, the anti-shake assembly is enabled to acquire a first image frame.

[0081] In the above embodiment, in response to the first target signal, and when it is detected that the second offset distance is less than the second target distance, a second image frame is acquired, the anti-shake component is disabled, and the lens component is controlled to move from the current position toward the optical center. This can reduce the number of times the lens moves from the current position toward the optical center, thereby reducing the power consumption of the electronic device.

[0082] In some embodiments, after acquiring the second image frame, the step further includes: in response to a second target signal, controlling the anti-shake component to be enabled, acquiring the first image frame, and then executing in response to a third target signal, acquiring the position of the lens component, and calculating the second offset distance between the position of the lens component and the stroke boundary, if the second offset distance is less than the second target distance, acquiring the second image frame, wherein the generation time of the first target signal is earlier than the generation time of the second target signal, and the generation time of the second target signal is earlier than the generation time of the third target signal.

[0083] In this embodiment, in response to the first target signal, the second image frame is acquired. In response to the second target signal, since the lens assembly has moved to the optical center during the process of acquiring the second image frame, and the second offset distance is greater than or equal to the second target distance, the anti-shake assembly is controlled to be enabled, and the first image frame is acquired. In response to the third target signal, the position of the lens assembly is acquired, and the second offset distance between the position of the lens assembly and the stroke boundary is calculated, and the size relationship between the second offset distance and the second target distance is determined. If the second offset distance is less than the second target distance, the second image frame is acquired, and the anti-shake assembly is controlled to be disabled, and the lens assembly is controlled to move from the current position toward the optical center. If the second offset distance is greater than or equal to the second target distance, the anti-shake assembly is continuously controlled to be enabled, and the first image frame is acquired. By analogy, an image frame sequence alternating between the first image frame and the second image frame can be obtained.

[0084] In the above embodiment, in response to the first target signal, the second image frame is acquired, and in response to the second target signal, the anti-shake component is enabled to acquire the first image frame, until in response to the third target signal and when the second offset distance between the position of the lens component and the stroke boundary is less than the second target distance, the second image frame is acquired, the anti-shake component is disabled, and the lens component is controlled to move toward the optical center. The above process is continuously repeated until the image sensor stops capturing images. The number of first image frames can be increased, the overall clarity of the image can be improved, and at the same time the number of times the lens component moves toward the optical center can be reduced, thereby reducing the power consumption of the electronic device.

[0085] In one embodiment, acquiring the second image frame in response to the first target signal includes: acquiring the second image frame in response to the first target signal when the number of first image frames acquired continuously meets the target number of frames.

[0086] In this embodiment, it can be understood that when the electronic device starts to capture an image, the position of the lens assembly is initialized to the optical center, and in response to the first target signal, the anti-shake assembly is controlled to be enabled, and the first image frame is acquired, that is, after the first frame of the first image frame is acquired, the first first target signal is generated, and the second frame of the image is acquired. In response to the second target signal, the electronic device determines whether the number of first image frames acquired continuously meets the target number of frames. If the number of first image frames acquired continuously meets the target number of frames, the second image frame is acquired, the anti-shake assembly is controlled to be disabled, and the lens assembly is controlled to move from the current position toward the optical center; if the number of first image frames acquired continuously does not meet the target number of frames, the anti-shake assembly is continued to be controlled to be enabled, and the first image frame is acquired.

[0087] The target frame rate can be set according to the actual application scenario. For example, the target frame rate can be 1, 2, 3, 4 or 5.

[0088] In this embodiment, when the electronic device responds to the first target signal and continuously collects the first image frame of the target frame number, it obtains the second image frame, controls the anti-shake component to be disabled, and controls the lens component to move toward the optical center, so that the lens component has at least one frame time to move from the current position toward the optical center, so that there is sufficient time for the lens component to move to the optical center, so that when the anti-shake component is restored, the lens component has the maximum anti-shake angle, and the corresponding first image frame can perform optical anti-shake at the maximum angle, which can reduce the number of times the lens moves toward the optical center while ensuring a larger anti-shake angle, thereby reducing the power consumption of the electronic device.

[0089] In some embodiments, after acquiring the second image frame, it also includes: in response to a second target signal, controlling the anti-shake component to enable, acquiring the first image frame, and then executing the step of acquiring the second image frame in response to a third target signal and when the number of first image frames continuously acquired meets the target frame number, wherein the generation time of the first target signal is earlier than the generation time of the second target signal, and the generation time of the second target signal is earlier than the generation time of the third target signal.

[0090] In this embodiment, the electronic device responds to the first target signal to acquire the second image frame. During the process of acquiring the second image, the anti-shake component is controlled to be disabled, and the lens component is controlled to move toward the optical center. In response to the second target signal, the anti-shake component is controlled to be enabled to acquire the first image frame. In response to the third target signal, it is determined whether the number of first image frames continuously acquired meets the target frame number. If the number of first image frames continuously acquired meets the target frame number, the second image frame is acquired, the anti-shake component is controlled to be disabled, and the lens assembly is controlled to move toward the optical center, and so on, until the image sensor stops acquiring images and an image frame sequence alternating between the first image frames and the second image frames of the target frame number is acquired.

[0091] In the above embodiment, in response to the first target signal and when the number of first image frames continuously acquired meets the target frame number, the second image frame is acquired, and in response to the second target signal, the anti-shake component is controlled to be enabled to acquire the first image frame, until the first image frame of the target frame number is acquired after continuous acquisition, and then in response to the third target signal, the second image frame is acquired, that is, after acquiring the first image frame of the target frame number, a second image frame is acquired, and then the first image frame of the target frame number is acquired again, and so on. In the process of acquiring the second image frame, the anti-shake component is controlled to be disabled, and the lens assembly is controlled to move toward the optical center, and the number of first image frames can be adjusted according to the target frame number, thereby ensuring a larger anti-shake angle while reducing the power consumption of the electronic device.

[0092] In one embodiment, step 204 of performing a deblurring process on the second image frame to obtain a target image frame corresponding to the second image frame includes: performing a deblurring process on the second image frame according to the first image frame to obtain a target image frame corresponding to the second image frame.

[0093] The first image frame is an image frame acquired when the anti-shake component is enabled, that is, the first image frame is an anti-shake image after the anti-shake component is used to perform anti-shake, and is a relatively clear image; while the second image frame is an image acquired when the anti-shake component is disabled, and may be a relatively blurred image. Therefore, the second image frame can be deblurred using the clear first image frame to obtain a target image frame corresponding to the second image frame, so that the captured image has higher clarity.

[0094] Optionally, the second image frame may be deblurred using a first image frame adjacent to the second image frame to obtain a target image frame corresponding to the second image frame, and finally a target image sequence consisting of the first image frame and the target image frame is output.

[0095] In the above embodiment, since the first image frame and the second image frame are different frame images obtained by shooting the same picture, the second image frame is deblurred according to the first image frame to obtain a target image frame with higher clarity and more consistent with the actual shot picture.

[0096] In some embodiments, Figure 7 As shown, step 206 of performing deblurring processing on the second image frame according to the first image frame to obtain a target image frame corresponding to the second image frame includes the following steps 702 to 708.

[0097] Step 702: extract features from an image frame sequence consisting of a first image frame and a second image frame based on a feature extraction network to obtain a feature matrix sequence.

[0098] The feature extraction network refers to a network used for feature extraction, such as a convolutional neural network, AIGC (generative artificial intelligence, Artificial Intelligence Generated Content) or GAN (Generative adversarial network, generative adversarial network), etc. It can be understood that the corresponding feature extraction network can be selected for feature extraction according to the actual usage scenario.

[0099] Optionally, the image frame sequence consisting of one second image frame and multiple first image frames is subjected to feature extraction based on a feature extraction network to obtain a feature matrix sequence corresponding to the image frame sequence. For example, the image frame sequence is subjected to convolution processing to obtain a feature matrix sequence of the image frame sequence.

[0100] Step 704: Perform motion prediction on the feature matrix sequence to obtain motion vector information.

[0101] Motion prediction refers to predicting the motion trajectory of a target object. Motion prediction is performed on the feature matrix sequence to obtain motion vector information, where the motion vector information includes motion distance and motion direction.

[0102] Step 706, performing blur prediction according to the feature matrix sequence and the motion vector information to obtain a deblurred convolution kernel.

[0103] The electronic device performs blur prediction based on the feature matrix sequence and the motion vector information to obtain a deblurring convolution kernel corresponding to the second image frame. Optionally, the motion vector information is convolved to obtain motion convolution data; the motion convolution data is connected in series with the feature matrix sequence to obtain motion prediction data, and blur prediction is performed on the motion prediction data to obtain a deblurring convolution kernel corresponding to the second image frame.

[0104] Step 708: Deblur the second image frame according to the deblurred convolution kernel to obtain a target image frame corresponding to the second image frame.

[0105] Optionally, the deblurring process is to multiply the second image frame by a deblurring convolution kernel to obtain a product of the second image frame and the deblurring convolution kernel, and use the product as a target image frame corresponding to the second image frame.

[0106] Optionally, the motion information corresponding to the second image frame is aligned with the motion information corresponding to the first image frame to obtain the aligned motion information; the second image frame is deblurred according to the deblurred convolution kernel to obtain a deblurred image; the aligned motion information and the deblurred image are connected in series to obtain the first image feature, the first image feature is passed through a reconstruction network to obtain an image residual, the image residual is summed with the second image frame to obtain a target image frame with higher definition, and finally a target image sequence consisting of the first image frame and the target image frame is output. The sum of the image residual and the second image frame is the sum of the pixels at the corresponding positions of the corresponding channels.

[0107] In one example, the process of performing deblurring processing on the second image frame according to the first image frame to obtain the target image frame is as follows: Figure 8 As shown, the first image frame is an image frame adjacent to the second image frame before and after, that is, at least one second image frame and two first image frames form an image frame sequence, and feature extraction is performed based on a feature extraction network.

[0108] In the above embodiment, feature extraction is performed on an image frame sequence consisting of a first image frame and a second image frame based on a feature extraction network to obtain a feature matrix sequence, motion prediction is performed on the feature matrix sequence to obtain motion vector information, blur prediction is performed based on the feature matrix sequence and the motion vector information to obtain a deblurred convolution kernel, and deblurring is performed on the second image frame based on the deblurred convolution kernel to obtain a target image frame corresponding to the second image frame, thereby achieving deblurring of the second image frame, obtaining a target image frame with higher clarity, and improving the clarity of the captured image.

[0109] In some embodiments, the first image frame includes at least one of a first image frame acquired before the anti-shake component is disabled and a first image frame acquired when the anti-shake component is restored after being disabled.

[0110] In the process of deblurring the second image frame according to the first image frame to obtain the target image frame corresponding to the second image frame, the first image frame may be the first image frame obtained before the anti-shake component is disabled, or the first image frame obtained when the anti-shake component is restored after being disabled, or the first image frame obtained before the anti-shake component is disabled and the first image frame obtained when the anti-shake component is restored after being disabled. In other words, the second image frame may be deblurred using the first image frame obtained before the anti-shake component is disabled to obtain the target image frame; the second image frame may be deblurred using the first image frame obtained when the anti-shake component is restored after being disabled to obtain the target image frame; the second image frame may be deblurred using the first image frame obtained before the anti-shake component is disabled and the first image frame obtained when the anti-shake component is restored after being disabled to obtain the target image.

[0111] In the above embodiment, the first image frame obtained in different states of the anti-shake component can be selected to perform deblurring processing on the second image frame to obtain a target image frame corresponding to the second image frame, thereby improving the efficiency of the deblurring processing.

[0112] In some embodiments, controlling the lens assembly to move from a current position toward an optical center of the lens assembly includes: controlling the center of the lens assembly to move from the current position toward the optical center of the lens assembly.

[0113] In this embodiment, the center of the lens assembly is controlled to move from the current position toward the optical center, thereby controlling the lens assembly to move from the current position toward the optical center.

[0114] By controlling the center of the lens assembly to move from the current position toward the optical center, the center of the lens assembly can be determined more conveniently and quickly, thereby speeding up the movement of the lens assembly toward the optical center and further improving the success rate of the lens assembly moving to the optical center.

[0115] In an exemplary embodiment, Fig. 9 As shown, step 206 of performing deblurring processing on the second image frame to obtain a target image frame corresponding to the second image frame includes steps 902 to 904 below.

[0116] Step 902: extract a blurred image sequence from an initial image sequence formed by a first image frame set and a second image frame set.

[0117] The first image frame set is a set consisting of first image frames, the second image frame set is a set consisting of second image frames, and the initial image sequence consisting of the first image frame set and the second image frame set is an image sequence arranged in the order of acquisition time. It can be understood that the first image frame set may include one or more first image frames, and the second image frame set may include one or more second image frames.

[0118] It can be understood that the camera module also includes an image sensor. During the process of capturing the first image frame and the second image frame, the image sensor captures the images at a preset frequency. Therefore, in the initial image sequence, the interval between each two adjacent frames is the same, and the interval is the time corresponding to capturing a first image frame or a second image frame.

[0119] A blurred image sequence refers to an image sequence including at least two first image frames and at least one second image frame, and the second image frame is located between the first image frames. For example, the blurred image sequence is CBC, CBBC, CBBBC, CCBBC or CBBCC, where C represents the first image frame and B represents the second image frame.

[0120] Optionally, each blurred image sequence may be extracted from the initial image sequence in chronological order, and then the first image frame in each blurred image sequence may be interpolated to obtain a target image frame corresponding to the second image frame in the corresponding blurred image sequence.

[0121] Step 904 : performing frame interpolation processing based on the first image frame in the blurred image sequence to obtain a target image frame corresponding to the second image frame in the blurred image sequence.

[0122] Among them, interpolation refers to a technology that generates one or more image frames based on every two image frames. In this embodiment, interpolation can be performed based on two first image frames in the blurred image sequence to obtain target image frames corresponding to the second image frames. The clarity of the target image frames is higher than that of the second image frames, that is, two first image frames can be interpolated to obtain the same number of target image frames as the second image frames. Interpolation can be achieved through hardware design, AI (Artificial Intelligence) interpolation, or motion compensation-based interpolation.

[0123] Optionally, the first image frame in the blurred image sequence is interpolated based on the timestamp to obtain a target image frame corresponding to the acquisition timestamp of the second image frame.

[0124] In this embodiment, by extracting a blurred image sequence from an initial image frame formed by the first image frame and the second image frame, interpolation processing is performed based on the first image frame in the blurred image sequence to obtain a target image frame corresponding to the second image frame in the blurred image sequence. It is possible to obtain a clearer image corresponding to the second image frame through interpolation processing, that is, to achieve deblurring of the second image frame. Compared with deblurring through a model, the deblurring cost can be reduced and the fluency of the image sequence can be improved.

[0125] In some embodiments, the above method further comprises:

[0126] The second image frame in the blurred image sequence is replaced by the target image frame to obtain a target image sequence.

[0127] The target image sequence is an image sequence composed of images with higher definition. In a video playback scenario, the target image sequence can be played to improve the fluency of the played video.

[0128] Optionally, the second image frame in the blurred image sequence is replaced with the corresponding target image frame to obtain a target image sequence, and the target image sequence is displayed at a preset frame rate. The preset frame rate can be set as needed, for example, the preset frame rate is 30fps (frames per second), 60fps or 90fps, etc.

[0129] In actual application scenarios, after acquiring the first image frame and the second image frame, a blurred image sequence can be extracted in real time from the initial image sequence composed of the first image frame set and the second image frame set, and then interpolation processing is performed based on the first image frame in the blurred image sequence to obtain a target image corresponding to the second image frame in the blurred image sequence. That is, the blurred image sequence is extracted while acquiring the first image frame and the second image frame. As long as the blurred image sequence can be extracted, interpolation processing is performed based on the first image frame to obtain the target image frame corresponding to the second image frame, and then the second image frame is replaced with the target image frame to obtain the target image sequence, and the target image sequence is displayed. Alternatively, after acquiring all the first image frames and the second image frames, the blurred image sequence may be extracted in sequence from the initial image sequence composed of all the first image frames and all the second image frames, and then interpolation processing may be performed based on the first image frame in the blurred image sequence to obtain a target image corresponding to the second image frame in the blurred image sequence. That is, after acquiring the initial image sequence composed of all the first image frames and the second image frames, each blurred image sequence may be acquired in sequence from the initial image sequence, and then interpolation processing may be performed based on the first image frame in each blurred image sequence to obtain a target image frame corresponding to the second image frame, and then the second image frame may be replaced with the target image frame to obtain a target image sequence, and the target image sequence corresponding to each blurred image sequence may be displayed in sequence.

[0130] In this embodiment, the target image sequence is obtained by replacing the second image frame in the blurred image sequence with the target image frame, that is, an image sequence with higher clarity can be quickly obtained, and the continuity between the target image sequences can be improved. When the target image sequence is displayed, the clarity and smoothness of the displayed image sequence can be improved at the same time.

[0131] In some embodiments, step 902 of extracting a blurred image sequence from an initial image sequence consisting of a first image frame set and a second image frame set includes:

[0132] Two first image frames are selected from the initial image sequence, and at least one second image frame is included between the two first image frames; an image sequence corresponding to the two first image frames and the second image frame between the two first image frames is used as a fuzzy image sequence.

[0133] In this embodiment, in the initial image sequence, two first image frames are selected, and the second image frame is included between the two first image frames, and the second image frame can be one frame or multiple consecutive frames, and the image sequence composed of the two first image frames and the second image frame between the two first image frames is used as a blurred image sequence, so as to extract the blurred image sequence. It can be understood that the first image frame may be included between the two selected first image frames, or the first image frame may not be included. Regardless of whether the first image frame is included between the two selected first image frames or not, as long as the second image frame is located between the first image frames, the corresponding image sequence is a blurred image sequence.

[0134] Optionally, two first image frames are selected from the initial image sequence, and at least one second image frame is included between the two first image frames, and the first image frame may also be included between the two first image frames, that is, the blurred image sequence may include the first image frame at the head and tail positions and at least one second image frame, and the second image frame is located between the first image frames. Exemplarily, the blurred image sequence is CBC, CBBC, CBBCC or CBBBC, etc.

[0135] In one example, the search can be performed sequentially from the initial image sequence. If a first image frame is found, the search continues from the first image frame. If the second image frame and the first image frame are found sequentially, the image sequence corresponding to the two first image frames searched before and after and the second image frame between the two first image frames is used as a blurred image sequence.

[0136] In this embodiment, by using the image sequence corresponding to the two first image frames and the second image frame between the two first image frames as a blurred image sequence, it is possible to obtain a target image frame corresponding to the second image frame in the middle by interpolating the two first image frames, so that the obtained target image frame is more similar to the adjacent first image frame, and finally when the target image sequence obtained by replacing the second image frame with the target image frame is displayed, it will be smoother.

[0137] In some embodiments, step 904 of performing interpolation processing based on the first image frame in the blurred image sequence to obtain a target image frame corresponding to the second image frame in the blurred image sequence includes:

[0138] An interpolation process is performed based on the first image frame in the blurred image sequence to obtain a target image frame corresponding to the acquisition timestamp of the second image frame in the blurred image sequence.

[0139] In the process of collecting images, the image sensor collects images at a fixed frequency, such as collecting 30 frames per second, 60 frames per second, or 90 frames per second. The image collection frequency can be set as needed. Then, when the image collection frequency is fixed, the duration corresponding to each frame of image collection is the same, such as 1 / 30 second, 1 / 60 second, or 1 / 90 second. In the process of interpolation based on the first image frame in the blurred image sequence, the target image frame corresponding to the acquisition timestamp of the second image frame is obtained, that is, the number of second image frames included in the blurred image sequence is the same as the number of second image frames. For example, based on the interpolation of two first image frames, multiple interpolation image frames can be obtained. The multiple interpolation image frames have corresponding interpolation timestamps. The interpolation timestamps can be determined based on the acquisition timestamps corresponding to the two first image frames. Then, the interpolation image frame corresponding to the interpolation timestamp consistent with the acquisition timestamp of the second image frame is selected as the target image frame.

[0140] In one example, the blurred image sequence is CBC, then the average value of two C frames is calculated, that is, the pixel average value of the pixels at corresponding positions of the two C frames is calculated to obtain the target image frame C mid That is, for a blurred image sequence including a second image frame, a target image frame corresponding to the acquisition timestamp of the second image frame can be obtained by interpolating the frame by the average value. Then, the target image frame C mid Replace the middle B frame to get the target image sequence CC mid C.

[0141] In another example, the blurred image sequence is CBBBC, and the target image frame corresponding to the acquisition timestamp of the second image frame can be obtained by interpolation processing based on optical flow estimation. For example, optical flow estimation can be performed on two C frames to obtain the motion vector of each pixel between the two C frames, and the acquisition timestamp corresponding to each B frame can be obtained. Based on the acquisition timestamp, the motion vector, and the two C frames, the target image frame corresponding to each acquisition timestamp can be obtained. It can be understood that the electronic device can directly obtain the acquisition timestamp of each frame of the image.

[0142] In this embodiment, by performing interpolation processing based on the first image frame in the blurred image sequence, a target image frame corresponding to the acquisition timestamp of the second image frame in the blurred image sequence is obtained. This can make the target image frame more closely match the actual clear image at the corresponding moment, thereby improving the accuracy of the target image frame. Deblurring of the blurred frame is achieved by interpolating the clear frame, thereby also improving the deblurring effect.

[0143] In an exemplary embodiment, the interpolation method corresponding to the interpolation processing includes at least one of hardware-based interpolation, AI-based interpolation, or motion compensation-based interpolation.

[0144] Hardware-based interpolation refers to the technology of interpolating frames through hardware structure. For example, the hardware interpolation module can be integrated into the display chip, and the interpolation calculation can be completed by a dedicated hardware unit, which can reduce the computing pressure of the processor and avoid the problem of screen playback jamming caused by processor overload. The hardware interpolation module can be an interpolator or an interpolation device.

[0145] AI interpolation is a technology that uses artificial intelligence technology to improve the smoothness of videos. It increases the frame rate of the video by inserting new frames generated by AI calculations between every two frames of the original video, making the picture smoother and more natural and avoiding stuttering. AI interpolation technology is mainly based on deep learning algorithms. It predicts and generates intermediate frames that conform to the laws of motion by performing motion estimation and optical flow analysis on continuous frames in the video. In one example, the intermediate frame of two first image frames can be predicted based on a frame prediction model to obtain a target image frame, and then the second image frame is replaced with the target image frame. Among them, the frame prediction model is obtained by training based on an image training set, and the image training set includes a first training image and a second training image. The first training image and the second training image are images obtained by continuous acquisition, and the second training image is located in the middle of the first training image. By inputting the first training image into the basic model, the basic model learns the ability to output the second training image, and the basic model is trained to obtain a frame prediction model.

[0146] Interpolation based on motion compensation refers to the technology of interpolating by calculating the motion trajectory of an object in an image sequence. The motion trajectory of an object can be estimated by comparing the position changes of pixels or feature points in two frames of images, that is, obtaining the object motion vector, and then based on the motion vector and the two frames of images corresponding to the motion vector, an interpolated image between the two frames of images can be obtained, and the interpolated image reflects the motion state between the two frames of images. In an example, the corresponding motion vector can be obtained based on the two first image frames, and then based on the running vector and the two first image frames, the interpolated image located between the two first image frames is obtained, or the acquisition timestamp of the second image frame can be obtained, and the interpolated frame is obtained based on the running vector and the two first image frames to obtain the target image frame corresponding to the acquisition timestamp of the second image frame, and then the second image frame is replaced with the target image frame corresponding to the acquisition timestamp to obtain the target image sequence. When the target image sequence is displayed at a preset frame rate, the clarity and smoothness of the image sequence display can be improved.

[0147] It is understandable that one or more of the above interpolation methods can be selected according to actual application scenarios, and the specific selected method is not limited here.

[0148] In an exemplary embodiment, Fig.10 As shown, the image processing method includes the following steps 1002 to 1010.

[0149] Step 1002, in response to the first target signal, acquire a first image frame and a second image frame; in the process of acquiring the first image frame, control the anti-shake component to be enabled, and in the process of acquiring the second image frame, control the anti-shake component to be disabled, and control the lens assembly to move from the current position toward the optical center direction of the lens assembly.

[0150] Step 1004 : Select two first image frames from the initial image sequence formed by the first image frame set and the second image frame set, wherein at least one second image frame is included between the two first image frames.

[0151] Step 1006: Use the image sequence corresponding to the two first image frames and the second image frame between the two first image frames as a blurred image sequence.

[0152] Step 1008 , performing frame interpolation processing based on the first image frame in the blurred image sequence to obtain a target image frame corresponding to the acquisition timestamp of the second image frame in the blurred image sequence.

[0153] Step 1010: Replace the second image frame in the blurred image sequence with the target image frame to obtain a target image sequence.

[0154] In the above embodiment, by controlling the anti-shake component to be enabled during the process of acquiring the first image frame, and controlling the anti-shake component to be disabled during the process of acquiring the second image frame, and controlling the lens component to move from the current position toward the optical center direction of the lens component, it is possible to control the lens component to move from the current position toward the optical center direction through the acquisition time of at least one frame of image, so that there is enough time for the lens component to move from the current position to the optical center, and restore the maximum anti-shake angle corresponding to the enabled lens, so that the electronic device can also obtain a clearer anti-shake image when the vibration amplitude is large; in addition, for the second image frame acquired when the anti-shake component is disabled, by extracting the blurred image sequence corresponding to the two first image frames and the second image frame between the two first image frames, the first image frame in the blurred image sequence is interpolated to obtain the corresponding target image frame, so that the target image frame is used to replace the original second image frame, and the obtained target image sequences are all image sequences composed of clear images, so that the initial image sequence is deblurred, the clarity of the image sequence can be improved, and the display smoothness of the image sequence can also be improved.

[0155] In one embodiment, Fig.11As shown, the image processing method includes the following steps 1102 to 1112.

[0156] Step 1102, initializing the image sensor.

[0157] Initialize the image sensor to indicate that the image sensor can start capturing images.

[0158] Step 1104, control the anti-shake component to enable and obtain the first image frame.

[0159] Step 1106, in response to the first target signal and when the target condition is met, acquiring a second image frame, wherein during the process of acquiring the second image frame, the anti-shake component is disabled and the lens component is controlled to move from the current position toward the optical center.

[0160] Step 1108 , in response to the second target signal, returning to step 1104 to control the anti-shake component to enable and obtain the first image frame.

[0161] Step 1110 , deblurring the second image frame according to the first image frame to obtain a target image frame corresponding to the second image frame.

[0162] Step 1112: output an image frame sequence consisting of a first image frame and a target image frame.

[0163] In the above embodiment, in response to the first target signal and when the target condition is met, the second image frame is acquired, and at the same time, the anti-shake component is disabled and the lens component is controlled to move from the current position toward the optical center. Otherwise, the anti-shake component is enabled and the first image frame is acquired. When the anti-shake component is disabled, the lens component can be controlled to move from the current position toward the optical center by capturing at least one frame of image, so that there is enough time for the lens component to move from the current position to the optical center, and the lens after being enabled corresponds to the maximum anti-shake angle, so that a clearer anti-shake image can be acquired even when the vibration amplitude of the electronic device is large.

[0164] In some practical application scenarios, the optical image stabilization device corresponding to the non-exposure time range of the image sensor is disabled, and when the image sensor enters the next exposure time range adjacent to the non-exposure time range, the optical image stabilization device is controlled to be enabled. The enabled optical image stabilization device can drive the lens to follow the vibration of the electronic device based on the position after the movement to perform jitter compensation, and obtain a clear anti-shake image corresponding to the exposure. The relationship between the exposure situation and the working state of the optical image stabilization device is as follows: Fig.12 shown.

[0165] However, some electronic devices working in long exposure mode have a long exposure time and a short non-exposure time during the image capture process, for example, 3-5 milliseconds. It is often too late to control the lens to successfully move to the optical center within the non-exposure time range, resulting in the lens's anti-shake angle still being small within the exposure time, and the anti-shake effect is not ideal. In addition, if the image sensor's reading time is long, it will also squeeze the non-exposure time, resulting in a shorter non-exposure time.

[0166] Based on the above problems, an image processing method is provided. Fig.13 As shown, the electronic device includes a camera module, and the camera module includes a lens assembly, an anti-shake assembly and an image sensor. When the electronic device starts the camera module to shoot, the anti-shake assembly is enabled to obtain the first image frame; in the process of shooting the image, in response to the first target signal, the second image frame is obtained, the anti-shake assembly is disabled, and the lens assembly is controlled to move from the current position to the optical center; in response to the second target signal, the first image frame is obtained; wherein the optical center is the center point within the moving range of the lens assembly; in response to the third target signal, the second image frame is obtained, the anti-shake assembly is disabled, and the lens assembly is controlled to move from the current position to the optical center, and so on, each time a first image frame is separated, a second image frame is obtained, and the above process is repeated until the image sensor stops collecting images, and an image frame sequence alternating between the first image frame and the second image frame is obtained. Wherein, the generation time of the first target signal is earlier than the second target signal, and the generation time of the second target signal is earlier than the third target signal. The second image frame is deblurred according to the first image frame to obtain a clear target image frame corresponding to the second image frame, and finally an image frame sequence alternating between the target image frame and the first image frame is obtained and output.

[0167] The above-mentioned image processing method can provide the acquisition time corresponding to at least one second image frame, ensure that the motor successfully and stably moves the lens to the optical center, and fully stabilizes the lens, reducing the dependence on the stability auxiliary control and hardware performance, for example, reducing the dependence on PID (Proportional Integral Derivative) control, or the dependence on the sensor reading ability, etc., thereby improving the success rate of the lens assembly moving from the current position to the optical center. For example, for an image acquisition frame rate of 30 frames per second, the time required for each frame of image acquisition is 33 milliseconds, that is, at least 33 milliseconds of time are required for the lens assembly to move from the current position to the optical center and move to the optical center; for an image acquisition frame rate of 60 frames per second, the time required for each frame of image acquisition is 16.7 milliseconds, that is, at least 16.7 milliseconds of time are required for the lens assembly to move from the current position to the optical center and move to the optical center. In addition, the image processing method in the above-mentioned embodiment can also avoid the influence of the exposure time, and can be applied to more long exposure scenes, increasing the scope of application.

[0168] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0169] Based on the same inventive concept, the embodiment of the present application also provides an image processing device for implementing the above-mentioned image processing method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image processing device embodiments provided below can refer to the limitations on the image processing method above, and will not be repeated here.

[0170] In one embodiment, Fig.14 As shown, an image processing device is provided, including: an image acquisition module 1402, a lens control module 1404 and a blur processing module 1406, wherein:

[0171] The image acquisition module 1402 is used to acquire a first image frame and a second image frame in response to a first target signal; wherein, in the process of acquiring the first image frame, the anti-shake component is controlled to be enabled;

[0172] The lens control module 1404 is used to control the anti-shake component to be disabled and control the lens component to move from a current position toward the optical center of the lens component during the process of acquiring the second image frame;

[0173] The blur processing module 1406 is used to perform a deblurring process on the second image frame to obtain a target image frame corresponding to the second image frame.

[0174] In one embodiment, the image acquisition module 1402 is further used to:

[0175] In response to the first target signal, the position of the lens assembly is acquired, and a first offset distance between the position of the lens assembly and the optical center is calculated; if the first offset distance is greater than the first target distance, the second image frame is acquired.

[0176] In one embodiment, the above device further includes an offset distance calculation module, which is used to:

[0177] If the first offset distance is less than or equal to the first preset distance, continue to control the anti-shake component to be enabled to acquire the first image frame.

[0178] In one embodiment, the apparatus further comprises a first cycle management module, which is used to, after acquiring the second image frame,

[0179] In response to a second target signal, the anti-shake component is controlled to be enabled, a first image frame is acquired, and then in response to a third target signal, a position of the lens component is acquired, and a first offset distance between the position of the lens component and the optical center is calculated; if the first offset distance is greater than the first target distance, the second image frame is acquired; and if the first offset distance is less than or equal to the first target distance, the anti-shake component is continued to be controlled to be enabled, and the first image frame is acquired; the first target signal is generated earlier than the second target signal; the second target signal is generated earlier than the third target signal.

[0180] In one embodiment, the image acquisition module 1402 is further used to:

[0181] In response to the first target signal, the position of the lens assembly is acquired, and a second offset distance between the position of the lens assembly and a stroke boundary is calculated; if the second offset distance is less than the second target distance, the second image frame is acquired.

[0182] In one embodiment, the above-mentioned device also includes a second loop management module, which is used to: after acquiring the second image frame, in response to the second target signal, control the anti-shake component to be enabled, acquire the first image frame, and then execute in response to the third target signal, acquire the lens component position, and calculate the second offset distance between the lens component position and the stroke boundary; if the second offset distance is less than the second target distance, acquire the second image frame; the first target signal is generated earlier than the second target signal; the second target signal is generated earlier than the third target signal.

[0183] In one embodiment, the image acquisition module 1402 is further used to:

[0184] In response to the first target signal and when the number of first image frames acquired continuously meets the target number of frames, the second image frame is acquired.

[0185] In one embodiment, the above device further includes a third cycle management module, which is used to:

[0186] In response to a second target signal, the anti-shake component is controlled to be enabled, a first image frame is acquired, and then a step of acquiring the second image frame in response to a third target signal is executed when the number of first image frames acquired continuously meets the target number of frames; the first target signal is generated earlier than the second target signal; the second target signal is generated earlier than the third target signal.

[0187] In one embodiment, the blur processing module 1406 is further configured to: perform a deblurring process on the second image frame according to the first image frame to obtain a target image frame corresponding to the second image frame.

[0188] In one embodiment, the obfuscation processing module 1406 is further configured to:

[0189] The image frame sequence composed of the first image frame and the second image frame is subjected to feature extraction based on a feature extraction network to obtain a feature matrix sequence; motion prediction is performed on the feature matrix sequence to obtain motion vector information; blur prediction is performed based on the feature matrix sequence and the motion vector information to obtain a deblurring convolution kernel; and deblurring is performed on the second image frame based on the deblurring convolution kernel to obtain a target image frame corresponding to the second image frame.

[0190] In one embodiment, the image acquisition module 1402 is further used to control the center of the lens assembly to move from a current position toward the optical center of the lens assembly.

[0191] In some embodiments, the obfuscation processing module 1406 is further configured to:

[0192] A blurred image sequence is extracted from an initial image sequence composed of a first image frame set and a second image frame set; the blurred image sequence includes at least two first image frames and at least one second image frame, and the second image frame is located between the first image frames; interpolation processing is performed based on the first image frame in the blurred image sequence to obtain a target image frame corresponding to the second image frame in the blurred image sequence.

[0193] In some embodiments, the above-mentioned apparatus further includes an image frame replacement module, which is used to replace the second image frame in the blurred image sequence with the target image frame to obtain a target image sequence.

[0194] In some embodiments, the obfuscation processing module 1406 is further configured to:

[0195] Two first image frames are selected from the initial image sequence, and at least one second image frame is included between the two first image frames; an image sequence corresponding to the two first image frames and the second image frame between the two first image frames is used as a fuzzy image sequence.

[0196] In some embodiments, the obfuscation processing module 1406 is further configured to:

[0197] An interpolation process is performed based on the first image frame in the blurred image sequence to obtain a target image frame corresponding to the acquisition timestamp of the second image frame in the blurred image sequence.

[0198] Each module in the above image processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0199] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.15As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an image processing method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0200] Those skilled in the art will understand that Fig.15 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0201] The embodiment of the present application further provides a computer-readable storage medium, one or more non-volatile computer-readable storage media containing computer-executable instructions, which, when executed by one or more processors, enable the processors to perform the steps of the image processing method.

[0202] An embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the image processing method.

[0203] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0204] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0205] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0206] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image processing method, applied to a camera module, wherein the camera module includes an anti-shake component and a lens component, characterized in that: include: In response to a first target signal, acquiring a first image frame and a second image frame; wherein, in the process of acquiring the first image frame, controlling the anti-shake component to be enabled; In the process of acquiring the second image frame, controlling the anti-shake component to be disabled, and controlling the lens assembly to move from a current position toward an optical center direction of the lens assembly; The second image frame is deblurred to obtain a target image frame corresponding to the second image frame.

2. The method according to claim 1, characterized in that: In response to the first target signal, acquiring a second image frame includes: In response to the first target signal, acquiring a position of the lens assembly, and calculating a first offset distance between the position of the lens assembly and the optical center; If the first offset distance is greater than the first target distance, the second image frame is acquired.

3. The method according to claim 2, characterized in that The method further comprises: If the first offset distance is less than or equal to the first preset distance, continue to control the anti-shake component to be enabled to acquire the first image frame.

4. The method according to claim 3, characterized in that After acquiring the second image frame, the method further includes: In response to a second target signal, the anti-shake component is controlled to be enabled, a first image frame is acquired, and then in response to a third target signal, a position of the lens component is acquired, and a first offset distance between the position of the lens component and the optical center is calculated; if the first offset distance is greater than the first target distance, the second image frame is acquired; and if the first offset distance is less than or equal to the first target distance, the anti-shake component is continued to be controlled to be enabled, and the first image frame is acquired; the first target signal is generated earlier than the second target signal; the second target signal is generated earlier than the third target signal.

5. The method according to claim 1, characterized in that In response to the first target signal, acquiring a second image frame includes: In response to the first target signal, acquiring a lens assembly position, and calculating a second offset distance between the lens assembly position and a stroke boundary; If the second offset distance is less than the second target distance, the second image frame is acquired.

6. The method according to claim 5, characterized in that After acquiring the second image frame, the method further includes: In response to a second target signal, the anti-shake component is controlled to be enabled, a first image frame is acquired, and then in response to a third target signal, a position of the lens component is acquired, and a second offset distance between the position of the lens component and a stroke boundary is calculated; if the second offset distance is less than the second target distance, the step of acquiring the second image frame is performed; the first target signal is generated earlier than the second target signal; the second target signal is generated earlier than the third target signal.

7. The method according to claim 1, characterized in that In response to the first target signal, acquiring a second image frame includes: In response to the first target signal and when the number of first image frames acquired continuously meets the target number of frames, the second image frame is acquired.

8. The method according to claim 7, characterized in that After acquiring the second image frame, the method further includes: In response to a second target signal, the anti-shake component is controlled to be enabled, a first image frame is acquired, and then a step of acquiring the second image frame in response to a third target signal is executed when the number of first image frames acquired continuously meets the target number of frames; the first target signal is generated earlier than the second target signal; the second target signal is generated earlier than the third target signal.

9. The method according to any one of claims 1 to 8, characterized in that: The deblurring the second image frame to obtain a target image frame corresponding to the second image frame includes: According to the first image frame, the second image frame is deblurred to obtain a target image frame corresponding to the second image frame.

10. The method according to claim 9, characterized in that The deblurring process is performed on the second image frame according to the first image frame to obtain a target image frame corresponding to the second image frame, including: Performing feature extraction on an image frame sequence consisting of the first image frame and the second image frame based on a feature extraction network to obtain a feature matrix sequence; Performing motion prediction on the characteristic matrix sequence to obtain motion vector information; Perform blur prediction according to the feature matrix sequence and the motion vector information to obtain a deblurred convolution kernel; The second image frame is deblurred according to the deblurred convolution kernel to obtain a target image frame corresponding to the second image frame.

11. The method according to claim 9, characterized in that The first image frame includes at least one of a first image frame acquired before the anti-shake component is disabled and a first image frame acquired when the anti-shake component is restored after being disabled.

12. The method according to claim 1, characterized in that The controlling the lens assembly to move from a current position toward an optical center direction of the lens assembly comprises: The center of the lens assembly is controlled to move from a current position toward the optical center of the lens assembly.

13. The method according to claim 1, characterized in that The deblurring the second image frame to obtain a target image frame corresponding to the second image frame includes: Extracting a blurred image sequence from an initial image sequence consisting of a first image frame set and a second image frame set; the blurred image sequence includes at least two first image frames and at least one second image frame, and the second image frame is located between the first image frames; An interpolation process is performed based on the first image frame in the blurred image sequence to obtain a target image frame corresponding to the second image frame in the blurred image sequence.

14. The method according to claim 13, characterized in that The method further comprises: The second image frame in the blurred image sequence is replaced by the target image frame to obtain a target image sequence.

15. The method according to claim 13, characterized in that The step of extracting a blurred image sequence from an initial image sequence formed by the first image frame set and the second image frame set comprises: Selecting two first image frames from the initial image sequence, wherein at least one second image frame is included between the two first image frames; An image sequence corresponding to the two first image frames and the second image frame between the two first image frames is used as the blurred image sequence.

16. The method according to claim 13, characterized in that The performing interpolation processing based on the first image frame in the blurred image sequence to obtain a target image frame corresponding to the second image frame in the blurred image sequence includes: An interpolation process is performed based on the first image frame in the blurred image sequence to obtain a target image frame corresponding to the acquisition timestamp of the second image frame in the blurred image sequence.

17. The method according to claim 13, characterized in that The interpolation method corresponding to the interpolation processing includes at least one of hardware-based interpolation, AI-based interpolation, or motion compensation-based interpolation.

18. An image processing device, characterized in that: include: An image acquisition module, configured to acquire a first image frame and a second image frame in response to a first target signal; wherein, in the process of acquiring the first image frame, the anti-shake component is controlled to be enabled; a lens control module, used for controlling the anti-shake component to be disabled and controlling the lens component to move from a current position toward an optical center direction of the lens component during the process of acquiring the second image frame; The blur processing module is used to perform a deblurring process on the second image frame to obtain a target image frame corresponding to the second image frame.

19. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the computer program is executed by the processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 17.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 17 are implemented.

21. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 17 are implemented.

Images