Image processing method, electronic device, storage medium and program product

Abstract

The embodiment of the invention provides an image processing method, electronic equipment, a storage medium and a program product, and is applied to the technical field of terminals. In the method, the electronic device receives a shooting operation. The electronic device obtains first image data, wherein the first image data comprises reference frame image data. And the electronic equipment generates a first thumbnail according to the reference frame image data. And in the process of generating the first thumbnail, the electronic equipment obtains first motion data corresponding to the multiple frames of second image data. And the electronic equipment performs deblurring processing on the first image data according to the first motion data to generate a result image. The electronic device displays a third interface, the third interface is an application interface of the gallery application, and the third interface comprises the result image. The electronic equipment adopts the first motion data, and the timestamp of the multi-frame second image data is closer to the timestamp of the reference frame image data. Therefore, the image deblurring effect can be improved.

Description

Image processing methods, electronic devices, storage media and software products

[0001] This application claims priority to Chinese Patent Application No. 202411540838.X, filed with the State Intellectual Property Office of China on October 30, 2024, entitled "A Method and Electronic Device for Image Deblurring", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of terminal technology, and in particular to an image processing method, electronic device, storage medium, and program product. Background Technology

[0003] With the development of terminal technology, users have increasingly higher requirements for the camera function of electronic devices (such as mobile phones). When users take pictures with electronic devices, the images captured by the electronic devices are usually blurry due to factors such as the shaking of the electronic device, the movement of the subject, and / or the failure of the electronic device to focus accurately.

[0004] Currently, the motion data acquired by electronic devices for deblurring images undergoing optimization has a significant time delay compared to the image itself, resulting in poor deblurring performance. For example, after receiving a user's photo, the electronic device acquires an image with a timestamp of 8:00:00:00 milliseconds. The motion data used by the electronic device to perform deblurring on this image carries a timestamp of 8:00:00:30 milliseconds. When the electronic device uses this 8:00:00:30 milliseconds motion data to deblurr the image at 8:00:00:00 milliseconds, the 30-millisecond delay leads to poor deblurring results.

[0005] Therefore, how to improve the deblurring effect of images is a problem that needs to be solved. Summary of the Invention

[0006] This application provides an image processing method, electronic device, storage medium, and program product that can acquire motion data with a smaller time delay between the image to be optimized and the image to be optimized, and perform deblurring processing on the image to be optimized based on the motion data. This improves the deblurring effect of the image to be optimized.

[0007] In a first aspect, embodiments of this application provide an image processing method applied to an electronic device, which may include a camera application and a camera, the method comprising:

[0008] After the camera application is launched, the electronic device captures raw image data in real time through the camera. Then, the electronic device displays a first interface, which is a shooting preview interface. The electronic device can receive shooting operations triggered by the user in the first interface. Next, the electronic device acquires first image data, which includes reference frame image data. The first image data can refer to multiple frames of raw image data acquired by the electronic device from the raw image data captured in real time by the camera based on the received shooting operation. The electronic device can determine one frame from these multiple frames of raw image data as the reference frame image data. Then, the electronic device generates a first thumbnail based on the reference frame image data; the first thumbnail can be in YUV format. During the generation of the first thumbnail, the electronic device acquires first motion data corresponding to multiple frames of second image data. After generating the first thumbnail, the electronic device generates a second thumbnail based on the first thumbnail. The electronic device displays a second interface, which includes the second thumbnail. After acquiring the first image data, the electronic device processes the first image data to generate a result image. The processing of the first image data includes: the electronic device performing deblurring processing on the first image data based on the first motion data. The electronic device displays a third interface, which is the application interface of the gallery application and includes the result image.

[0009] In this embodiment, the electronic device acquires first motion data corresponding to multiple frames of second image data during the generation of the second thumbnail. The second thumbnail is generated by the electronic device based on reference frame image data acquired at the moment the electronic device receives the shooting operation. The first motion data is used to deblur the first image data when the module in the electronic device that performs deblurring processing (i.e., the processing module hereinafter referred to as the processing module) receives the first image data. Therefore, it can be understood that the electronic device acquires the first motion data between the moment it receives the shooting operation and the moment its processing module receives one frame of image data from the first image data.

[0010] On the one hand, since motion data is calculated by electronic devices based on raw image data, the time when the electronic device acquires motion data is lagging behind the time when the raw image data is acquired. In other words, when acquiring motion data and reference frame image data at the moment of receiving the shooting operation, the timestamp of the motion data acquired by the electronic device is much earlier than the timestamp of the reference frame image data.

[0011] On the other hand, because the process of processing the first image data by the electronic device is relatively complex, the processing module in the electronic device receives a frame of image data from the first image data at a relatively late time. That is to say, when the electronic device acquires motion data only when the processing module receives a frame of image data from the first image data, the timestamp of the motion data is significantly later than the timestamp of the reference frame image data.

[0012] Therefore, in this embodiment, the timestamp of the first motion data obtained between the moment the shooting operation is received and the moment the processing module in the electronic device receives a frame of image data from the first image data is closer to the timestamp corresponding to the reference frame image data. Consequently, the electronic device achieves better deblurring effect when using the first motion data to perform deblurring processing on the first image data.

[0013] Moreover, the first motion data acquired by the electronic device can, to some extent, make up for the motion data missed between the moment the shooting operation is received and the moment the processing module in the electronic device receives a frame of image data from the first image data. This can increase the probability that the electronic device can determine the motion data that matches the reference frame image data, thereby improving the success rate of deblurring.

[0014] In one possible implementation of the first aspect, after the electronic device receives the shooting operation, the method further includes: the electronic device acquiring second motion data corresponding to multiple frames of third image data. The electronic device performs deblurring processing on the first image data based on the first motion data, including: when the multiple frames of second image data meet preset conditions, the electronic device performs deblurring processing on the first image data based on the first motion data. The preset conditions include: the absolute value of the difference between the timestamp of a frame of image data and a reference timestamp is less than a preset threshold, and / or, the absolute value of the difference between the timestamp of that frame of image data and the reference timestamp is less than the absolute value of the difference between the first timestamp and the reference timestamp; the first timestamp is the timestamp of any frame of image data other than the frame of image data in the multiple frames of second image data and the multiple frames of third image data. The reference timestamp is the timestamp of a reference frame of image data.

[0015] In this implementation, when the electronic device also acquires second motion data corresponding to multiple frames of third image data, it uses the first motion data to perform deblurring when the timestamps of the multiple frames of second image data are closer to the reference timestamp. In other words, even if the timestamps of the multiple frames of third image data acquired by the electronic device are too early compared to the reference timestamp, the electronic device can still use the first motion data to perform deblurring. This improves the deblurring effect.

[0016] In one possible implementation of the first aspect, the method further includes: when multiple frames of third image data meet preset conditions, the electronic device performs deblurring processing on the first image data based on the second motion data.

[0017] In this implementation, when the electronic device acquires second motion data corresponding to multiple frames of third image data and first motion data corresponding to multiple frames of second image data, the electronic device uses the second motion data to perform deblurring when the timestamp of the multiple frames of third image data is closer to the reference timestamp. That is, even if the timestamp of the acquired multiple frames of second image data is too far behind the reference timestamp, the electronic device can still use the second motion data to perform deblurring. This improves the deblurring effect.

[0018] In one possible implementation of the first aspect, the method further includes: when neither the multiple frames of second image data nor the multiple frames of third image data satisfy a preset condition, the electronic device acquires third motion data corresponding to the multiple frames of fourth image data. The electronic device can then perform deblurring processing on the first image data based on the third motion data.

[0019] In this implementation, if neither multiple frames of second image data nor multiple frames of third image data meet the preset conditions, the electronic device acquires the third motion data corresponding to multiple frames of fourth image data. This gives the electronic device more options for the motion data used in deblurring, thus increasing the probability of successful deblurring.

[0020] In one possible implementation of the first aspect, the electronic device performs deblurring processing on the first image data based on the first motion data, including: the electronic device performs deblurring processing on the first image data based on the first motion data corresponding to the target image data in multiple frames of second image data. Wherein, the target image data is: image data in the multiple frames of second image data whose absolute value of the difference between the timestamp and the reference timestamp is less than a preset threshold, and / or, the image data corresponding to the smallest first time difference among the multiple frames of second image data; the reference timestamp is the timestamp of the reference frame image data; the first time difference corresponding to any second image data is: the absolute value of the difference between the timestamp of that second image data and the reference timestamp.

[0021] In this implementation, based on the first motion data corresponding to the acquired multi-frame second image data, the electronic device further selects the first motion data corresponding to the second image data that is closer to the reference timestamp from the multi-frame second image data to perform deblurring processing on the first image data, thereby further improving the deblurring effect.

[0022] It should be understood that the process by which the electronic device performs deblurring on the first image data based on the second motion data and the process by which the electronic device performs deblurring on the first image data based on the third motion data are similar to the process by which the electronic device performs deblurring on the first image data based on the first motion data.

[0023] For example, the electronic device performs deblurring processing on the first image data based on the second motion data, including: the electronic device performs deblurring processing on the first image data based on the second motion data corresponding to the target image data in multiple frames of third image data. The target image data is: image data in the multiple frames of third image data where the absolute value of the difference between the timestamp and the reference timestamp is less than a preset threshold, and / or, the image data corresponding to the smallest second time difference among the multiple frames of third image data; the second time difference corresponding to any third image data is: the absolute value of the difference between the timestamp of that third image data and the reference timestamp.

[0024] For example, the electronic device performs deblurring processing on the first image data based on the third motion data, including: the electronic device performs deblurring processing on the first image data based on the third motion data corresponding to the target image data in multiple frames of fourth image data. The target image data is: image data in the multiple frames of fourth image data whose absolute value of the difference between the timestamp and the reference timestamp is less than a preset threshold, and / or, the image data corresponding to the smallest third time difference among the multiple frames of fourth image data; the third time difference corresponding to any fourth image data is: the absolute value of the difference between the timestamp of that fourth image data and the reference timestamp.

[0025] In one possible implementation of the first aspect, the first motion data corresponding to one frame of second image data includes: pixel velocities of pixels in multiple frames of second image data. The electronic device performs deblurring processing on the first image data based on the first motion data, including: the electronic device calculating the pixel velocity corresponding to the target image data based on the pixel velocities of pixels in the multiple frames of second image data within the first motion data corresponding to the target image data; and then, the electronic device performs deblurring processing on the first image data based on the pixel velocity corresponding to the target image data.

[0026] In this implementation, the electronic device performs deblurring based on the pixel velocity of the target image data calculated from the pixel velocities of pixels in multiple frames of original image data. In other words, the pixel velocity of the target image data used for deblurring in this embodiment takes into account the influence of the pixel velocities in multiple frames of original image data. Therefore, when there is a large deviation in the actual pixel velocity values ​​of the target image data, the deviation in the calculated pixel velocity of the target image data will decrease. The larger the deviation in pixel velocity, the worse the deblurring effect using that pixel velocity. Therefore, compared to directly using the actual pixel velocity values ​​of the target image data for deblurring, the pixel velocity of the target image data that considers the pixel velocities in multiple frames of original image data is more effective in this embodiment.

[0027] It should be understood that the content included in the second motion data corresponding to the third image data, and the content included in the third motion data corresponding to the fourth image data, are similar to the content included in the first motion data corresponding to the second image data. The process by which the electronic device performs deblurring processing on the first image data based on the second motion data, and the process by which the electronic device performs deblurring processing on the first image data based on the third motion data, are similar to the process by which the electronic device performs deblurring processing on the first image data based on the first motion data.

[0028] For example, the second motion data corresponding to one frame of third image data includes the pixel velocity of pixels in multiple frames of third image data. The electronic device performs deblurring processing on the first image data based on the second motion data, including: the electronic device calculates the pixel velocity corresponding to the target image data based on the pixel velocity of pixels in multiple frames of third image data within the second motion data corresponding to the target image data. Then, the electronic device performs deblurring processing on the first image data based on the pixel velocity corresponding to the target image data.

[0029] For example, the third motion data corresponding to one frame of fourth image data includes the pixel velocity of pixels in multiple frames of fourth image data. The electronic device performs deblurring processing on the first image data based on the third motion data, including: the electronic device calculates the pixel velocity corresponding to the target image data based on the pixel velocity of pixels in multiple frames of fourth image data within the third motion data corresponding to the target image data. Then, the electronic device performs deblurring processing on the first image data based on the pixel velocity corresponding to the target image data.

[0030] In one possible implementation of the first aspect, the electronic device calculates the pixel velocity corresponding to the target image data based on the pixel velocities of pixels in multiple frames of second image data within the first motion data corresponding to the target image data. This includes: the electronic device fitting the pixel velocities of pixels in the multiple frames of second image data within the first motion data corresponding to the target image data to obtain a fitting function. The fitting function reflects the correspondence between the pixel velocity and the timestamps of the image data. Then, the electronic device inputs the timestamps of the target image data into the fitting function to obtain the pixel velocity corresponding to the target image data.

[0031] In this implementation, the pixel velocity corresponding to the target image data is not the pixel velocity of pixels in a single frame of image data, but rather a pixel velocity calculated by fitting the pixel velocities of pixels in multiple frames of original image data. Therefore, when there is a large deviation in the pixel velocity of pixels in the original image data at a certain time point, after that original image data at that time point is determined as the target image data, the deviation in the pixel velocity corresponding to the target image data will be smaller compared to the pixel velocity of pixels in the original image data at that time point. The larger the deviation in pixel velocity, the worse the deblurring effect using that pixel velocity. Therefore, in this embodiment, the pixel velocity corresponding to the target image data with a smaller deviation, obtained through fitting calculation, provides a better deblurring effect.

[0032] It should be understood that the process by which an electronic device calculates the pixel velocity of the target image data based on the pixel velocity of pixels in multiple frames of third image data, and the process by which an electronic device calculates the pixel velocity of the target image data based on the pixel velocity of pixels in multiple frames of fourth image data, are similar to the process by which an electronic device calculates the pixel velocity of the target image data based on the pixel velocity of pixels in multiple frames of second image data.

[0033] For example, the electronic device calculates the pixel velocity corresponding to the target image data based on the pixel velocities of pixels in multiple frames of third image data within the second motion data corresponding to the target image data. This includes: the electronic device fitting the pixel velocities of pixels in multiple frames of third image data within the second motion data corresponding to the target image data to obtain a fitting function. Then, the electronic device inputs the timestamp of the target image data into the fitting function to obtain the pixel velocity corresponding to the target image data.

[0034] For example, the electronic device calculates the pixel velocity corresponding to the target image data based on the pixel velocities of pixels in multiple frames of fourth image data within the third motion data corresponding to the target image data. This includes: the electronic device fitting the pixel velocities of pixels in multiple frames of fourth image data within the third motion data corresponding to the target image data to obtain a fitting function. Then, the electronic device inputs the timestamp of the target image data into the fitting function to obtain the pixel velocity corresponding to the target image data.

[0035] In one possible implementation of the first aspect, the first motion data corresponding to a frame of second image data further includes: object distance and / or gyroscope information. The object distance is the distance between the electronic device and the object being photographed when the electronic device captures the second image data; the gyroscope information includes the angular velocity information of the electronic device when the electronic device captures the second image data.

[0036] In this embodiment, the electronic device also considers object distance and gyroscope information when performing deblurring. Since the degree of blurring of the original image data varies depending on the object distance, and also varies depending on the motion posture of the electronic device (e.g., whether the phone is shaking or not), considering object distance and gyroscope information during deblurring can further improve the image deblurring effect.

[0037] It should be understood that the second motion data corresponding to a frame of third image data may also include: object distance and / or gyroscope information. Similarly, the third motion data corresponding to a frame of fourth image data may also include: object distance and / or gyroscope information.

[0038] In one possible implementation of the first aspect, after the electronic device displays the first interface, the process includes: the electronic device acquiring image data in real time; the electronic device detecting and storing motion data corresponding to the image data in real time; and then, the electronic device updating the stored motion data, storing motion data corresponding to the latest detected preset number of image data. Specifically, the electronic device acquiring first motion data corresponding to multiple frames of second image data includes: the electronic device acquiring motion data corresponding to the currently stored preset number of image data.

[0039] In this implementation, the electronic device stores the motion data corresponding to the image data of the latest detected preset data in real time. Therefore, the electronic device can obtain the latest motion data every time it acquires motion data.

[0040] It should be understood that the way electronic devices acquire second motion data corresponding to multiple frames of third image data and the way electronic devices acquire third motion data corresponding to multiple frames of fourth image data are similar to the way electronic devices acquire first motion data corresponding to multiple frames of second image data.

[0041] For example, after the electronic device displays the first interface, the process includes: the electronic device acquiring image data in real time; the electronic device detecting and storing motion data corresponding to the image data in real time; and then, the electronic device updating the stored motion data, storing motion data corresponding to the latest detected preset number of image data. Specifically, the electronic device acquiring second motion data corresponding to multiple frames of third image data includes: the electronic device acquiring motion data corresponding to the currently stored preset number of image data.

[0042] For example, after the electronic device displays the first interface, the process includes: the electronic device acquiring image data in real time; the electronic device detecting and storing motion data corresponding to the image data in real time; and then, the electronic device updating the stored motion data, storing motion data corresponding to the latest detected preset number of image data. Specifically, the electronic device acquiring third motion data corresponding to multiple frames of fourth image data includes: the electronic device acquiring motion data corresponding to the currently stored preset number of image data.

[0043] In one possible implementation of the first aspect, after the electronic device generates the resulting image, the method further includes: the electronic device replacing the second thumbnail with a third thumbnail, the third thumbnail being the thumbnail corresponding to the resulting image. The electronic device displays a third interface, including: in response to a triggering operation on the third thumbnail, the electronic device displays the third interface.

[0044] In this implementation, the resulting image is obtained by deblurring the motion data corresponding to image data whose timestamp is closer to the reference timestamp. Therefore, the image seen by the user on the third interface of the electronic device is likely to be clearer, thus improving the user experience to some extent.

[0045] In one possible implementation of the first aspect, after the electronic device generates the resulting image, the method further includes: the electronic device displaying a fourth interface, the fourth interface being an application interface of a gallery application, the fourth interface including a third thumbnail, the third thumbnail being a thumbnail corresponding to the resulting image. The electronic device displaying the third interface includes: in response to a triggering operation on the third thumbnail, the electronic device displaying the third interface.

[0046] In this implementation, the resulting image is obtained by deblurring the motion data corresponding to image data whose timestamp is closer to the reference timestamp. Therefore, the image seen by the user on the third interface of the electronic device is likely to be clearer, thus improving the user experience to some extent.

[0047] Secondly, embodiments of this application provide an electronic device, including: a memory and a processor, the processor including a first processing unit and a second processing unit. The memory and the processor are coupled. The memory is used to store computer program code, the computer program code including computer instructions. When the processor executes the computer instructions, it causes the electronic device to perform the method as described in the first aspect and its possible implementations.

[0048] Thirdly, embodiments of this application provide a computer-readable storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in the first aspect and its possible implementations.

[0049] Fourthly, embodiments of this application provide a chip system applied to an electronic device. The chip system includes one or more processors, which are used to invoke computer instructions to cause the electronic device to perform the methods described in the first aspect and their possible implementations.

[0050] Fifthly, embodiments of this application provide a computer program product that, when run on a computer, causes the computer to perform the method as described in the first aspect and its possible implementations.

[0051] It is understood that the beneficial effects achieved by the electronic device described in the second aspect, the computer-readable storage medium described in the third aspect, the chip system described in the fourth aspect, and the computer program product described in the fifth aspect can be referred to the beneficial effects of the first aspect and any possible implementation thereof, which will not be repeated here. Attached Figure Description

[0052] Figure 1 is a schematic diagram of a photographing scene provided in an embodiment of this application;

[0053] Figure 2 is a schematic diagram of the position change of a pixel corresponding to a moving object according to an embodiment of this application;

[0054] Figure 3 is a schematic diagram of an image and data used to deblur the image according to an embodiment of this application;

[0055] Figure 4 is a schematic diagram of an image deblurring effect provided in an embodiment of this application;

[0056] Figure 5 is a schematic diagram of another image and data used to perform deblurring on the image provided in an embodiment of this application;

[0057] Figure 6 is a schematic diagram of another image deblurring effect provided in an embodiment of this application;

[0058] Figure 7 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0059] Figure 8 is a schematic diagram of the software structure of an electronic device provided in an embodiment of this application;

[0060] Figure 9 is a schematic diagram of the framework of an image processing method provided in an embodiment of this application;

[0061] Figure 10 is a flowchart illustrating an image processing method provided in an embodiment of this application;

[0062] Figure 11 is a schematic diagram of data storage in a motion detection manager according to an embodiment of this application;

[0063] Figure 12 is a flowchart illustrating another image processing method provided in an embodiment of this application;

[0064] Figure 13 is a schematic diagram of data storage in another motion detection manager provided in an embodiment of this application;

[0065] Figure 14 is a schematic diagram of an interface provided in an embodiment of this application;

[0066] Figure 15 is a schematic diagram of another interface provided in an embodiment of this application;

[0067] Figure 16 is a schematic diagram of another interface provided in an embodiment of this application;

[0068] Figure 17 is a schematic diagram of the hardware structure of another electronic device provided in an embodiment of this application;

[0069] Figure 18 is a schematic diagram of a chip system provided in an embodiment of this application. Detailed Implementation

[0070] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0071] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner to facilitate understanding.

[0072] With the development of terminal technology, users have increasingly higher requirements for the camera function of electronic devices (such as mobile phones).

[0073] For example, taking a mobile phone as an electronic device, the following describes a photography scenario in conjunction with Figure 1. Figure 1 is a schematic diagram of a photography scenario provided by an embodiment of this application.

[0074] As shown in Figure 1, the mobile phone desktop 100 may include clock application icons, weather application icons, smart life application icons, settings application icons, recorder application icons, browser application icons, gallery application icons, messaging application icons, contacts application icons, phone application icons, and camera application icons 101, etc.

[0075] In response to a user's trigger action on the camera application icon 101 on the phone's home screen 100, the phone can activate the camera. The camera can then capture raw image data (also known as image data) in real time. The phone can then display the shooting preview interface 110 as shown in Figure 1, based on the user's trigger action on the camera application icon 101 on the phone's home screen 100 and the raw image data captured by the camera.

[0076] As shown in the shooting preview interface 110, this interface includes an "Aperture" option button, a "Night Scene" option button, a "Portrait" option button, a "Take Photo" option button, a "Video" option button, a "Pro" option button, a "More" option button, a preview area 111, a thumbnail display area 113, a shooting button 112, and a front-facing camera / rear-facing camera switch button. The image displayed in the preview area 111 can be obtained based on raw image data captured in real-time by the camera. The format of the raw image data captured by the phone's camera can be RAW format. Each raw image data carries a timestamp indicating when it was captured. The thumbnail display area 113 can display a thumbnail of the image most recently captured by the phone based on the user's triggering of the shooting button 112.

[0077] When a user takes a photo using an electronic device, the image captured may be blurry due to factors such as device shake, movement of the subject, and / or the device's inaccurate focus (also known as camera defocusing). Therefore, after the user clicks the shutter button 112, the electronic device can perform deblurring on the captured image (hereinafter referred to as the image to be optimized) to obtain a clearer image.

[0078] In some implementations, electronic devices can perform deblurring based on motion data of the image, such as pixel speed, thereby eliminating the blurring effect caused by motion.

[0079] The pixel velocity of a frame can refer to the rate at which the position of the pixel corresponding to a moving object changes between consecutive frames in that frame.

[0080] For example, Figure 2 is a schematic diagram of the position change of a pixel corresponding to a moving object provided in an embodiment of this application.

[0081] As shown in Figure 2, images 200 and 210 are adjacent frames, where the timestamp of image 200 is t. i-1 The timestamp of image 210 is t i Timestamp t i-1 and timestamp t i The time difference between them is Δt. The coordinates of pixel 201 corresponding to the center point of the moving object 220 in image 200 are (a, b), and the coordinates of pixel 211 corresponding to the center point of the moving object 220 in image 210 are (c, d). Therefore, the position change rate of the pixel corresponding to the center point of the moving object 220 in the x-axis direction can be expressed as (ca) / Δt; the position change rate of the pixel corresponding to the center point of the moving object 220 in the y-axis direction can be expressed as (db) / Δt. The timestamp is t. i The position change rate of pixel 211 in image 210 can include the position change rate of pixel 211 corresponding to the center point of the moving object 220 in the x-axis direction and the position change rate of pixel 211 in the y-axis direction. The timestamp is t. i The pixel velocity of the image can include the timestamp t. i The speed at which the pixels corresponding to moving objects in the image move.

[0082] However, in these implementations, the motion data acquired by the electronic device has a high latency with the image to be optimized. Using this high-latency motion data to perform deblurring on the image to be optimized results in poor performance.

[0083] The above implementation method will be illustrated below with reference to Figures 3 and 4.

[0084] For example, Figure 3 is a schematic diagram of an image and data used to deblur the image according to an embodiment of this application.

[0085] As shown in Figure 3, image 300 is the image to be optimized. Image 300 includes a moving object 301, and the region where the moving object 301 is located in image 300 is region 302.

[0086] Coordinate graph 310 represents the region in the image where the moving object is located, corresponding to the pixel velocity used for deblurring image 300. This region can also be called the region of interest (ROI). In coordinate graph 310, coordinate axis 311 corresponds to the x-axis coordinate of image 300, and coordinate axis 312 corresponds to the y-axis coordinate of image 300. Region 313 in coordinate graph 310 with a value of 1 is the ROI. That is, when the electronic device optimizes image 300 using the pixel velocity corresponding to the ROI represented by coordinate graph 310, it optimizes region 303 in image 300, which is the same as region 313. Image optimization can be understood as deblurring the image.

[0087] The pixel speeds used to deblur image 300 can be shown in coordinate graphs 320 and 330.

[0088] Coordinate graph 320 can be represented as the pixel velocity of each pixel in the image along the x-axis, corresponding to the pixel velocity used for deblurring image 300. In coordinate graph 320, the denser the lines, the greater the pixel velocity in the x-axis direction. The coordinate axes in coordinate graph 320 can be found in coordinate graph 310, and will not be repeated here.

[0089] Coordinate graph 330 can be represented as the pixel velocity of each pixel in the image along the y-axis, corresponding to the pixel velocity used for deblurring image 300. In coordinate graph 330, the denser the lines, the greater the pixel velocity along the y-axis. The coordinate axes in coordinate graph 330 can be found in coordinate graph 310, and will not be repeated here.

[0090] As shown in Figure 3, in image 300, the region where the moving object 301 is located is region 303. In the image corresponding to the pixel speed used for deblurring image 300, represented by coordinate graph 310, the moving object 301 has moved from region 303 to region 313. That is to say, there is a large time delay between image 300 and the image corresponding to the pixel speed used for deblurring image 300. Therefore, when the electronic device uses the pixel speeds represented by coordinate graphs 320 and 330 to deblur image 300, it optimizes region 303 in image 300, rather than the moving object 301 in image 300, resulting in poor optimization effect of image 300, and even making the sharpness of the optimized image 300 worse than that of the unoptimized image 300.

[0091] The effect of deblurring using the pixel speed in Figure 3 can be seen in Figure 4. For example, Figure 4 is a schematic diagram of an image deblurring effect provided by an embodiment of this application.

[0092] As shown in Figure 4, image 401 is the image to be optimized, and image 402 is the image obtained by the electronic device after deblurring image 401 based on the pixel speeds represented by coordinate graphs 320 and 330 in Figure 3. It can be seen that the improvement in blurriness of image 402 is not significant compared to image 401.

[0093] In view of this, embodiments of this application provide an image processing method in which an electronic device acquires motion data with a smaller time delay between itself and the image to be optimized, and then performs deblurring processing on the image to be optimized. This improves the deblurring effect of the image to be optimized.

[0094] The following description, in conjunction with Figures 5 and 6, illustrates how the electronic device described above uses motion data with a smaller time delay between itself and the image to be optimized to perform deblurring processing on the image to be optimized.

[0095] For example, Figure 5 is another image and a data diagram for deblurring the image provided in an embodiment of this application.

[0096] As shown in Figure 5, image 500 is the image to be optimized. Image 500 includes a moving object 501, and the region where the moving object 501 is located in image 500 is region 502.

[0097] Coordinate graph 510 represents the region of moving objects in the image corresponding to the pixel velocity used for deblurring image 500. In coordinate graph 510, coordinate axis 511 corresponds to the x-axis coordinate of image 500, and coordinate axis 512 corresponds to the y-axis coordinate of image 500. Region 513 with a value of 1 in coordinate graph 510 is the Region of Interest (ROI). That is, when the electronic device optimizes image 500 using the pixel velocity corresponding to the ROI represented by coordinate graph 510, it optimizes region 502 in image 500, which is the same as region 513.

[0098] The pixel speeds used to deblur image 500 can be shown in coordinate graphs 520 and 530.

[0099] Coordinate graph 520 can be represented as the pixel velocity of each pixel in the image along the x-axis, corresponding to the pixel velocity used for deblurring image 500. In coordinate graph 520, the denser the lines, the greater the pixel velocity in the x-axis direction. The coordinate axes in coordinate graph 520 can be found in coordinate graph 510.

[0100] Coordinate graph 530 can be represented as the pixel velocity of each pixel in the image along the y-axis, corresponding to the pixel velocity used for deblurring image 500. In coordinate graph 530, the denser the lines, the greater the pixel velocity along the y-axis. The coordinate axes in coordinate graph 530 can be found in coordinate graph 510.

[0101] As shown in Figure 5, in image 500, the region where the moving object 501 is located is region 502. In the image corresponding to the pixel speed used for deblurring image 500, represented by coordinate graph 510, the moving object 501 is located in region 513. As shown in Figure 5, regions 502 and 513 are basically aligned in both the x-axis and y-axis directions. That is to say, the time delay between image 500 and the image corresponding to the pixel speed used for deblurring image 500 is small. Therefore, when the electronic device uses the pixel speeds represented by coordinate graphs 520 and 530 to deblurr image 500, it optimizes region 502 in image 500 that exactly needs optimization. This improves the deblurring effect.

[0102] The effect of deblurring using the pixel speed in Figure 5 can be seen in Figure 6. For example, Figure 6 is a schematic diagram of another image deblurring effect provided by an embodiment of this application.

[0103] As shown in Figure 6, image 601 is the image to be optimized, and image 602 is the image obtained by the electronic device after deblurring image 601 based on the pixel speeds represented by coordinate graphs 520 and 530 in Figure 5. It can be seen that the blurring in image 602 is significantly improved compared to image 601.

[0104] For example, the electronic device in the embodiments of this application may be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), augmented reality (AR) or virtual reality (VR) device, or other electronic device with display function. The embodiments of this application do not impose any special restrictions on the specific form of the electronic device.

[0105] The technical solutions provided in this application will be further described below using a mobile phone as an example of an electronic device. It should be understood that this application does not impose any limitations on the product form of the electronic device.

[0106] For example, Figure 7 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0107] Electronic devices may include a processor 710, an external memory interface 720, an internal memory 721, a universal serial bus (USB) interface 730, a charging management module 740, a power management module 741, a battery 742, a sensor module 750, buttons 760, a camera 761, and a display screen 762, etc.

[0108] The sensor module 750 may include a gyroscope sensor 750A, etc.

[0109] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0110] The processor 710 may include one or more processing units. These processing units may be independent devices or integrated within one or more processors. The processor 710 may also include memory for storing instructions and data.

[0111] Electronic devices utilize a GPU, a display screen 762, and an application processor to achieve display functionality. The GPU is a microprocessor for image processing, connecting the display screen 762 and the application processor. The GPU performs mathematical and geometric calculations and is used for graphics rendering.

[0112] The display screen 762 is used to display images, videos, etc. The display screen 762 includes a display panel. In some embodiments, the electronic device may include one or N display screens 762, where N is a positive integer greater than 1.

[0113] Electronic devices can achieve shooting functions through ISP, camera 761, video codec, GPU, display 762 and application processor.

[0114] Camera 761 is used to capture still images or videos. An object is projected onto a photosensitive element by an optical image generated through a lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device may include one or N cameras 761, where N is a positive integer greater than 1.

[0115] The ISP (Image Signal Processor) is used to process data fed back from the camera 761. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 761.

[0116] The external memory interface 720 can be used to connect external memory cards, such as Micro SD cards, to expand the storage capacity of electronic devices. The external memory card communicates with the processor 710 through the external memory interface 720 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0117] Internal memory 721 can be used to store executable program code, including instructions. Internal memory 721 may include a program storage area and a data storage area.

[0118] The gyroscope sensor 750A can be used to acquire gyroscope information from an electronic device, which is used to determine the degree of jitter in the electronic device. The gyroscope information may include the angular velocity of the electronic device rotating around the x-axis, the angular velocity of the electronic device rotating around the y-axis, and / or the angular velocity of the electronic device rotating around the z-axis. The x-axis, y-axis, and z-axis can be set according to the actual scenario. For example, the x-axis is parallel to the short side of the electronic device, the y-axis is parallel to the long side of the electronic device, and the z-axis is perpendicular to the screen of the electronic device. In real-world scenarios, when a user takes a photo using the electronic device, the device is prone to jitter, resulting in blurry images. Therefore, the electronic device in this embodiment can acquire gyroscope information during photo taking using the gyroscope sensor 750A. Then, the electronic device can perform deblurring processing on the resulting image based on the gyroscope information.

[0119] Buttons 760 include a power button, volume buttons, etc. Buttons 760 can be mechanical buttons or touch-sensitive buttons. Electronic devices can receive button input and generate key signal inputs related to user settings and function control.

[0120] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a limitation on the structure of the electronic device. In other embodiments of this application, the electronic device may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0121] The software system of an electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This invention embodiment uses a layered architecture of Android. TM Taking a system as an example, the software structure of an electronic device is illustrated by way of example. For example, Figure 8 is a schematic diagram of the software structure of an electronic device provided in an embodiment of this application.

[0122] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, Android... TM The system is divided into five layers, from top to bottom: application layer, application framework (FWK) layer, hardware abstraction layer (HAL) layer, kernel layer, and hardware layer.

[0123] The application layer can include a series of application packages.

[0124] As shown in Figure 8, the application package can include applications such as video applications, gallery applications, and camera applications.

[0125] A gallery application can be understood as an application used to store multimedia files such as images and videos. For example, a gallery application can also be a file manager, photo editing application, etc. This application does not specifically limit the form of the gallery application.

[0126] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes a set of predefined functions.

[0127] In some embodiments, the application framework layer may include camera access interfaces such as camera services. These camera access interfaces provide application programming interfaces and programming frameworks for camera applications.

[0128] The Hardware Abstraction Layer (HAL) is an interface layer located between the application framework layer and the kernel layer, providing a virtual hardware platform for the operating system.

[0129] In this embodiment of the application, the hardware abstraction layer may include a data acquisition unit, a motion detection manager, a photo request processing unit, a foreground photo capture unit, and a background photo capture unit, etc.

[0130] The data acquisition unit is used to acquire raw image data and corresponding motion data in real time after the camera application is running in the foreground, such as when the user triggers the camera application icon 101 on the mobile phone desktop 100 shown in Figure 1. The motion detection manager is used to store the motion data corresponding to a preset number of raw image data with the latest timestamp acquired by the data acquisition unit in real time. The photo capture request processing unit is used to acquire reference frame image data after receiving a photo capture request, such as when the user triggers the capture button 112 in Figure 1, and to acquire motion data from the motion detection manager. The foreground photo capture unit is used to generate a quick thumbnail (also called a second thumbnail) based on the reference frame image data. The background photo capture unit is used to match motion data with the reference frame image data and to perform deblurring processing based on the motion data matched with the reference frame image data.

[0131] The kernel layer is the layer between the hardware layer and the hardware abstraction layer. The kernel layer includes drivers for various hardware components. These may include camera drivers, webcam drivers, digital signal processor (DSP) drivers, and image processor (IPF) drivers. Specifically, the camera driver drives the image sensors of one or more cameras in the camera module to acquire images and drives the IPF to preprocess the images. The DSP driver drives the DSP to process images. The IPF driver drives the IPF to process images.

[0132] The hardware layer can include camera modules, image signal processors (ISPs), digital signal processors (DSPs), graphics processing units (GPUs), etc. A camera module can include image sensors from multiple cameras.

[0133] The ISP can include an Image Frontend (IFE) module and an Image Processing Engine (IPE) module. The IFE module can perform one or more front-end processing tasks on the image data output by the sensor in the camera module, such as color correction, downsampling, depigmentation, and statistical 3A data for the preview stream or video stream. The IPE module can perform one or more back-end processing tasks on the image data, such as hardware noise reduction, image cropping, software noise reduction, color processing, and detail enhancement.

[0134] The workflow of the electronic device software and hardware is illustrated below with reference to Figures 7-9. Figure 9 is a schematic diagram of the framework of an image processing method provided in an embodiment of this application.

[0135] After the user triggers the camera application in the application layer, the camera application calls the interface of the application framework layer to start the camera application, which in turn starts the camera driver. Then, the camera driver starts driving the camera 761 to capture raw image data. The electronic device can display the shooting preview interface 110 in Figure 1. After the camera 761 starts capturing raw image data, the data acquisition unit in the hardware abstraction layer begins to calculate the motion data corresponding to the raw image data based on the raw image data.

[0136] Then, the data acquisition unit sends the acquired motion data to the motion detection manager for storage. The motion data stored in the motion detection manager can be updated in real time as time changes. The motion detection manager can store motion data corresponding to a preset number of raw image data recently acquired by the data acquisition unit. When the user triggers the shooting button 112 in the shooting preview interface 110 in Figure 1, the camera application calls the interface of the application framework layer to send a photo-taking request to the hardware abstraction layer. The photo-taking request carries a timestamp indicating when the shooting button was triggered.

[0137] The photo capture request processing unit in the hardware abstraction layer obtains first image data from the raw image data captured by the camera 761 based on the photo capture request. The first image data may include multiple frames of raw image data. The photo capture request processing unit can determine one frame of raw image data from the first image data as the reference frame image data.

[0138] This application does not specifically limit which frame of the first image data the reference frame image data is.

[0139] For example, the reference frame image data can be the original image data of a frame whose timestamp precedes the timestamp carried in the photo-taking request. The preset value can be set according to the actual scenario, and this embodiment does not specifically limit the preset value. For example, when the preset value is 3, the reference frame image data is the original image data of the 3rd frame whose timestamp precedes the timestamp carried in the photo-taking request. The reference frame image data can also be the original image data of a frame whose timestamp follows the timestamp carried in the photo-taking request. For example, when the preset value is 2, the reference frame image data is the original image data of the 2nd frame whose timestamp follows the timestamp carried in the photo-taking request. The reference frame image data can also be the original image data corresponding to the timestamp carried in the photo-taking request.

[0140] As described above, the step of the photo request processing unit acquiring the reference frame image data occurs after the user triggers the shooting button 112 in the shooting preview interface 110 of Figure 1. After acquiring the reference frame image data, the photo request processing unit can send the reference frame image data to the foreground shooting unit before the mobile phone displays the shooting preview interface 120 of Figure 1. The foreground shooting unit can quickly generate a second thumbnail based on the reference frame image data to reduce the time delay between the user clicking the shooting button and the user seeing the thumbnail generated by the mobile phone, thereby improving the user experience.

[0141] As shown in Figure 1, the shooting preview interface 120 includes a thumbnail display area 121, which displays a second thumbnail 122 quickly generated by the foreground shooting unit based on the reference frame image data. The remaining content of the shooting preview interface 120 can be found in the shooting preview interface 110, and will not be described in detail here.

[0142] Upon receiving a photo-taking request, before the phone displays the shooting preview interface 120 as shown in Figure 1, the photo-taking request processing unit can also obtain second motion data corresponding to multiple frames of third image data from the motion detection manager. Then, the photo-taking request processing unit can send the reference frame image data and the second motion data corresponding to the multiple frames of third image data to the background photo-taking unit.

[0143] The background photography unit can determine whether there is motion data in the second motion data corresponding to multiple frames of third image data that can be used to perform deblurring processing on the first image data. The process of the background photography unit determining whether there is motion data in each second motion data that can be used to perform deblurring processing on the first image data can be completed before the phone displays the shooting preview interface 120 in Figure 1, after the phone displays the shooting preview interface 120 in Figure 1, or at the moment the phone displays the shooting preview interface 120 in Figure 1. This application embodiment does not limit this.

[0144] The background image capture unit determines whether there is motion data in each of the second motion data sets that can be used to perform deblurring on the first image data. This can include: the background image capture unit determining whether there is any third image data in the multi-frame third image data sets where the absolute value of the difference between the timestamp and the reference timestamp is less than or equal to a preset threshold. The reference timestamp is the timestamp of the reference frame image data.

[0145] If there is a third image data where the absolute value of the difference between the timestamp and the reference timestamp is less than or equal to a preset threshold, the background image capture unit can determine the second motion data corresponding to the third image data as motion data that can be used to perform deblurring on the first image data; if there is no third image data where the absolute value of the difference between the timestamp and the reference timestamp is less than or equal to a preset threshold, the background image capture unit can determine that among the second motion data, there is no motion data that can be used to perform deblurring on the first image data.

[0146] The preset threshold can be set according to the actual scenario, and this application embodiment does not specifically limit the preset threshold. For example, the preset threshold can be 33 milliseconds.

[0147] It is understandable that the timestamp of any motion data is the same as the timestamp of the original image data corresponding to that motion data.

[0148] For example, if the background image capture unit determines that there is motion data in each of the second motion data that can be used to perform deblurring on the first image data, the background image capture unit can perform deblurring on the first image data. Then, the background image capture unit can output the result image. Afterwards, the second thumbnail output by the foreground image capture unit can be replaced by the thumbnail corresponding to the result image. For example, the second thumbnail 122 in the thumbnail display area 121 in Figure 1 can be replaced by the thumbnail corresponding to the result image.

[0149] If the background image capture unit determines that there is no motion data among the second motion data that can be used to perform deblurring on the first image data, the background image capture unit can obtain the third motion data corresponding to multiple frames of fourth image data from the motion detection manager. Then, the background image capture unit can determine whether there is any motion data among the third motion data that can be used to perform deblurring on the first image data.

[0150] It is understandable that the photo-taking request processing unit acquires the second motion data corresponding to the multi-frame third image data after receiving a trigger operation from the user on the shooting button 112 in the shooting preview interface 110 shown in Figure 1. The background photo-taking unit, however, acquires the third motion data corresponding to the multi-frame fourth image data only after determining that there is no motion data among the second motion data that can be used to perform deblurring processing on the first image data. In other words, the time when the photo-taking request processing unit acquires the second motion data corresponding to the multi-frame third image data from the motion detection manager is different from the time when the background photo-taking unit acquires the third motion data corresponding to the multi-frame fourth image data from the motion detection manager. Since the motion data stored in the motion detection manager is updated in real time, the third motion data corresponding to the multi-frame fourth image data acquired by the background photo-taking unit may be partially the same as or completely different from the second motion data corresponding to the multi-frame third image data acquired by the photo-taking request processing unit.

[0151] If the background image capture unit determines that there is motion data among the third motion data that can be used to perform deblurring on the first image data, the background image capture unit can perform deblurring on the first image data. Then, the background image capture unit can output the result image. Afterwards, the second thumbnail output by the foreground image capture unit can be replaced by the thumbnail corresponding to the result image.

[0152] If the background image capture unit determines that there is no motion data among the third motion data that can be used to perform deblurring on the first image data, the background image capture unit may not perform deblurring on the first image data. The background image capture unit directly generates the result image based on the first image data. Subsequently, the second thumbnail output by the foreground image capture unit can be replaced by the thumbnail corresponding to the result image.

[0153] The image processing method provided in the embodiments of this application will now be described with reference to the accompanying drawings. The image processing method in the embodiments of this application can be applied to an electronic device that combines the hardware structure shown in FIG7 and the software structure shown in FIG8.

[0154] For example, Figure 10 is a schematic flowchart of an image processing method provided in an embodiment of this application.

[0155] As shown in Figure 10, after the user triggers the shooting button 112 in the shooting preview interface 110 of Figure 1, when the shooting request processing unit receives the shooting request, it acquires the second motion data corresponding to multiple frames of third image data, and acquires the first image data including reference frame image data. The second motion data corresponding to multiple frames of third image data can be understood as any motion data stored in the motion detection manager at the moment the shooting request processing unit acquires motion data from the motion detection manager. The third image data can be understood as the original image data corresponding to any motion data stored in the motion detection manager at the moment the shooting request processing unit acquires motion data from the motion detection manager.

[0156] Different motion data in the motion detection manager can be stored in different data storage units. The motion detection manager can include a preset number of data storage units. The preset number can be set according to the actual scenario, and this embodiment does not specifically limit it. For example, the preset number can be 10, that is, the motion detection manager can include 1 to 10 data storage units.

[0157] The photo request processing unit can send the acquired reference frame image data to the front-end photo capture unit so that the front-end photo capture unit can quickly generate a second thumbnail.

[0158] The photo-taking request processing unit can also send the acquired reference frame image data and the second motion data corresponding to the multi-frame third image data to the background photo-taking unit.

[0159] As shown in Figure 10, the background image capture unit includes at least an input plugin, a processing module, and an output plugin. The input plugin of the background image capture unit receives reference frame image data and the second motion data corresponding to each frame of third image data, and forwards the reference frame image data and the second motion data corresponding to each frame of third image data to the processing module. After receiving the reference frame image data and the second motion data corresponding to each frame of third image data, if there is no image data in each frame of third image data where the absolute value of the difference between the timestamp and the reference timestamp is less than or equal to a preset threshold, the processing module acquires the third motion data corresponding to multiple frames of fourth image data. When the processing module receives the next frame of original image data after the reference frame image data sent by the image capture request processing unit, the processing module can determine whether there is image data in each frame of fourth image data where the absolute value of the difference between the timestamp and the reference timestamp is less than or equal to a preset threshold.

[0160] The reference timestamp is the timestamp of the reference frame image data. The third motion data can be understood as any motion data stored in the motion detection manager at the moment the processing module retrieves motion data from the motion detection manager, provided that the absolute value of the difference between the timestamp and the reference timestamp in any frame of the third image data is less than or equal to a preset threshold. The fourth image data can be understood as the original image data corresponding to any motion data stored in the motion detection manager at the moment the photo-taking request processing unit retrieves motion data from the motion detection manager, provided that the absolute value of the difference between the timestamp and the reference timestamp in any frame of the third image data is less than or equal to a preset threshold.

[0161] In each frame of the third image data, if there is original image data where the absolute value of the difference between the timestamp and the reference timestamp is less than or equal to a preset threshold, the processing module performs deblurring processing on the first image data received by the processing module, including the reference frame image data, based on the motion data corresponding to the third image data where the absolute value of the difference between the timestamp and the reference timestamp is less than or equal to the preset threshold. Then, the processing module outputs a thumbnail in YUV format and sends the YUV format thumbnail to the output plugin of the background imaging unit. Afterwards, the output plugin of the background imaging unit outputs the result image based on the YUV format thumbnail.

[0162] However, on the one hand, after the user triggers the camera application icon 101 in Figure 1, the camera begins to capture raw image data in real time. As described in the embodiment corresponding to Figure 9, the motion data stored in the motion detection manager is calculated by the data acquisition unit based on the raw image data. This calculation process takes time; therefore, the motion data stored in the motion detection manager lags behind the raw image data captured by the camera. This results in no raw image data in the multi-frame third image data where the absolute value of the difference between the timestamp and the reference timestamp is less than or equal to a preset threshold. In other words, the time delay between the multi-frame third image data and the reference frame image data is relatively large.

[0163] For example, the camera has already captured raw image data with timestamps t1, t2, t3, t4, t5, t6, t7, t8, t9...t20. However, when the photo request processing unit obtains motion data from the motion detection manager, the data acquisition unit may only calculate the motion data corresponding to the raw image data with timestamp t1 up to the raw image data with timestamp t10. In other words, when the photo request processing unit obtains motion data from the motion detection manager, the motion detection manager only stores the motion data corresponding to the raw image data with timestamp t1 up to the raw image data with timestamp t10. Therefore, when the first image data includes raw image data with timestamps t18, t19, and t20, regardless of which frame in the first image data the photo request processing unit determines as the reference frame image data, the time delay between the raw image data with timestamps t1 to t10 and the reference frame image data is relatively large. In other words, the timestamp of the third image data is much earlier than the reference timestamp.

[0164] On the other hand, due to the numerous processing steps in the background image capture unit, the time delay between the moment the image capture request processing unit acquires the second motion data and the moment the background image capture unit acquires the third motion data is significant. This means the motion detection manager may have been updated multiple times. Consequently, the time delay between the timestamp of the fourth image data and the reference timestamp is substantial. In other words, the timestamp of the fourth image data is much later than the reference timestamp. Furthermore, this reduces the probability that the background image capture unit will identify motion data that matches the reference frame image data. Motion data matching the reference frame image data can be understood as motion data corresponding to the original image data whose absolute value of the difference between its timestamp and the reference timestamp is less than or equal to a preset threshold.

[0165] The motion data acquired by the photo-taking request processing unit and the motion data acquired by the background photo-taking unit in this embodiment of the present application will be described below with reference to Figure 11. For example, Figure 11 is a schematic diagram of data storage in a motion detection manager provided in an embodiment of the present application.

[0166] As shown in Figure 11, the motion detection manager includes ten data storage units (1-10), each storing motion data corresponding to one frame of original image data. When the photo capture request processing unit retrieves the second motion data from the motion detection manager, the motion detection manager stores motion data corresponding to the original image data with timestamps t1, t2, t3, t4, t5, t6, t7, t8, t9, and t10. That is, the timestamps of each frame of the third image data are t1, t2, t3, t4, t5, t6, t7, t8, t9, and t10, respectively. For ease of description, the motion data corresponding to the original image data with timestamp t1 is called frame 1, the motion data corresponding to the original image data with timestamp t2 is called frame 2, and so on, until the motion data corresponding to the original image data with timestamp t10 is called frame 10.

[0167] The motion data stored in the motion detection manager is updated over time. As mentioned above, there is a significant time delay between the moment the photo capture request processing unit acquires the second motion data and the moment the background photo capture unit acquires the third motion data. Therefore, the time delay between the timestamp of the fourth image data and the reference timestamp is also likely to be significant. For example, when the background photo capture unit acquires the third motion data from the motion detection manager, the motion detection manager stores motion data corresponding to the original image data with timestamps t35, t36, t37, t38, t39, t40, t41, t42, t43, and t44. That is, the timestamps of the fourth image data for each frame are t35, t36, t37, t38, t39, t40, t41, t42, t43, and t44, respectively. Therefore, when the background photo capture unit matches motion data for the reference frame image data, it will miss the motion data corresponding to the original image data from timestamp t11 to timestamp t34.

[0168] For ease of description, the motion data corresponding to the original image data with timestamp t35 is called frame 35, the motion data corresponding to the original image data with timestamp t36 is called frame 36, and so on, the motion data corresponding to the original image data with timestamp t44 is called frame 44.

[0169] In some embodiments, the mobile phone can also acquire first motion data corresponding to multiple frames of second image data through the foreground camera unit. Then, the first motion data corresponding to the multiple frames of second image data can be used to perform deblurring processing on the first image data. In this way, the background camera unit can use the first motion data to perform deblurring processing on the first image data. Since the timestamp of the second image data is closer to the reference timestamp, the background camera unit achieves better results in using the first motion data to perform deblurring processing on the first image data.

[0170] The first motion data corresponding to the multiple frames of second image data can be understood as any motion data stored in the motion detection manager at the moment when the foreground imaging unit acquires motion data from the motion detection manager. The second image data can be understood as the original image data corresponding to any motion data stored in the motion detection manager at the moment when the foreground imaging unit acquires motion data from the motion detection manager.

[0171] The technical solutions provided in the embodiments of this application will now be described in detail with reference to Figure 12.

[0172] For example, Figure 12 is a schematic flowchart of another image processing method provided in an embodiment of this application. The method may include steps S1201-S1210.

[0173] S1201, The photo request processing unit receives a photo request.

[0174] As shown in Figure 1, in response to a user's trigger operation on the camera application icon 101 on the phone's desktop 100, the phone can display a shooting preview interface 110. In response to a user's trigger operation on the shooting preview interface 110's shooting button 112, the camera application can send a shooting request to the shooting request processing unit. The shooting request processing unit receives the shooting request.

[0175] S1202, The photo request processing unit obtains the second motion data.

[0176] After receiving a photo request, the photo request processing unit can obtain the second motion data corresponding to multiple frames of third image data from the motion detection manager.

[0177] In one possible implementation, the second motion data may include the pixel velocity corresponding to the original image data of a single frame.

[0178] In another possible implementation, the second motion data may include the pixel velocities corresponding to each of the multiple frames of original image data.

[0179] In another possible implementation, the second motion data may include the pixel velocity corresponding to each of the multiple frames of original image data, the object distance (also known as the object distance) corresponding to the original image data, and / or the gyroscope information corresponding to the original image data.

[0180] Wherein, the object distance corresponding to any raw image data is the distance between the mobile phone and the object being photographed at the moment the camera captures the raw image data. The gyroscope information corresponding to any raw image data may include the angular velocity of the mobile phone rotating around the x-axis, the angular velocity of the mobile phone rotating around the y-axis, and / or the angular velocity of the mobile phone rotating around the z-axis.

[0181] S1203, The photo request processing unit obtains reference frame image data.

[0182] Upon receiving a photo-taking request, the photo-taking request processing unit can obtain first image data from a tiny stream, which includes reference frame image data. For example, the tiny stream can be understood as multiple frames of low-resolution image data captured in real-time by the camera (the low-resolution image data is the aforementioned original image data).

[0183] It should be understood that the execution order of the above steps S1202 and S1203 can be adjusted according to actual usage requirements, and they can also be executed in parallel. This application embodiment does not limit this.

[0184] S1204, The image processing unit sends the reference frame image data to the front-end image processing unit.

[0185] The photo request processing unit sends the reference frame image data to the front-end photo capture unit.

[0186] S1205, the image processing unit sends the reference frame image data and the second motion data to the background image processing unit.

[0187] The photo request processing unit sends the reference frame image data and each second motion data to the background photo unit.

[0188] It should be understood that the execution order of the above steps S1204 and S1205 can be adjusted according to actual usage requirements, and they can also be executed in parallel. This application embodiment does not limit this.

[0189] S1206, The front-end photo unit's fast thumbnail generation module obtains the first motion data.

[0190] As shown in Figure 12, the front-end image capture unit includes at least an input plugin, a fast thumbnail generation module, and an output plugin. The input plugin of the front-end image capture unit receives reference frame image data and forwards it to the fast thumbnail generation module. After receiving the reference frame image data, the fast thumbnail generation module can acquire the first motion data corresponding to multiple frames of second image data. The fast thumbnail generation module can also send the acquired first motion data corresponding to multiple frames of second image data to the processing module of the back-end image capture unit.

[0191] To quickly output the second thumbnail, the foreground capturing unit performs fewer processing steps. Therefore, the foreground capturing unit acquires the first motion data corresponding to the multiple frames of the second image data earlier than the background capturing unit acquires the third motion data corresponding to the multiple frames of the fourth image data. In other words, the foreground capturing unit acquires the first motion data corresponding to the multiple frames of the second image data between the time the capturing request processing unit acquires the second motion data corresponding to the multiple frames of the third image data and the time the background capturing unit acquires the third motion data corresponding to the multiple frames of the fourth image data. Therefore, the timestamp of the second image data is closer to the reference timestamp than the third and fourth image data. Consequently, the background capturing unit achieves better deblurring results when using the first motion data to process the first image data. Furthermore, the first motion data acquired by the foreground capturing unit, to some extent, compensates for the motion data missed by the background capturing unit, thereby increasing the probability that the background capturing unit can determine motion data that matches the reference frame image data.

[0192] The motion data acquired by the photo-taking request processing unit, the motion data acquired by the background photo-taking unit, and the motion data acquired by the foreground photo-taking unit in this embodiment of the present application will be described below with reference to Figure 13. For example, Figure 13 is a schematic diagram of another motion detection manager storing data provided in an embodiment of the present application.

[0193] As shown in Figure 13, the photo-taking request processing unit acquires the second motion data corresponding to multiple frames of third image data with timestamps t1, t2, t3, t4, t5, t6, t7, t8, t9, and t10 in sequence. As the motion data is stored, the motion data in the motion detection manager changes. When the foreground photo-taking unit acquires motion data, it acquires the first motion data corresponding to multiple frames of second image data with timestamps t19, t20, t21, t22, t23, t24, t25, t26, t27, and t28 in sequence. As the motion data is stored, the motion data in the motion detection manager changes. When the background photo-taking unit acquires motion data, it acquires the third motion data corresponding to the fourth image data with timestamps t35, t36, t37, t39, t40, t41, t42, t43, t44, and t45 in sequence.

[0194] When the reference timestamp is t25, and the absolute value of the difference between the timestamps from timestamps t15 to t35 and the reference timestamp is less than or equal to a preset threshold, the background image capture unit can determine any one of the motion data corresponding to the original image data at timestamp t19, the motion data corresponding to the original image data at timestamp t28, and the motion data corresponding to the original image data at timestamp t35 as the motion data matching the reference frame image data. Compared to the embodiment corresponding to Figure 11, since the foreground image capture unit in the embodiment corresponding to Figure 11 did not acquire the first motion data, in the embodiment corresponding to Figure 11, the background image capture unit determines the motion data corresponding to the original image data at timestamp t35 as the motion data matching the reference frame image data. Compared to the delay between timestamps t25 and t35, the delay between any one of the timestamps from timestamps t19 to t28 and timestamp t25 is smaller. Therefore, compared to using the third motion data corresponding to the fourth image data obtained by the background photography unit for deblurring, the background photography unit achieves better results by using the first motion data corresponding to the second image data obtained by the foreground photography unit for deblurring.

[0195] For ease of description, the motion data corresponding to the original image data with timestamp t19 is called frame 19, the motion data corresponding to the original image data with timestamp t20 is called frame 20, and so on, the motion data corresponding to the original image data with timestamp t28 is called frame 28.

[0196] S1207, The quick thumbnail generation module generates the first thumbnail.

[0197] In a possible implementation, after receiving the reference frame image data, the fast thumbnail generation module can also generate a first thumbnail based on the reference frame image data. The first thumbnail can be a YUV format image. Then, the fast thumbnail generation module can send the first thumbnail to the output plugin of the front-end imaging unit. The output plugin of the front-end imaging unit can process the first thumbnail, such as converting its format, to obtain a second thumbnail. The second thumbnail can be a JPEG format image. This second thumbnail can be displayed in the thumbnail display area 121 as shown in Figure 1.

[0198] S1208, the processing module of the background photo-taking unit determines whether target image data exists.

[0199] As shown in Figure 12, the input plugin of the background image capture unit receives the second motion data corresponding to the reference frame image data and the multi-frame third image data, and forwards the second motion data corresponding to the reference frame image data and the multi-frame third image data to the processing module. The processing module receives the second motion data corresponding to the reference frame image data and the multi-frame third image data. The processing module can also receive the first motion data corresponding to the multi-frame second image data sent by the fast thumbnail generation module. Then, the processing module determines whether target image data exists in the multi-frame third image data and the multi-frame second image data.

[0200] In one possible implementation, the processing module sequentially iterates through the timestamps of each frame of third image data and each frame of second image data in ascending order of timestamps. Any original image data whose absolute value of the difference between its timestamp and a reference timestamp is less than or equal to a preset threshold is identified as the target image data. The motion data corresponding to the target image data is used for deblurring processing by the subsequent background image capture unit.

[0201] For example, the timestamps of the third image data in each frame are t1, t2, t3, t4, t5, t6, t7, t8, t9, and t10, respectively, and the timestamps of the second image data in each frame are t19, t20, t21, t22, t23, t24, t25, t26, t27, and t28, respectively. If the reference timestamp is t25, and the absolute value of the difference between the timestamps from t15 to t35 and the reference timestamp is less than or equal to a preset threshold, the background imaging unit will determine any one of the original image data with timestamps t19, t20, t21, t22, t23, t24, t25, t26, t27, and t28 as the target image data.

[0202] In another possible implementation, the processing module determines the target motion data as the original image data whose timestamp is closest to the reference timestamp in the third image data of each frame and the second image data of each frame.

[0203] For example, the timestamps of the third image data in each frame are t1, t2, t3, t4, t5, t6, t7, t8, t9, and t10, respectively, and the timestamps of the second image data in each frame are t19, t20, t21, t22, t23, t24, t25, t26, t27, and t28, respectively. With a reference timestamp of t30, the background imaging unit determines the motion data corresponding to the original image data with timestamp t28 as the target image data.

[0204] In another possible implementation, in order to make the timestamp corresponding to the target image data closer to the reference timestamp, thereby improving the image deblurring effect, the processing module can sequentially traverse the timestamps of each frame of the third image data and each frame of the second image data in the order of timestamps from early to late, and determine the original image data whose absolute value of the difference between the timestamp and the reference timestamp is less than or equal to a preset threshold and whose timestamp is closest to the reference timestamp as the target image data.

[0205] For example, the timestamps of the third image data in each frame are t1, t2, t3, t4, t5, t6, t7, t8, t9, and t10, respectively, and the timestamps of the second image data in each frame are t19, t20, t21, t22, t23, t24, t25, t26, t27, and t28, respectively. If the reference timestamp is t18, and the absolute value of the difference between the timestamps from t8 to t28 and the reference timestamp is less than or equal to a preset threshold, the background imaging unit determines the second image data with timestamp t19 as the target image data.

[0206] If the processing module determines that target image data exists in the multi-frame third image data and multi-frame second image data, the background photo capture unit executes step S1209.

[0207] S1209. The processing module performs deblurring and outputs an intermediate image.

[0208] In a possible implementation, the first image data acquired by the photo-taking request processing unit includes multiple frames of raw image data. The photo-taking request processing unit determines one of these frames of raw image data as the reference frame image data. The photo-taking request processing unit can sequentially send each frame of raw image data in the first image data to the background photo-taking unit according to the timestamp order of each frame of raw image data in the first image data.

[0209] Then, the processing module can first perform the deblurring step, and then perform the fusion step. For example, the processing module can first perform deblurring on each frame of the original image data in the first image data based on the motion data corresponding to the target image data, to obtain multiple frames of deblurred image data. Then, the processing module performs fusion processing on the multiple frames of deblurred image data to obtain a first fusion result. After that, the processing module outputs an intermediate image based on the first fusion result. The intermediate image can be a YUV format image.

[0210] Alternatively, the processing module can perform the fusion step first, followed by the deblurring step. The processing module can also first fuse the original image data of each frame in the first image data to obtain a second fusion result. Then, the processing module performs deblurring on the second fusion result based on the motion data corresponding to the target image data to obtain a deblurred result. Finally, the processing module outputs an intermediate image based on the deblurred result.

[0211] Alternatively, the processing module can execute the fusion step and the deblurring step simultaneously. This application embodiment does not specify the order in which the processing module performs the fusion step and the deblurring step.

[0212] The processing module can then send the intermediate image to the output plugin of the background image-taking unit. The output plugin of the background image-taking unit can process the intermediate image, such as converting its format, to obtain the result image. The result image can be a JPEG format image. Then, the thumbnail corresponding to this result image can replace the second thumbnail and be displayed in the thumbnail display area 121 as shown in Figure 1.

[0213] In this embodiment, between the moment the photo-taking request processing unit acquires the second motion data corresponding to multiple frames of third image data and the moment the background photo-taking unit acquires the third motion data corresponding to multiple frames of fourth image data, the mobile phone acquires the first motion data corresponding to multiple frames of second image data through the foreground photo-taking unit. Thus, the background photo-taking unit can determine the presence of target image data not only from the multiple frames of third image data but also from the multiple frames of second image data. This increases the probability that the background photo-taking unit determines the target image data, thereby increasing the probability that the background photo-taking unit obtains the motion data corresponding to the target image data, and thus improving the success rate of the deblurring process. Furthermore, the timestamp of the second image data is earlier than that of the fourth image data. Therefore, the timestamp of the target image data determined by the processing module from the multiple frames of second image data is closer to the timestamp of the reference frame image data. Therefore, the deblurring effect based on the motion data corresponding to the target image data determined from the multiple frames of second image data is better.

[0214] It is understood that in the embodiments of this application, the timestamp of any original image data is the same as the timestamp of the motion data corresponding to that original image data. Therefore, when comparing the timestamp of the original image data with the reference timestamp in the embodiments of this application, it can also be a comparison between the timestamp of the motion data corresponding to the original image data and the reference timestamp. The embodiments of this application do not limit this.

[0215] If the processing module determines that there is no target image data in the multi-frame third image data and multi-frame second image data, the background photography unit can also execute step S1210 after the above step S1208.

[0216] S1210, The processing module acquires the third motion data.

[0217] If the processing module determines that there is no target image data in the multi-frame third image data and multi-frame second image data, the processing module obtains the third motion data corresponding to the multi-frame fourth image data.

[0218] Then, the processing module can determine whether the target image data exists in the multi-frame fourth image data.

[0219] In a possible implementation, the processing module determines whether target image data exists in the multi-frame fourth image data, which may include:

[0220] The processing module determines whether the timestamp of the earliest fourth image data in the multi-frame fourth image data is later than the reference timestamp. If the timestamp of the earliest fourth image data is later than the reference timestamp, and the earliest fourth image data is not the target image data, the processing module will not further determine whether the target image data exists in the multi-frame fourth image data. In other words, for this photo-taking request, the processing module will not perform deblurring processing.

[0221] If the timestamp of the earliest fourth image data in the multi-frame fourth image data is earlier than the reference timestamp, the processing module determines whether the target image data exists in the multi-frame fourth image data.

[0222] Then, if target image data exists in the multi-frame fourth image data, the processing module can perform deblurring processing on the first image data based on the third motion data corresponding to the target image data in the third motion data corresponding to the multi-frame fourth image data.

[0223] If the target image data is absent in the multi-frame fourth image data, the processing module can acquire the fourth motion data corresponding to the multi-frame fifth image data. Here, the fourth motion data corresponding to the multi-frame fifth image data can be understood as any motion data stored in the motion detection manager at the moment the processing module determines that the target image data is absent from the multi-frame fourth image data. The fifth image data can be understood as the original image data corresponding to any motion data stored in the motion detection manager at the moment the processing module determines that the target image data is absent from the multi-frame fourth image data.

[0224] In other words, after determining that the target image data is absent in the multiple frames of third and second image data, each time the processing module receives a frame of raw image data, it checks whether the target image data exists in the raw image data corresponding to the previously acquired motion data. If it exists, the processing module stops acquiring motion data; if it does not exist, the processing module acquires motion data again. Until the processing module receives the last frame of raw image data from the first image data, if the target image data is still absent, the processing module does not perform deblurring processing for this photo capture request.

[0225] As one possible implementation, the first motion data corresponding to any frame of second image data stored in the motion detection manager includes: the pixel velocity of pixels in multiple frames of second image data.

[0226] It should be understood that when target image data is present in multiple frames of second image data, the first motion data corresponding to the target image data also includes the pixel velocities of the pixels in the multiple frames of second image data. Then, the processing module can calculate the pixel velocity corresponding to the target image data based on the pixel velocities of the pixels in the multiple frames of second image data within the first motion data corresponding to the target image data. Subsequently, the processing module performs deblurring processing on the first image data based on the pixel velocity corresponding to the target image data.

[0227] It is understandable that the pixel velocity corresponding to the target image data can be used as the pixel velocity of the pixels in the target image data. However, the pixel velocity corresponding to the target image data is a calculated value, and it may differ from the actual pixel velocity of the pixels in the target image data. The processing module uses the pixel velocity corresponding to the target image data when performing deblurring on the first image data.

[0228] Referring again to Figure 12, the mobile phone may include a data acquisition unit, which may include a camera hardware abstraction layer and a motion detection and perception engine. First, in response to the user triggering the camera application icon 101 in Figure 1, the phone launches the camera application, which in turn launches the camera driver. Then, the camera begins capturing raw image data in real time. The multiple frames of raw image data continuously captured by the camera can be referred to as a tiny stream.

[0229] Next, starting from when the camera captures the raw image data, the motion detection engine can calculate the pixel velocity corresponding to the raw image data in the tiny stream sequentially from earliest to latest timestamp. Then, the motion detection engine can send the pixel velocity corresponding to the raw image data to the motion detection manager for storage.

[0230] For example, the first motion data corresponding to any frame of second image data stored in the motion detection manager can be seen in Figure 12. As shown in Figure 12, the motion detection manager includes ten data storage units, numbered 1-10. The first data storage unit of the motion detection manager can store the pixel velocity of pixels in the second image data with timestamp t19. This first data storage unit can also store the pixel velocities of pixels in the second image data from timestamp t20 to timestamp t28.

[0231] Similarly, the second data storage unit of the motion detection manager can store the pixel velocity of pixels in the second image data with timestamp t20. This second data storage unit can also store the pixel velocity of pixels in the second image data with timestamp t19, and the pixel velocities of pixels in the second image data with timestamps t21 to t28. In other words, when the data storage unit stores the pixel velocities of pixels from multiple frames of original image data, the pixel velocities stored in each data storage unit are the same.

[0232] In a possible implementation, the processing module of the background image capture unit can calculate the first motion data corresponding to the target image data based on methods such as interpolation, moving average, local regression, time series analysis, machine learning, piecewise fitting, kernel smoothing, Bayesian methods, empirical formulas, and data-driven methods to determine the pixel velocity corresponding to the target image data. This application does not limit the implementation method for determining the pixel velocity corresponding to the target image data in its embodiments.

[0233] In one possible implementation, the processing module calculates the pixel velocity corresponding to the target image data based on the pixel velocities of pixels in multiple frames of second image data within the first motion data corresponding to the target image data. This includes: the processing module fitting the pixel velocities of pixels in the multiple frames of second image data within the first motion data corresponding to the target image data to obtain a fitting function; the fitting function reflects the correspondence between the pixel velocity and the timestamps of the image data. Then, the processing module inputs the timestamps of the target image data into the fitting function to obtain the pixel velocity corresponding to the target image data.

[0234] For example, each data storage unit stores the pixel velocities of pixels in the second image data with timestamps t19, t20, t21, t22, t23, t24, t25, t26, t27, and t28 in sequence. The processing module determines that the target image data is the second image data with timestamp t21. The processing module can fit the pixel velocities of pixels in the first motion data corresponding to the second image data with timestamp t21, specifically the image data with timestamps t19, t20, t21, t22, t23, t24, t25, t26, t27, and t28, to obtain a fitting function. Then, the processing module can input the timestamp t21 into the fitting function, and the output of the fitting function is the pixel velocity corresponding to the target image data. Subsequently, the processing module can perform deblurring processing on the first image data based on the pixel velocity corresponding to the target image data.

[0235] In this embodiment, the second motion data corresponding to any frame of third image data stored in the motion detection manager may also include the pixel velocity of pixels in multiple frames of third image data. The process by which the processing module calculates the pixel velocity corresponding to the target image data based on the pixel velocity of pixels in multiple frames of third image data in the second motion data corresponding to the target image data is similar to or the same as the process by which the processing module calculates the pixel velocity corresponding to the target image data based on the pixel velocity of pixels in multiple frames of second image data in the first motion data corresponding to the target image data, and will not be described in detail here.

[0236] The third motion data corresponding to any frame of fourth image data stored in the motion detection manager may also include the pixel velocity of pixels in multiple frames of fourth image data. The process by which the processing module calculates the pixel velocity corresponding to the target image data based on the pixel velocity of pixels in multiple frames of fourth image data in the third motion data corresponding to the target image data is similar to or the same as the process by which the processing module calculates the pixel velocity corresponding to the target image data based on the pixel velocity of pixels in multiple frames of second image data in the first motion data corresponding to the target image data, and will not be described in detail here.

[0237] In this embodiment, the pixel velocity corresponding to the target image data is not the pixel velocity of pixels in a single frame of image data, but rather a pixel velocity calculated by fitting the pixel velocities of pixels in multiple frames of original image data. Therefore, when there is a large deviation in the pixel velocity of pixels in the original image data at a certain time stamp, after that original image data at that time stamp is determined as the target image data, the deviation in the pixel velocity corresponding to the target image data will be smaller compared to the pixel velocity of pixels in the original image data at that time stamp. The larger the deviation in pixel velocity, the worse the deblurring effect using that pixel velocity. Therefore, in this embodiment, using the pixel velocity corresponding to the target image data with a smaller deviation, obtained through fitting calculation, yields better deblurring results.

[0238] Optionally, the first motion data corresponding to any frame of the second image data may further include: object distance and / or gyroscope information. The object distance is the distance between the electronic device and the object being photographed when the electronic device captures the second image data; the gyroscope information includes the angular velocity information of the electronic device when it captures the second image data.

[0239] For example, as shown in Figure 12, the first data storage unit in the motion detection manager can also store the object distance corresponding to the second image data with timestamp t19 and / or the gyroscope information corresponding to the second image data with timestamp t19. Similarly, the second data storage unit in the motion detection manager can also store the object distance corresponding to the second image data with timestamp t20 and / or the gyroscope information corresponding to the second image data with timestamp t20. That is, the object distances stored in different data storage units are different, and the gyroscope information stored in different data storage units is different. Subsequently, when the processing module performs deblurring, the processing module performs deblurring based on the object distances and / or gyroscope information in the target image data.

[0240] In a possible implementation, the data acquisition unit may also include a gyroscope sensor hardware abstraction layer. The object distance corresponding to the raw image data can be obtained by the camera during autofocus while capturing the raw image data, and then the camera can send the object distance to the camera hardware abstraction layer. Gyroscope information can be acquired by the gyroscope sensor, and then the gyroscope sensor sends the gyroscope information to the gyroscope sensor hardware abstraction layer.

[0241] It is understood that the second motion data corresponding to any frame of third image data may also include object distance and / or gyroscope information. Similarly, the third motion data corresponding to any frame of fourth image data may also include object distance and / or gyroscope information.

[0242] In this embodiment, the processing module also considers object distance and gyroscope information when performing deblurring. Since the degree of blurring of the original image data varies depending on the object distance, and also varies depending on the phone's motion posture (e.g., whether the phone is shaking or not), considering object distance and gyroscope information during deblurring can further improve the image deblurring effect.

[0243] The technical solution provided in this application embodiment will be further described below in conjunction with the process of users taking photos and viewing the captured images.

[0244] In response to a user's triggering action on the camera application icon 101 on the phone's desktop 100 in Figure 1, the phone can activate the camera. The camera can then capture raw image data in real time. Subsequently, based on the user's triggering action on the camera application icon 101 on the phone's desktop 100 and the raw image data captured by the camera, the phone can display a first interface, which may be the shooting preview interface 110 as shown in Figure 1.

[0245] In response to a user's triggering operation on the shooting preview interface 110, the mobile phone acquires first image data, which includes reference frame image data. Then, on one hand, the mobile phone acquires first motion data corresponding to multiple frames of second image data; on the other hand, the mobile phone generates a first thumbnail based on the reference frame image data.

[0246] Next, the phone generates a second thumbnail based on the first thumbnail. Then, the phone can display a second interface, which includes the second thumbnail. This second interface can be the shooting preview interface 120 shown in Figure 1.

[0247] After acquiring the first image data, the mobile phone can further process the first image data to generate a result image. The time when the mobile phone generates the result image is later than the time when the mobile phone acquires the first motion data corresponding to multiple frames of second image data.

[0248] The mobile phone's processing of the first image data may include: the mobile phone performing deblurring processing on the first image data based on the first motion data. Then, the mobile phone may display a third interface, which is the application interface of the gallery application, and includes the resulting image.

[0249] In one scenario, after the phone generates the resulting image, the phone can respond to a triggering operation on the second thumbnail and display a third interface.

[0250] For example, Figure 14 is a schematic diagram of an interface provided in an embodiment of this application. As shown in Figure 14, the interface 1400 includes a second thumbnail 1401 and a thumbnail display area 1402. The remaining content of the interface 1400 can be found in the shooting preview interface 120 in Figure 1, and will not be described again here.

[0251] Before the background camera unit in the phone completes the process of generating the resulting image, in response to the user's trigger operation on the second thumbnail 1401, the phone can display an intermediate interface 1410. The intermediate interface 1410 may include a photo button 1411, an attribute viewing button 1412, an image 1413 corresponding to the second thumbnail 1401, the shooting time, a "share" button, a "favorite" button, an "edit" button, a "delete" button, and a "more" option button, etc.

[0252] After the background camera unit in the mobile phone completes the process of generating the result image, the mobile phone can use the result image to replace the image 1413 corresponding to the second thumbnail 1401, enabling the mobile phone to access the third interface 1420. The third interface 1420 may include the result image 1421. The remaining content of the third interface 1420 can be found in the intermediate interface 1410, and will not be described in detail here.

[0253] This scenario can be understood as follows: when the user triggers the second thumbnail 1401, the background camera unit on the phone has not yet completed the process of generating the result image, and the phone can display an intermediate interface 1410. The intermediate interface 1410 includes the image 1410 corresponding to the second thumbnail 1401. After the background camera unit on the phone completes the process of generating the result image, the phone displays a third interface 1420, which includes the result image 1421.

[0254] In another scenario, after the phone generates the resulting image, it can replace the second thumbnail with a third thumbnail, which is the thumbnail corresponding to the resulting image. Then, in response to the triggering operation of the third thumbnail, the phone displays the third interface.

[0255] For example, Figure 15 is another schematic diagram of an interface provided in an embodiment of this application. As shown in Figure 15, the interface 1500 includes a second thumbnail 1501 and a thumbnail display area 1502. The remaining content of the interface 1500 can be found in the shooting preview interface 120 in Figure 1, and will not be described again here.

[0256] Before the user triggers the second thumbnail 1501, the process of generating the result image by the background camera unit in the phone has been completed. The phone can replace the second thumbnail 1501 with the thumbnail corresponding to the result image. Then, when the phone receives the user's trigger operation on the thumbnail corresponding to the result image, the phone can display the third interface 1510. The third interface 1510 may include the result image 1511. The remaining content of the third interface 1510 can be referred to the intermediate interface 1410 shown in Figure 14, and will not be described again here.

[0257] This scenario can be understood as follows: before the user triggers the second thumbnail 1501, the background camera unit in the phone has already completed the process of generating the result image. Therefore, the phone can replace the second thumbnail 1501 in the thumbnail display area 1502 with the thumbnail corresponding to the result image. Then, when the user triggers the thumbnail corresponding to the result image, the phone displays the third interface 1510, which includes the result image 1511.

[0258] In another scenario, after the phone generates the resulting image, in response to a user's trigger action on the gallery app, a fourth interface is displayed. This fourth interface is the gallery app's interface and includes a third thumbnail, which is a thumbnail corresponding to the resulting image. Then, in response to a trigger action on the third thumbnail, the third interface is displayed.

[0259] For example, Figure 16 is another schematic diagram of an interface provided in an embodiment of this application. As shown in Figure 16, the interface 1600 includes a home button 1601. The content of the interface 1600 can be referred to the interface 1400 in Figure 14, and will not be described again here.

[0260] In response to the user's triggering of the home button 1601, the phone can display interface 1610. Interface 1610 includes the gallery application icon 1611. Interface 1611 can be seen in the phone desktop 100 in Figure 1, and will not be described in detail here.

[0261] In response to a user's trigger action on the Gallery app icon 1611, the phone can display interface 1620. The display interface 1620 may include a third thumbnail 1621, a photo "button", an "Album" button, a "Moments" button, a "Discover" button, and other thumbnails.

[0262] In response to a user's triggering action on the third thumbnail 1621, the mobile phone can display a third interface 1630, which includes the result image 1631.

[0263] This scenario can be understood as follows: when the user clicks the shutter button, exits the camera app, and enters the gallery app, the background camera unit on the phone has completed the process of generating the resulting image. A third thumbnail can be displayed in the gallery app interface. The user can view the resulting image by triggering the third thumbnail.

[0264] It is understood that the above-described method of exiting the camera application from the foreground running state in response to the triggering operation of the home button 1601 is an exemplary description, and the embodiments of this application do not limit the method of exiting the camera application from the foreground running state.

[0265] For example, the phone can also respond to a user's swipe-up action on interface 1600 to exit the camera application from the foreground. The phone can also respond to a user's triggering of the multitasking key (virtual key 1602 in interface 1600 as shown in Figure 16) to display the recently opened application interface. If the recently opened application interface includes the gallery application interface, the phone responds to an operation on the gallery application interface to exit the camera application from the foreground and bring the gallery application into the foreground.

[0266] This application also provides an electronic device. Figure 17 is a schematic diagram of the hardware structure of another electronic device provided in this application. As shown in Figure 17, the electronic device may include one or more processors 1701, memory 1702, and communication interfaces 1703.

[0267] The memory 1702, communication interface 1703, and processor 1701 are coupled together. For example, the memory 1702, communication interface 1703, and processor 1701 can be coupled together via bus 1704.

[0268] The communication interface 1703 is used for data transmission with other devices. The memory 1702 stores computer program code. The computer program code includes computer instructions, which, when executed by the processor 1701, cause the electronic device to perform the relevant method steps in the above-described method embodiments of this application.

[0269] The processor 1701 can be a processor or controller, such as a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with this disclosure. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc. The bus 1704 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 1704 can be categorized as an address bus, data bus, control bus, etc. For ease of illustration, only one line is used in Figure 17, but this does not indicate that there is only one bus or one type of bus.

[0270] This application also provides a chip system, and Figure 18 is a schematic diagram of the structure of a chip system provided in this application embodiment. As shown in Figure 18, the chip system 1800 includes at least one processor 1801 and at least one interface circuit 1802. The processor 1801 and the interface circuit 1802 can be interconnected via lines. For example, the interface circuit 1802 can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit 1802 can be used to send signals to other devices (e.g., the processor 1801). Exemplarily, the interface circuit 1802 can read instructions stored in the memory and send the instructions to the processor 1801. When the instructions are executed by the processor 1801, the electronic device can perform the various steps in the above embodiments. Of course, the chip system may also include other discrete devices, and this application embodiment does not specifically limit this.

[0271] This application also provides a computer-readable storage medium storing computer program code. When the processor executes the computer program code, the electronic device executes the relevant method steps in the above method embodiments.

[0272] This application also provides a computer program product that, when run on a computer, causes the computer to execute the relevant method steps described in the above method embodiments.

[0273] The electronic devices, computer-readable storage media, or computer program products provided in this application are all used to perform the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0274] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0275] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0276] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0277] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0278] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, in essence, or the part that contributes, or all or part of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0279] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, The method includes: displaying a first interface, which is a shooting preview interface; receiving a shooting operation; acquiring first image data, which includes reference frame image data; generating a first thumbnail based on the reference frame image data; wherein, during the generation of the first thumbnail, acquiring first motion data corresponding to multiple frames of second image data; generating a second thumbnail based on the first thumbnail; displaying a second interface, which includes the second thumbnail; processing the first image data to generate a result image; wherein, processing the first image data includes: performing deblurring processing on the first image data based on the first motion data; and displaying a third interface, which is an application interface of a gallery application, including the result image.

2. The method according to claim 1, characterized in that, After receiving the shooting operation, the method further includes: acquiring second motion data corresponding to multiple frames of third image data; wherein, performing deblurring processing on the first image data based on the first motion data includes: performing deblurring processing on the first image data based on the first motion data when the multiple frames of second image data meet preset conditions; wherein, the preset conditions include: the absolute value of the difference between the timestamp of a frame of image data and a reference timestamp is less than a preset threshold, and / or, the absolute value of the difference between the timestamp of the frame of image data and the reference timestamp is less than the absolute value of the difference between a first timestamp and the reference timestamp; the first timestamp is the timestamp of any frame of image data other than the first frame of image data in the multiple frames of second image data and the multiple frames of third image data; the reference timestamp is the timestamp of the reference frame image data.

3. The method according to claim 2, characterized in that, The method further includes: when the multi-frame third image data meets the preset conditions, performing deblurring processing on the first image data based on the second motion data.

4. The method according to claim 2 or 3, characterized in that, The method further includes: when neither the multiple frames of second image data nor the multiple frames of third image data satisfy the preset condition, obtaining third motion data corresponding to the multiple frames of fourth image data; and performing deblurring processing on the first image data based on the third motion data.

5. The method according to any one of claims 1-4, characterized in that, The step of performing deblurring processing on the first image data based on the first motion data includes: performing deblurring processing on the first image data based on the first motion data corresponding to the target image data in the multi-frame second image data; wherein, the target image data is: image data in the multi-frame second image data whose absolute value of the difference between the timestamp and the reference timestamp is less than a preset threshold, and / or, the image data corresponding to the smallest first time difference among the first time differences corresponding to the multi-frame second image data; the reference timestamp is the timestamp of the reference frame image data; the first time difference corresponding to any second image data is: the absolute value of the difference between the timestamp of any second image data and the reference timestamp.

6. The method according to claim 5, characterized in that, The first motion data corresponding to a frame of second image data includes: the pixel velocity of the pixels in the multiple frames of second image data; wherein, the step of performing deblurring processing on the first image data based on the first motion data includes: calculating the pixel velocity corresponding to the target image data based on the pixel velocity of the pixels in the multiple frames of second image data in the first motion data corresponding to the target image data; and performing deblurring processing on the first image data based on the pixel velocity corresponding to the target image data.

7. The method according to claim 6, characterized in that, The step of calculating the pixel velocity corresponding to the target image data based on the pixel velocity of the pixels in the multi-frame second image data in the first motion data corresponding to the target image data includes: fitting the pixel velocity of the pixels in the multi-frame second image data in the first motion data corresponding to the target image data to obtain a fitting function; the fitting function is used to reflect the correspondence between the pixel velocity and the timestamp of the image data; and inputting the timestamp of the target image data into the fitting function to obtain the pixel velocity corresponding to the target image data.

8. The method according to any one of claims 1-7, characterized in that, The first motion data corresponding to a frame of second image data further includes: object distance and / or gyroscope information; wherein, the object distance is the distance between the electronic device and the object being photographed when the electronic device captures the second image data; the gyroscope information includes the angular velocity information of the electronic device when the electronic device captures the second image data.

9. The method according to any one of claims 1-8, characterized in that, After displaying the first interface, the process includes: real-time acquisition of image data, detection and storage of motion data corresponding to the image data; updating the stored motion data, and storing motion data corresponding to the latest detected preset number of image data; wherein, the step of obtaining the first motion data corresponding to multiple frames of second image data includes: obtaining motion data corresponding to the currently stored preset number of image data.

10. The method according to any one of claims 1-9, characterized in that, After generating the result image, the method further includes: replacing the second thumbnail with a third thumbnail, the third thumbnail being the thumbnail corresponding to the result image; displaying the third interface includes: displaying the third interface in response to a triggering operation on the third thumbnail.

11. The method according to any one of claims 1-9, characterized in that, After generating the result image, the method further includes: displaying a fourth interface, the fourth interface being the application interface of a gallery application, the fourth interface including a third thumbnail, the third thumbnail being a thumbnail corresponding to the result image; displaying the third interface includes: displaying the third interface in response to a triggering operation on the third thumbnail.

12. An electronic device, characterized in that, The electronic device includes a processor and a memory; the processor is coupled to the memory; the memory is used to store computer program code; the computer program code includes computer instructions, which, when executed by the processor, cause the electronic device to perform the method as described in any one of claims 1-11.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-11.

14. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the processors being configured to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1-11.

15. A computer program product, characterized in that, The computer program product includes instructions that, when the computer program product is run on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-11.

Images