Image processing method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to the technical field of electronic devices, and specifically provides an image processing method and device, an electronic device, and a storage medium. An image processing method is applied to an electronic device, and the method comprises: receiving initial imaging data of each frame of to-be-displayed image sent by an image sensor; the initial imaging data comprises an exposure time length of the to-be-displayed image; for each frame of to-be-displayed image, determining a target number of inserted frame images corresponding to the to-be-displayed image according to the exposure time length; determining target imaging data corresponding to the to-be-displayed image according to the initial imaging data and inserted frame image data of the inserted frame images with the target number; processing the target imaging data, and displaying an image corresponding to the target imaging data. In the present disclosure, the imaging quality and preview display fluency of the image system in a dark light scene are taken into account.

Description

Technical Field

[0001] This disclosure relates to the field of electronic equipment technology, specifically to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of electronic devices, imaging systems have become one of the most important research directions in electronic devices. In related technologies, in order to improve the imaging quality of imaging systems in low-light scenes, it is often necessary to extend the imaging exposure time. However, extending the exposure time leads to a decrease in the frame rate of the preview image in the camera viewfinder, causing stuttering. Summary of the Invention

[0003] To improve the smoothness of preview images in an imaging system, this disclosure provides an image processing method, apparatus, electronic device, and storage medium.

[0004] In a first aspect, embodiments of this disclosure provide an image processing method applied to an electronic device, the method comprising:

[0005] Receive initial imaging data of each frame of the image to be displayed from the image sensor; the initial imaging data includes the exposure time of the image to be displayed;

[0006] For each frame of an image to be displayed, the target number of interpolated frames corresponding to the image to be displayed is determined based on the exposure duration;

[0007] Based on the initial imaging data and the interpolated image data of the target number of interpolated images, the target imaging data corresponding to the image to be displayed is determined;

[0008] The target imaging data is processed, and the image corresponding to the target imaging data is displayed.

[0009] In some implementations, determining the target number of interpolated frames corresponding to the image to be displayed based on the exposure duration includes:

[0010] Based on a pre-established correspondence between exposure duration and the number of interpolated images, the target number corresponding to the exposure duration of the image to be displayed is determined; wherein the number of interpolated images is proportional to the exposure duration.

[0011] In some embodiments, determining the target imaging data corresponding to the image to be displayed based on the initial imaging data and the interpolated image data of the target number of interpolated images includes:

[0012] Based on the initial imaging data, the target number of interpolated image data is copied;

[0013] The target imaging data is determined based on the initial imaging data and the interpolated image data of the target number.

[0014] In some embodiments, processing the target imaging data and displaying the image corresponding to the target imaging data includes:

[0015] For each interpolated image, the interpolated image data of the interpolated image and the initial imaging data of the previous image to be displayed are fused to obtain fused imaging data.

[0016] The fused imaging data is cropped according to the motion direction of the electronic device to obtain the interpolated imaging data corresponding to the interpolated image.

[0017] Based on the interpolated imaging data of each interpolated image and the initial imaging data of the current frame to be displayed, each interpolated image and the current frame to be displayed are displayed sequentially.

[0018] In some implementations, the initial imaging data of each frame of the image to be displayed sent by the receiving image sensor includes:

[0019] Receive the initial imaging data of each frame of the image to be displayed, and cache the initial imaging data of each frame of the image to be displayed into the management queue in sequence.

[0020] In some embodiments, the image processing method includes:

[0021] In response to the number of initial imaging data in the management queue exceeding a preset threshold, the initial imaging data of the currently received current frame image to be displayed is discarded.

[0022] Secondly, this disclosure provides an image processing apparatus for use in an electronic device, the apparatus comprising:

[0023] The receiving module is configured to receive initial imaging data of each frame of the image to be displayed sent by the image sensor; the initial imaging data includes the exposure time of the image to be displayed.

[0024] The first determining module is configured to determine the target number of interpolated frames corresponding to each frame of the image to be displayed, based on the exposure duration.

[0025] The second determining module is configured to determine the target imaging data corresponding to the image to be displayed based on the initial imaging data and the interpolated image data of the target number of interpolated images;

[0026] The processing module is configured to process the target imaging data and display the image corresponding to the target imaging data.

[0027] In some implementations, the first determining module is configured to:

[0028] Based on a pre-established correspondence between exposure duration and the number of interpolated images, the target number corresponding to the exposure duration of the image to be displayed is determined; wherein the number of interpolated images is proportional to the exposure duration.

[0029] In some implementations, the second determining module is configured to:

[0030] Based on the initial imaging data, the target number of interpolated image data is copied;

[0031] The target imaging data is determined based on the initial imaging data and the interpolated image data of the target number.

[0032] In some implementations, the processing module is configured to:

[0033] For each interpolated image, the interpolated image data of the interpolated image and the initial imaging data of the previous image to be displayed are fused to obtain fused imaging data.

[0034] The fused imaging data is cropped according to the motion direction of the electronic device to obtain the interpolated imaging data corresponding to the interpolated image.

[0035] Based on the interpolated imaging data of each interpolated image and the initial imaging data of the current frame to be displayed, each interpolated image and the current frame to be displayed are displayed sequentially.

[0036] In some implementations, the receiving module is configured to:

[0037] Receive the initial imaging data of each frame of the image to be displayed, and cache the initial imaging data of each frame of the image to be displayed into the management queue in sequence.

[0038] In some implementations, the receiving module is configured to:

[0039] In response to the number of initial imaging data in the management queue exceeding a preset threshold, the initial imaging data of the currently received current frame image to be displayed is discarded.

[0040] Thirdly, embodiments of this disclosure provide an electronic device, including:

[0041] Processor; and

[0042] The memory stores computer instructions that can be read by the processor, and when the computer instructions are read, the processor executes the method according to any embodiment of the first aspect.

[0043] Fourthly, embodiments of this disclosure provide a storage medium for storing computer-readable instructions for causing a computer to perform the method according to any embodiment of the first aspect.

[0044] The image processing method of this disclosure includes receiving initial imaging data of each frame of an image to be displayed sent by an image sensor; determining a target number of interpolated frames corresponding to the image to be displayed based on the exposure time; determining target imaging data corresponding to the image to be displayed based on the initial imaging data and the interpolated frame data of the target number of interpolated frames; processing the target imaging data; and displaying the image corresponding to the target imaging data. In this disclosure, the number of interpolated frames is determined based on the exposure time of the image to be displayed, and then the preview interface is displayed using interpolation. This increases the frame rate of the preview interface without reducing the exposure time, resulting in a smooth and lag-free preview image observed by the user. It also balances imaging effects and smoothness in low-light scenes, improving the user experience. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the specific embodiments of this disclosure or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] Figure 1 This is a structural block diagram of an imaging system according to some embodiments of the present disclosure.

[0047] Figure 2 This is a flowchart of an image processing method according to some embodiments of the present disclosure.

[0048] Figure 3 This is a flowchart of an image processing method according to some embodiments of the present disclosure.

[0049] Figure 4 This is a flowchart of an image processing method according to some embodiments of the present disclosure.

[0050] Figure 5 This is a schematic diagram of an image processing method according to some embodiments of the present disclosure.

[0051] Figure 6 This is a schematic diagram of an image processing method according to some embodiments of the present disclosure.

[0052] Figure 7 This is a structural block diagram of an image processing apparatus according to some embodiments of the present disclosure.

[0053] Figure 8 This is a block diagram of an electronic device structure suitable for implementing the method of this disclosure. Detailed Implementation

[0054] The technical solutions of this disclosure will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure. Furthermore, the technical features involved in the different embodiments of this disclosure described below can be combined with each other as long as they do not conflict with each other.

[0055] In low-light conditions, electronic devices often extend the exposure time to increase the amount of light entering the camera and improve image quality. For example, in low-light scenarios, electronic devices set exposure times of 50ms to 100ms to improve the image quality output by the camera system. However, this extended exposure time leads to a decrease in the preview frame rate of the camera viewfinder. For instance, when the exposure time is increased to 100ms, the preview frame rate will drop from 30fps to 10fps. Especially in moving scenes, users will clearly perceive stuttering and unresponsiveness when the preview view is in motion.

[0056] In related technologies, to avoid lag issues in preview interfaces in low-light scenes, a shorter exposure time is often set. For example, setting the exposure time to 50ms will increase the frame rate of the preview interface to 20fps. However, improving the lag issue of the preview interface by shortening the exposure time results in a reduction in the amount of light entering the camera sensor, causing a significant decrease in image quality, especially in low-light scenes where the image quality is poor.

[0057] Based on the deficiencies in the aforementioned related technologies, this disclosure provides an image processing method, apparatus, electronic device, and storage medium, aiming to optimize the problem of lag in the preview interface of an image system in low-light scenes and improve the smoothness of the preview image.

[0058] Firstly, this disclosure provides an image processing method applicable to electronic devices. The electronic device described in this disclosure can be any type of device suitable for having a camera function, such as a smartphone, tablet computer, wearable device, etc., and this disclosure does not impose any limitations on this.

[0059] Figure 1 A structural block diagram of the imaging system of the electronic device according to an embodiment of the present disclosure is shown below. Figure 1 The display principle of the preview interface will be explained.

[0060] like Figure 1 As shown, the imaging system of the electronic device includes an image sensor 100, an image processing unit 200, and a display unit 300.

[0061] Image sensor 100 refers to a photosensitive device in an imaging system, such as a camera module. Image sensor 100 is used to receive external light and output corresponding imaging data based on the received light. In this embodiment, image sensor 100 can be any suitable sensor type, such as a CMOS (Complementary Metal Oxide Semiconductor) sensor, a CCD (Charge-coupled Device) sensor, etc., and this disclosure does not limit it.

[0062] Image processing unit 200 refers to a device that receives imaging data from image sensor 100 and processes the imaging data, such as an ISP (Image Signal Processing) chip. Image processing unit 200 can establish a communicable connection through, for example, a CSI (CMOS Sensor Interface) bus interface.

[0063] Display unit 300 refers to a device used to display images, such as a mobile phone display screen or a camera viewfinder. Display unit 300 can be communicatively connected to image processing unit 200, thereby receiving imaging data processed by image processing unit 200 and outputting an image.

[0064] During image capture, the image sensor 100 receives external light to generate continuous frame imaging data, and sequentially sends the imaging data of each frame to the image processing unit 200. After receiving the imaging data, the image processing unit 200 processes each frame sequentially and then sends it to the display unit 300. The display unit 300 outputs and displays the received imaging data, allowing the user to see the continuous frame output images.

[0065] In one example, the display unit 300 is used as the viewfinder of a camera app. When the camera is taking pictures, each frame of imaging data acquired by the image sensor 100 is processed by the image processing unit 200 and then displayed sequentially on the display unit 300. Thus, the user can view the preview image captured by the camera through the viewfinder. When the phone moves, the image in the viewfinder moves continuously along with it.

[0066] In this example, as mentioned above, in low-light scenes, the increased exposure time leads to a decrease in the frame rate of the preview image in the viewfinder. Especially when the phone is moving, the low frame rate causes users to noticeably perceive stuttering and lack of smoothness between preview image frames. Based on this, the present invention improves the smoothness of the preview interface without reducing the exposure time through frame interpolation, while also considering both output image quality and smoothness, thus enhancing the user experience.

[0067] like Figure 2 As shown, in some embodiments, the image processing method of this disclosure includes:

[0068] S210: Receive initial imaging data of each frame of the image to be displayed from the image sensor.

[0069] Specifically, combined Figure 1 As shown, during imaging in the imaging system, the image sensor 100 samples the incident light at a fixed frequency to obtain initial imaging data corresponding to each frame of the image. In this embodiment, each frame of data collected by the image sensor 100 is defined as initial imaging data, and the image corresponding to the initial imaging data is defined as the image to be displayed.

[0070] In some embodiments, the imaging system may include a management queue unit, which can cache the initial imaging data of each frame of image to be displayed sent by the image sensor 100 into the management queue. The processor then retrieves each frame of initial imaging data from the management queue sequentially according to a first-come, first-served principle and performs the following processing. This will be described in detail below and will not be elaborated upon here.

[0071] S220. For each frame of image to be displayed, determine the target number of interpolated frames corresponding to the image to be displayed based on the exposure duration.

[0072] It's understandable that as the exposure time increases during each frame capture, the time interval between acquiring data from adjacent frames becomes longer, resulting in a lower frame rate displayed in the preview interface. For example, in well-lit scenes, the exposure time of the imaging system can be set to 15ms, and the preview interface's frame rate can reach approximately 1000 / 15 = 66fps. However, in low-light scenes, to improve image quality, the exposure time is set to 100ms, reducing the preview interface's frame rate to approximately 1000 / 100 = 10fps.

[0073] Therefore, in this embodiment of the present disclosure, after obtaining the initial imaging data collected by the image sensor, the initial imaging data is not directly processed and imaged. Instead, the exposure time included in the initial imaging data is obtained, and the target number of interpolated images to be interpolated is determined according to the exposure time.

[0074] Specifically, for each frame of the image to be displayed acquired by the image sensor 100, the initial imaging data includes the exposure time of the image to be displayed, so that the system can obtain the corresponding exposure time when it receives each frame of initial imaging data.

[0075] In some implementations, a correspondence between exposure time and the number of interpolated images can be pre-established, thereby determining the number of interpolated images corresponding to each frame of the image to be displayed based on this correspondence. It is understood that the longer the exposure time, the lower the display frame rate of the preview interface, and thus the more interpolated images are required, and vice versa. Therefore, in this embodiment of the disclosure, for each frame of the image to be displayed, the number of corresponding interpolated images can be directly proportional to the exposure time; that is, the longer the exposure time, the more interpolated images are required, and vice versa. This will be specifically described below and will not be elaborated upon here.

[0076] In some implementations, the interpolated image data for each frame of an image to be displayed can be directly copied from the initial imaging data of that frame. That is, after determining the target number of interpolated images for a given frame, the initial imaging data of that frame can be copied a target number of times to obtain the corresponding interpolated image data. This will be described in detail below and will not be elaborated upon here.

[0077] S230. Based on the initial imaging data and the interpolated image data of the target number of interpolated images, determine the target imaging data corresponding to the image to be displayed.

[0078] Specifically, for each frame of an image to be displayed, after determining the number of interpolated frames and obtaining the interpolated frame data, the initial imaging data of the image to be displayed and the interpolated frame data of all interpolated frames can be used as the target imaging data for that frame of the image to be displayed. That is, the target imaging data includes the data of one frame of the image to be displayed and the data of the target number of interpolated frames.

[0079] S240. Process the target imaging data and display the image corresponding to the target imaging data.

[0080] Specifically, the target imaging data of each frame of the image to be displayed includes the initial imaging data of the image to be displayed and the interpolated image data of each frame of the interpolated image, so that each frame of the target imaging data can be displayed sequentially.

[0081] In one example Figure 1 Taking the display unit 300 and camera viewfinder as examples, the system sequentially displays the corresponding frame images according to the target imaging data, so that the user can observe the shooting preview image through the viewfinder.

[0082] As can be seen from the above, in this embodiment of the disclosure, the number of interpolated images is determined according to the exposure time of the image to be displayed, and then the preview interface is displayed by interpolation, thereby increasing the display frame rate of the preview interface without reducing the exposure time, so that the preview image observed by the user is smooth and without lag, while taking into account the imaging effect and smoothness in low light scenes, and improving the user experience.

[0083] In some implementations, a correspondence between exposure time and the number of interpolated frames can be pre-established before the electronic device leaves the factory, and this correspondence can be stored in the electronic device. Therefore, when a user uses the electronic device to take a video, the camera system can directly access the stored correspondence and determine the target number of interpolated frames based on the acquired exposure time.

[0084] In one example, the pre-established correspondence between exposure duration and the number of interpolated images can be shown in Table 1 below:

[0085] Exposure duration Number of interpolated images <![CDATA[0~T0]]> 0 <![CDATA[T0~T1]]> 1 <![CDATA[T1~T2]]> 2 …… …… <![CDATA[T n-1 ~T n ]]> n

[0086] It is understood that in this example, the number of interpolated frames is directly proportional to the exposure time; that is, the longer the exposure time, the more interpolated frames there are, and vice versa.

[0087] In this example, upon receiving initial imaging data of a frame of an image to be displayed from the image sensor, the exposure duration of that frame of the image to be displayed, included in the initial imaging data, can be obtained. Then, based on the correspondence shown in Table 1 above, the target number of interpolated frames corresponding to that frame of the image to be displayed can be determined. For example, if the exposure duration is between T1 and T2, the target number of interpolated frames is determined to be 2.

[0088] Of course, those skilled in the art will understand that the correspondence between exposure duration and the number of interpolated images in the embodiments of this disclosure is not limited to that shown in Table 1, but can be any other suitable correspondence, and this disclosure does not impose any limitations on this. In addition, those skilled in the art can select and set the specific numerical range of exposure duration based on prior knowledge or a limited number of experiments, and based on the specific application scenario, and this disclosure will not elaborate on this further.

[0089] As can be seen from the above, in this embodiment of the disclosure, the target number of interpolated images is determined based on the pre-established correspondence between exposure time and the number of interpolated images, thereby realizing the interpolation processing of the preview display interface, improving the display frame rate of the preview interface, reducing stuttering, and improving smoothness.

[0090] In some embodiments of this disclosure, after determining the number of interpolated frames corresponding to the image to be displayed, the initial imaging data of the image to be displayed can be copied to obtain the target number of interpolated frame image data.

[0091] like Figure 3 As shown, in some embodiments, the process of determining target imaging data of the image to be displayed in the image processing method of this disclosure includes:

[0092] S310. Based on the initial imaging data, copy the target number of interpolated image data.

[0093] S320. Determine the target imaging data based on the initial imaging data and the interpolated image data of the target number.

[0094] Specifically, taking a single frame of an image to be displayed as an example, after determining the target number of interpolated frames for that frame, the initial imaging data of the image to be displayed can be copied a target number of times to obtain the target number of interpolated frame data.

[0095] In one example, for a certain frame of an image to be displayed, based on the aforementioned Figure 2 The implementation method determines that the target number of interpolated images corresponding to the image to be displayed is 2. Therefore, the system can perform two copy operations on the initial imaging data of the image to be displayed, thereby obtaining two interpolated image data identical to the initial imaging data.

[0096] In this example, for the image to be displayed, the corresponding target imaging data includes the initial imaging data of the image to be displayed and the interpolated image data of 2 interpolated images, for a total of 3 frames of image data.

[0097] It is understandable that since the target imaging data of the image to be processed includes one frame of the image to be processed and two frames of interpolated images, the display unit can display two additional frames of interpolated images when displaying the image based on the target imaging data. This improves the display frame rate of the preview interface, avoids stuttering or unsmooth visuals for the user, and enhances the user experience.

[0098] It is worth noting that if the number of target images corresponding to a certain frame of the image to be displayed is 0, then there is no need to copy the initial imaging data of the image to be displayed. The target imaging data corresponding to the frame of the image to be displayed only needs to include the initial imaging data. This disclosure will not elaborate further on this.

[0099] Meanwhile, the above description only applies to a single frame of image to be displayed acquired by the image sensor. Those skilled in the art will understand that for multiple frames of images to be displayed, the above process can be repeated sequentially to obtain the target imaging data corresponding to each frame of image to be processed. This disclosure will not elaborate further on this.

[0100] Furthermore, considering the above embodiments, the interpolated image data is obtained by copying the initial imaging data of the image to be displayed. That is, the display content of the interpolated image is the same as that of the image to be displayed. In scenarios where the electronic device moves slightly or slowly, the change in display content between adjacent frames is small. Therefore, the copied interpolated image can significantly improve the smoothness and stability of the preview interface. However, in scenarios where the electronic device moves rapidly, the change in display content between adjacent frames is large. If the content of the interpolated image is the same as that of the image to be displayed, the preview interface observed by the user may experience a certain degree of stuttering.

[0101] Therefore, in some embodiments, the image processing method of this disclosure can process the interpolated image based on the motion direction of the electronic device, thereby making the display content of the interpolated image more coherent with the display content of the adjacent frame image to be displayed, and further improving the display smoothness of the preview interface.

[0102] like Figure 4 As shown, in some embodiments, the image processing method of this disclosure, which processes target imaging data, includes:

[0103] S410. For each interpolated image, the interpolated image data and the initial imaging data of the previous frame to be displayed are fused to obtain fused imaging data.

[0104] S420. The fused imaging data is cropped according to the motion direction of the electronic device to obtain the interpolated imaging data corresponding to the interpolated image.

[0105] S430. Based on the interpolated imaging data of each interpolated image and the initial imaging data of the current frame to be displayed, display each interpolated image and the current frame to be displayed in sequence.

[0106] In an example scenario, such as Figure 5 As shown, assuming a user is using a smartphone camera, after opening the camera app, the viewfinder on the phone screen displays the image captured by the camera. The user then moves the phone from left to right. Figure 5 As shown, assume the area within the rectangle defined by the thin solid line represents the image content captured by the image sensor 100 in the previous frame, i.e., the previous image A to be displayed. The area within the rectangle defined by the thick solid line represents the image content captured by the image sensor 100 in the current frame, i.e., the current image B to be displayed.

[0107] By using the process described in any of the foregoing embodiments of this disclosure, the target number of interpolated images and the target imaging data of the image B to be displayed in the current frame can be determined. Those skilled in the art can refer to the foregoing implementation, and this disclosure will not elaborate further.

[0108] In this example, assuming the target number of interpolated images corresponding to the current frame image B is 2, the goal of interpolation in this embodiment is to display the 2 interpolated images between the previous frame image A and the current frame image B, thereby making the transition from image A to image B in the viewfinder smoother and providing a more fluid viewing experience for the user. This will be explained in detail below.

[0109] As mentioned above, in this example, the target imaging data of the current frame image B to be displayed includes data from one frame of the image to be displayed and two frames of interpolated images. The initial imaging data of the image to be displayed is not cropped; it simply displays the content corresponding to the initial imaging data. However, the interpolated image data needs to be fused and cropped according to the motion direction to display the cropped content.

[0110] Taking one interpolated image (interpolated image B1) as an example, the interpolated image data of this frame and the initial imaging data of the previous frame to be displayed image A are fused to obtain fused imaging data.

[0111] See Figure 5 For example, it can be understood that since the interpolated image data of interpolated image B1 is copied from the initial imaging data of the current frame image B to be displayed, the display content of interpolated image B1 is the same as that of the current frame image B to be displayed. Therefore, the fused imaging data obtained after fusing interpolated image B1 with the previous frame image A to be displayed is equivalent to... Figure 5 The image shown is a fused image obtained by taking the union of image A and image B.

[0112] Secondly, during the phone's movement, motion detection devices such as the gyroscope can detect the phone's current direction of motion. For example, in this case, the gyroscope can be used to determine that the phone's current direction of motion is horizontal to the right. Figure 5 (Arrow direction). It can be understood that since the phone's movement direction is horizontal to the right, when interpolating frames between image A and image B, the content of the interpolated image should also be located between image A and image B.

[0113] Therefore, in this embodiment of the disclosure, the display content between images A and B can be cropped from the union fused image corresponding to the fused imaging data to serve as the display content of the interpolated image B1, that is, the cropping process yields the interpolated imaging data of the interpolated image B1. For example, Figure 6As shown, the rectangle with a single dotted line represents the cropped interpolated image B1, and the corresponding data is the interpolated imaging data of interpolated image B1.

[0114] Similarly, for the interpolated image B2, the above process is repeated to obtain the processed interpolated image B2, which will not be elaborated further in this disclosure. For example, Figure 6 As shown, the rectangle with double dots and dashes represents the cropped interpolated image B2, and the corresponding data is the interpolated image data of B2.

[0115] It is understandable that when the image to be displayed corresponds to multiple interpolated frames, the cropping position can be adjusted sequentially along the direction of motion when cropping the merged image. For example... Figure 6 As shown, the content of the interpolated image B2 is located to the right of the content of the interpolated image B1, so that when the viewfinder interface is displayed, it sequentially displays image A, interpolated image B1, interpolated image B2 and image B, which has a better transition effect and improves the smoothness of display from image A to image B.

[0116] Through the above process, after determining the interpolated imaging data of each interpolated image corresponding to the current frame image B to be displayed, the interpolated images of each frame and the current frame image to be displayed are processed and displayed sequentially. For example... Figure 6 As shown, when the viewfinder interface is displayed, it sequentially displays image A, interpolated image B1, interpolated image B2, and image B.

[0117] The above describes the processing procedure for the current frame image B to be displayed. Those skilled in the art will understand that for each frame of image to be displayed acquired by the image sensor, the above procedure can be followed. This disclosure will not elaborate further on this.

[0118] As can be seen from the above, in this embodiment of the disclosure, by performing fusion and cropping processing on the interpolated frame image, the transition effect between the interpolated frame image and the image to be displayed in adjacent frames is improved, further enhancing the display smoothness of the preview interface, which is especially suitable for action shooting scenarios and improves the user experience.

[0119] In some embodiments, the imaging system of this disclosure includes a management queue module, which, after receiving the initial imaging data of each frame of image to be displayed acquired by the image sensor 100, sequentially caches each initial imaging data into the management queue for queuing. The system can sequentially retrieve the initial imaging data ranked first in the management queue according to the first-come-first-served principle and perform the above processing. The specific processing procedure can be referred to by those skilled in the art in the foregoing embodiments, and will not be described in detail here.

[0120] In addition, in some implementations, when the number of initial imaging data queued in the management queue exceeds a preset threshold, it indicates that the data of images to be displayed cached in the management queue has reached the upper limit. Therefore, the initial imaging data of the latest received image to be displayed in the current frame is skipped and discarded, that is, the initial imaging data of the image to be displayed in the current frame no longer enters the management queue.

[0121] As can be seen from the above, in this embodiment of the disclosure, the data of the image to be displayed is cached by a management queue, and the data that has reached the queue limit is skipped by a discarding mechanism, which effectively avoids the blockage of the processing and ensures the stability of the system data processing.

[0122] Secondly, embodiments of this disclosure provide an image processing apparatus applied to an electronic device. The electronic device described in these embodiments can be any type of device suitable for having a camera function, such as a smartphone, tablet computer, wearable device, etc., and this disclosure does not impose any limitations thereon.

[0123] like Figure 7 As shown, in some embodiments, the image processing apparatus of this disclosure includes:

[0124] The receiving module 10 is configured to receive initial imaging data of each frame of the image to be displayed sent by the image sensor; the initial imaging data includes the exposure time of the image to be displayed.

[0125] The first determining module 20 is configured to determine the target number of interpolated frames corresponding to each frame of the image to be displayed based on the exposure duration.

[0126] The second determining module 30 is configured to determine the target imaging data corresponding to the image to be displayed based on the initial imaging data and the interpolated image data of the interpolated images of the target number;

[0127] The processing module 40 is configured to process the target imaging data and display the image corresponding to the target imaging data.

[0128] As can be seen from the above, in this embodiment of the disclosure, the number of interpolated images is determined according to the exposure time of the image to be displayed, and then the preview interface is displayed by interpolation, thereby increasing the display frame rate of the preview interface without reducing the exposure time, so that the preview image observed by the user is smooth and without lag, while taking into account the imaging effect and smoothness in low light scenes, and improving the user experience.

[0129] In some implementations, the first determining module 20 is configured to:

[0130] Based on the pre-established correspondence between exposure duration and the number of interpolated images, the target number corresponding to the exposure duration of the image to be displayed is determined; wherein, the number of interpolated images is proportional to the exposure duration.

[0131] As can be seen from the above, in this embodiment of the disclosure, the target number of interpolated images is determined based on the pre-established correspondence between exposure time and the number of interpolated images, thereby realizing the interpolation processing of the preview display interface, improving the display frame rate of the preview interface, reducing stuttering, and improving smoothness.

[0132] In some implementations, the second determining module 30 is configured to:

[0133] Based on the initial imaging data, the target number of interpolated image data is obtained by copying the data.

[0134] The target imaging data is determined based on the initial imaging data and the interpolated image data of the target number.

[0135] In some implementations, the processing module 40 is configured to:

[0136] For each interpolated image, the interpolated image data and the initial imaging data of the previous image to be displayed are fused to obtain fused imaging data.

[0137] The fused imaging data is cropped according to the motion direction of the electronic device to obtain the interpolated imaging data corresponding to the interpolated image.

[0138] Based on the interpolated imaging data of each interpolated image and the initial imaging data of the current frame to be displayed, the interpolated images of each frame and the current frame to be displayed are displayed sequentially.

[0139] As can be seen from the above, in this embodiment of the disclosure, by performing fusion and cropping processing on the interpolated frame image, the transition effect between the interpolated frame image and the image to be displayed in adjacent frames is improved, further enhancing the display smoothness of the preview interface, which is especially suitable for action shooting scenarios and improves the user experience.

[0140] In some implementations, the receiving module 10 is configured to:

[0141] Receive the initial imaging data of each frame of the image to be displayed, and cache the initial imaging data of each frame of the image to be displayed into the management queue in sequence.

[0142] In some implementations, the receiving module 10 is configured to:

[0143] In response to the number of initial imaging data in the management queue exceeding a preset threshold, discard the initial imaging data of the currently received current frame image to be displayed.

[0144] As can be seen from the above, in this embodiment of the disclosure, the data of the image to be displayed is cached by a management queue, and the data that has reached the queue limit is skipped by a discarding mechanism, which effectively avoids the blockage of the processing and ensures the stability of the system data processing.

[0145] Thirdly, embodiments of this disclosure provide an electronic device, including:

[0146] Processor; and

[0147] The memory stores computer instructions that can be read by a processor, which, when the computer instructions are read, executes the method according to any embodiment of the first aspect.

[0148] Fourthly, embodiments of this disclosure provide a storage medium for storing computer-readable instructions for causing a computer to perform a method according to any embodiment of the first aspect.

[0149] Figure 8 The diagram shows a structural block diagram of an electronic device according to some embodiments of the present disclosure. The following is a description of the structure of the device in conjunction with the provided text. Figure 8 The principles related to the electronic devices and storage media of some embodiments of this disclosure will be explained.

[0150] Reference Figure 8 The electronic device 1800 may include one or more of the following components: a processing component 1802, a memory 1804, a power supply component 1806, a multimedia component 1808, an audio component 1810, an input / output (I / O) interface 1812, a sensor component 1816, and a communication component 1818.

[0151] Processing component 1802 typically controls the overall operation of electronic device 1800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 1802 may include one or more processors 1820 to execute instructions. Furthermore, processing component 1802 may include one or more modules to facilitate interaction between processing component 1802 and other components. For example, processing component 1802 may include a multimedia module to facilitate interaction between multimedia component 1808 and processing component 1802. As another example, processing component 1802 may read executable instructions from memory to implement relevant functions of the electronic device.

[0152] Memory 1804 is configured to store various types of data to support the operation of electronic device 1800. Examples of this data include instructions for any application or method operating on electronic device 1800, contact data, phonebook data, messages, pictures, videos, etc. Memory 1804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0153] Power supply component 1806 provides power to various components of electronic device 1800. Power supply component 1806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 1800.

[0154] The multimedia component 1808 includes a display screen that provides an output interface between the electronic device 1800 and the user. In some embodiments, the multimedia component 1808 includes a front-facing camera and / or a rear-facing camera. When the electronic device 1800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera can receive external multimedia data. Each front-facing camera and rear-facing camera can be a fixed optical lens system or have focal length and optical zoom capabilities.

[0155] Audio component 1810 is configured to output and / or input audio signals. For example, audio component 1810 includes a microphone (MIC) configured to receive external audio signals when electronic device 1800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1804 or transmitted via communication component 1818. In some embodiments, audio component 1810 includes a speaker for outputting audio signals.

[0156] I / O interface 1812 provides an interface between processing component 1802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0157] Sensor assembly 1816 includes one or more sensors for providing state assessments of various aspects of electronic device 1800. For example, sensor assembly 1816 may detect the on / off state of electronic device 1800, the relative positioning of components such as the display and keypad of electronic device 1800, changes in position of electronic device 1800 or a component of electronic device 1800, the presence or absence of user contact with electronic device 1800, the orientation or acceleration / deceleration of electronic device 1800, and temperature changes of electronic device 1800. Sensor assembly 1816 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1816 may include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 1816 may include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0158] Communication component 1818 is configured to facilitate wired or wireless communication between electronic device 1800 and other devices. Electronic device 1800 can access wireless networks based on communication standards, such as Wi-Fi, 2G, 3G, 4G, 5G, or 6G, or combinations thereof. In one exemplary embodiment, communication component 1818 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 1818 includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0159] In an exemplary embodiment, the electronic device 1800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components.

[0160] Obviously, the above embodiments are merely examples for clear illustration and are not intended to limit the embodiments. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all embodiments here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this disclosure.

Claims

1. An image processing method, characterized in that, include: Receive initial imaging data for each frame of the image to be displayed from the image sensor; The initial imaging data includes the exposure time of the image to be displayed; For each frame of an image to be displayed, the target number of interpolated frames corresponding to the image to be displayed is determined based on the exposure duration; Based on the initial imaging data and the interpolated image data of the target number of interpolated images, the target imaging data corresponding to the image to be displayed is determined; For each interpolated image, the interpolated image data and the initial imaging data of the previous frame to be displayed, which are included in the target imaging data, are fused to obtain fused imaging data. The fused imaging data is cropped according to the motion direction of the electronic device to obtain the interpolated imaging data corresponding to the interpolated image. Based on the interpolated imaging data of each interpolated image and the initial imaging data of the current frame to be displayed, each interpolated image and the current frame to be displayed are displayed sequentially.

2. The method according to claim 1, characterized in that, Determining the target number of interpolated frames corresponding to the image to be displayed based on the exposure duration includes: Based on a pre-established correspondence between exposure duration and the number of interpolated images, the target number corresponding to the exposure duration of the image to be displayed is determined; wherein the number of interpolated images is proportional to the exposure duration.

3. The method according to claim 1, characterized in that, The step of determining the target imaging data corresponding to the image to be displayed based on the initial imaging data and the interpolated image data of the target number of interpolated images includes: Based on the initial imaging data, the target number of interpolated image data is copied; The target imaging data is determined based on the initial imaging data and the interpolated image data of the target number.

4. The method according to claim 1, characterized in that, The initial imaging data for each frame of the image to be displayed, sent by the receiving image sensor, includes: Receive the initial imaging data of each frame of the image to be displayed, and cache the initial imaging data of each frame of the image to be displayed into the management queue in sequence.

5. The method according to claim 4, characterized in that, include: In response to the number of initial imaging data in the management queue exceeding a preset threshold, the initial imaging data of the currently received current frame image to be displayed is discarded.

6. An image processing apparatus, characterized in that, include: The receiving module is configured to receive initial imaging data of each frame of the image to be displayed sent by the image sensor; The initial imaging data includes the exposure time of the image to be displayed; The first determining module is configured to determine the target number of interpolated frames corresponding to each frame of the image to be displayed, based on the exposure duration. The second determining module is configured to determine the target imaging data corresponding to the image to be displayed based on the initial imaging data and the interpolated image data of the target number of interpolated images; The processing module is configured to, for each interpolated image, perform fusion processing on the interpolated image data included in the target imaging data and the initial imaging data of the previous frame to be displayed, to obtain fused imaging data; perform cropping processing on the fused imaging data according to the motion direction of the electronic device, to obtain interpolated imaging data corresponding to the interpolated image; and sequentially display each interpolated image and the current frame to be displayed based on the interpolated imaging data of each interpolated image and the initial imaging data of the current frame to be displayed.

7. An electronic device, characterized in that, include: processor; as well as The memory stores computer instructions that can be read by the processor, and when the computer instructions are read, the processor executes the method according to any one of claims 1 to 5.

8. A storage medium, characterized in that, Used to store computer-readable instructions for causing a computer to perform the method according to any one of claims 1 to 5.

Images