Image processing method and device, electronic equipment and readable storage medium

Abstract

This application discloses an image processing method, apparatus, electronic device, and readable storage medium, belonging to the field of image processing technology. The image processing method includes: acquiring M frames of images to be processed, which are then fused to form a target image; determining N frames of images to be processed from the M frames based on the degree of inter-frame difference corresponding to each of the M frames; determining a playback image from the N frames based on image feature information corresponding to each of the N frames, wherein the image feature information includes at least one of the following: brightness difference degree, color difference degree, and detail richness; storing the playback image in a photo album; and, if a target image is obtained, replacing the playback image in the photo album with the target image.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and specifically relates to an image processing method, apparatus, electronic device and readable storage medium. Background Technology

[0002] With the development of smart terminal technology, smart terminal devices are offering increasingly richer functions. For example, users can use the camera function provided by the terminal to capture images or videos. During the shooting process, the terminal often performs a series of algorithms such as enhancement, beautification, and compression on the image data, which means that users need to wait for a while after clicking to shoot before they can view the image.

[0003] Currently, after a user clicks to take a picture, an unprocessed image is selected from the cached image frames for the user to view. After the processed image is generated, the unprocessed image is replaced with the processed image. However, if the two frames are too different, the first image frame displayed will affect the user's judgment of the shooting satisfaction. Summary of the Invention

[0004] The purpose of this application is to provide an image processing method, apparatus, electronic device, and readable storage medium that can improve the similarity between the previously displayed echo image and the target captured image displayed later, and avoid excessive differences between the two frames affecting the user's judgment of shooting satisfaction.

[0005] In a first aspect, embodiments of this application provide an image processing method, the method comprising:

[0006] Acquire M frames of images to be processed; these M frames are used for fusion processing to create the target image.

[0007] Based on the degree of inter-frame difference corresponding to the M frames of images to be processed, determine the N frames of images to be processed from the M frames of images to be processed;

[0008] Based on the image feature information corresponding to each of the N frames to be processed, the image to be displayed is determined from the N frames to be processed. The image feature information includes at least one of the following: brightness difference degree, color difference degree, and detail richness.

[0009] Save the echoed image to the photo album;

[0010] Once the target image is obtained, replace the displayed image in the album with the target image.

[0011] Secondly, embodiments of this application provide an image processing apparatus, the apparatus comprising:

[0012] The acquisition module is used to acquire M frames of images to be processed, which are then fused together to form the target image.

[0013] The processing module is used to determine N frames of images to be processed from the M frames of images to be processed based on the degree of inter-frame differences corresponding to the M frames of images to be processed.

[0014] The processing module is also used to determine the echo image from the N frames of images to be processed based on the image feature information corresponding to each of the N frames of images to be processed. The image feature information includes at least one of the following: brightness difference degree, color difference degree, and detail richness.

[0015] The storage module is used to store the displayed images to the photo album;

[0016] The storage module is also used to replace the displayed image in the album with the target image when the target image is obtained.

[0017] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0018] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0019] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0020] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0021] In this embodiment, during the shooting process, after the terminal acquires the images to be processed for fusion processing into the target captured image, it determines N frames to be processed from the M frames based on the degree of inter-frame difference corresponding to each of the M frames to be processed. Then, based on the image feature information corresponding to each of the N frames to be processed, it determines the playback image from the N frames to be processed. Since the image feature information may include at least one of the following: degree of brightness difference, degree of color difference, and degree of detail richness, it can improve the similarity between the playback image displayed first and the target captured image displayed later, avoiding excessive differences between the two frames from affecting the user's judgment of shooting satisfaction. Attached Figure Description

[0022] Figure 1This is a schematic flowchart of an image processing method provided in an embodiment of this application;

[0023] Figure 2 This is a schematic diagram of a processing flow for selecting a playback image and generating a target image, provided in an embodiment of this application.

[0024] Figure 3 This application is a schematic diagram illustrating the positional relationship between a pixel and its neighboring pixels, provided in an embodiment.

[0025] Figure 4 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0026] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0027] Figure 6 This is a schematic diagram of the hardware structure of another electronic device provided in an embodiment of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0029] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0030] With the development of smart terminal technology, the functions offered by smart terminal devices are becoming increasingly diverse. For example, users can use the camera function provided by the terminal to capture images or videos. To improve image quality, the terminal often requires a series of algorithmic processing steps such as image enhancement, beautification, and compression. This results in users having to wait a period of time after clicking to capture before they can view the image.

[0031] Currently, after a user clicks to take a picture, an unprocessed image is selected from the cached image frames for the user to view. After the processed image is generated, the unprocessed image is replaced with the processed image. However, if the two frames are too different, the first image frame displayed will affect the user's judgment of the shooting satisfaction.

[0032] In view of this, the purpose of the embodiments of this application is to provide an image processing method, apparatus, electronic device and readable storage medium to improve the similarity between the displayed echo image and the target captured image displayed later, and to avoid the difference between the two frames being too large and affecting the user's judgment of the shooting satisfaction.

[0033] The image processing method provided in this application will be described in detail below with reference to the accompanying drawings and specific embodiments and application scenarios. The image processing method in this application can be applied to electronic devices that can be used for photography, such as mobile phones, tablets, and laptops, and is not specifically limited thereto.

[0034] Figure 1 This is a schematic flowchart of an image processing method provided in an embodiment of this application. Combined with... Figure 1 As shown, the image processing method may include steps 110 to 150.

[0035] Step 110: Obtain M frames of images to be processed. The M frames of images to be processed are used for fusion processing to obtain the target image.

[0036] Step 120: Determine N frames to be processed from the M frames to be processed based on the degree of inter-frame difference corresponding to each of the M frames to be processed.

[0037] Step 130: Determine the display image from the N frames of images to be processed based on the image feature information corresponding to each of the N frames of images to be processed. The image feature information includes at least one of the following: brightness difference degree, color difference degree, and detail richness.

[0038] Step 140: Save the displayed image to the album;

[0039] Step 150: If the target image is obtained, replace the displayed image in the album with the target image.

[0040] The steps described above are explained in detail below.

[0041] Specifically, regarding step 110 above, after the user activates the camera function of the electronic device, a preview image of the current shooting scene can be displayed on the camera preview interface of the electronic device. The electronic device can cache multiple frames of preview images. Upon receiving input from the user to control the electronic device to shoot, it can extract M images to be processed from the cached multiple frames of preview images for fusion processing into the target shooting image. It is understood that M is a positive integer greater than or equal to 1. The fusion processing of the M frames of images to be processed may include image registration, image denoising, image enhancement, image fusion calculation, etc., and is not specifically limited here.

[0042] Regarding step 120 above, for each of the M frames of images to be processed, the inter-frame difference value of each frame is calculated compared to the other M-1 frames, thus obtaining the degree of inter-frame difference for each frame. For example, the degree of inter-frame difference for each of the M frames can be determined through inter-frame difference calculation, optical flow amplitude calculation, etc. Based on the degree of inter-frame difference, the viewpoint difference between each frame and the remaining images in the M frames can be characterized.

[0043] In some embodiments, based on display requirements, a preset filtering condition can be used to determine N frames of images to be processed from M frames of images to be processed. For example, to reduce the small difference in viewing angle between the selected N frames of images to be processed and the final generated target image, the preset filtering condition can be set to obtain the N frames of images to be processed with the smallest inter-frame difference value. It is understood that N is a positive integer less than or equal to M. In this way, after determining the N frames of images to be processed through the preset filtering condition, the displayed image determined from the N frames of images to be processed can effectively reduce the visual effect of abrupt changes in viewing angle between the displayed image and the target image.

[0044] Regarding step 130 above, since the imaging time of each frame of the image to be processed is different, the corresponding image feature information for each frame of the image to be processed also differs. Image feature information includes, for example, the degree of brightness difference, the degree of color difference, and the richness of detail. Optionally, the image feature information used to select the image for playback can be adjusted according to the actual shooting scene.

[0045] For example, when the ambient brightness is high and the light source is stable, the degree of brightness difference can be used as image feature information to filter N frames of images to be processed. When the electronic device used to capture the image is in a stable state, the degree of detail richness can be used as image feature information to filter N frames of images to be processed. No specific limitation is made here.

[0046] In steps 140 to 150 above, after determining the echo image from N frames of images to be processed based on image feature information, the echo image is stored in the album.

[0047] In an optional example, if the user opens the photo album and needs to view the captured image, the echo image can be displayed directly. If the user does not need to open the photo album, and the shooting interface includes a small window to display the captured image, a thumbnail of the echo image can also be quickly displayed in the small window, effectively achieving a seamless shooting experience from shooting to display. Furthermore, the electronic device performs image fusion processing based on M frames of images to be processed to generate the target captured image. Once the target captured image is obtained, the echo image in the photo album is replaced with the target captured image.

[0048] Figure 2 This is a schematic diagram illustrating a processing flow for selecting a playback image and generating a target image, provided in an embodiment of this application. Combined with... Figure 2 As shown, the electronic device acquires M frames of images to be processed. On the one hand, it selects a display image from the M frames of images to be processed and stores the display image in the album. On the other hand, it performs fusion processing on the M frames of images to be processed, and after generating the target image, it replaces the target image in the album with the target image.

[0049] Based on the embodiments of this application, since the visual difference between the selected echo image and the target captured image is small, after storing the echo image in the album, when the target captured image is obtained, the echo image in the album is replaced with the target captured image. In this way, the similarity between the echo image displayed in front and the target captured image displayed later can be effectively improved, and the difference between the two frames is too large, which will affect the user's judgment of shooting satisfaction.

[0050] As a concrete example, based on the degree of inter-frame difference corresponding to the M frames to be processed, N frames to be processed are determined from the M frames to be processed. The specific steps are as follows:

[0051] For each of the M frames to be processed, the optical flow amplitude corresponding to the image to be processed is obtained. The optical flow amplitude is used to characterize the degree of inter-frame difference between the image to be processed and the remaining images to be processed in the M frames. Based on the optical flow amplitude corresponding to each image to be processed, N frames to be processed are determined from the M frames. The optical flow amplitudes corresponding to the N frames to be processed satisfy the target selection conditions.

[0052] For example, the optical flow amplitude can be used to characterize the degree of inter-frame difference between the image to be processed and the remaining images to be processed in the M frames of images to be processed. That is, the degree of inter-frame difference can reflect the viewpoint difference between each frame of images to be processed and the remaining images to be processed in the M frames of images to be processed.

[0053] Taking the calculation of the m-th frame of the M-frame images to be processed as an example, the optical flow amplitude of the m-th frame can characterize the degree of inter-frame difference between the m-th frame and the other M-frame images to be processed. Optionally, by calculating the optical flow difference values ​​between the m-th frame and the remaining M-1 frames of the M-frame images to be processed, the degree of inter-frame difference of the m-th frame can be determined based on the M-1 optical flow difference values.

[0054] Specifically, for each frame of the M-frame images to be processed, the optical flow amplitude corresponding to the image to be processed is obtained. The specific steps can be as follows: For each frame of the M-frame images to be processed, the optical flow difference value between the image to be processed and the remaining images to be processed in each frame is obtained; the mean value of the optical flow difference value between the image to be processed and the remaining images to be processed in each frame is obtained, and the mean value is used as the optical flow amplitude corresponding to the image to be processed.

[0055] For example, the optical flow difference value can be used to represent the displacement of each pixel representing the same object (or, object) between two frames of images to be processed. Optionally, this displacement can be represented by a two-dimensional vector. The optical flow difference value between the two frames can be obtained by calculating the difference between each pixel representing the same object (or, object) in the two frames. Optionally, the optical flow difference value between the two frames can be represented in matrix form, which is not specifically limited here.

[0056] As a specific example, the M frames to be processed are I1, I2, ..., Im, ..., IM.

[0057] Taking the first frame image I1 and the second frame image I2 as examples, for the first frame image I1, the optical flow difference values ​​between I1 and I2 to IM are calculated to obtain M-1 optical flow difference values. For ease of description, the M-1 optical flow difference values ​​of the second frame image I2 are represented in the form of an optical flow difference value set FI1.

[0058] For the second frame image I2 to be processed, calculate the optical flow difference values ​​between I2 and I1, I3 to IM respectively, and obtain M-1 optical flow difference values. For ease of description, the M-1 optical flow difference values ​​of the second frame image I2 to be processed are represented in the form of the optical flow difference value set FI2.

[0059] Understandably, for the m-th frame of the image to be processed, Im, the optical flow difference values ​​between Im and I1 to I(m-1) and I(m+1) to IM are calculated. For ease of description, the M-1 optical flow difference values ​​of the m-th frame of the image to be processed, Im, are represented in the form of the optical flow difference value set FIm, where m∈M, and are not listed here. Based on this, the optical flow difference values ​​FI1, FI2, ..., FIm, ..., FIM corresponding to the M frames of the image to be processed can be obtained.

[0060] Each optical flow difference value in the set of optical flow difference values ​​can be represented in the form of a matrix. In order to facilitate calculation and improve the selection speed of the displayed image, the mean of the optical flow difference values ​​between the image to be processed and the remaining images to be processed in each frame can be calculated separately, and the mean value can be used as the optical flow amplitude corresponding to the image to be processed.

[0061] For example, taking the first frame image I1 to be processed as an example, the mean of M-1 optical flow difference values ​​in the optical flow difference value set FI1 is calculated to obtain the optical flow amplitude of the first frame image I1 to be processed. Taking the second frame image I2 to be processed as an example, the mean of M-1 optical flow difference values ​​in the optical flow difference value set FI2 is calculated to obtain the optical flow amplitude of the second frame image I2 to be processed.

[0062] Understandably, for the m-th frame of the image to be processed, Im, the mean of the M-1 optical flow difference values ​​in the set FIm is calculated to obtain the optical flow amplitude of the m-th frame of the image to be processed, which will not be listed here. Based on this, the optical flow amplitude of each frame of the image to be processed can be obtained.

[0063] In some embodiments, the smaller the optical flow amplitude, the smaller the inter-frame difference between the image to be processed and the remaining images to be processed in the M frames of images to be processed, and the smaller the viewpoint difference between each frame of images to be processed and the remaining images to be processed in the M frames of images to be processed. As a specific example, the target selection criterion can be to select the N frames of images to be processed with the smallest optical flow amplitude from the M frames of images to be processed.

[0064] Based on the embodiments of this application, after determining N frames of images to be processed by preset filtering conditions, the echo image determined from the N frames of images to be processed can effectively reduce the visual effect of perspective jump between the echo image and the target captured image, improve the similarity between the echo image displayed in front and the target captured image displayed behind, and avoid the difference between the two frames being too large, which would affect the user's judgment of shooting satisfaction.

[0065] As a concrete example, for each of the N frames of images to be processed, the degree of brightness difference corresponding to the image to be processed is used to characterize the brightness difference between the image to be processed and the remaining images to be processed in the N frames of images to be processed.

[0066] You can refer to the following steps to determine the degree of brightness difference in the image to be processed:

[0067] Obtain the brightness value of the image to be processed; obtain the average brightness value of N frames of the image to be processed; determine the degree of brightness difference of the image to be processed based on the difference between the brightness value of the image to be processed and the average brightness value.

[0068] Specifically, the brightness value of the image to be processed can represent the brightness level of the image. The brightness values ​​of N frames of the image to be processed can be represented as A1, A2, ..., An, ..., AN, respectively. The calculation method of the brightness value is not specifically limited here.

[0069] Average brightness values ​​of N frames of images to be processed It can be as shown in formula (1).

[0070] (1)

[0071] For the first frame of N frames to be processed, calculate A1 and the mean luminance value. The difference between A2 and the mean brightness value yields the degree of brightness difference in the first frame of the image to be processed. For the second frame of the N frames of images to be processed, calculate A2 and the mean brightness value. The difference between the values ​​of An and the mean brightness value can be used to determine the degree of brightness difference in the second frame of the image to be processed. It can be understood that for the nth frame of N frames to be processed, An is calculated as a function of the mean brightness value. The difference can be used to obtain the brightness difference of the nth frame of the image to be processed, where n∈N. The calculation process of the brightness difference of the N frames of the image to be processed will not be listed here.

[0072] As a concrete example, for each of the N frames of images to be processed, the degree of color difference corresponding to the image to be processed is used to characterize the color difference between the image to be processed and the remaining images to be processed in the N frames.

[0073] You can refer to the following steps to determine the degree of color difference in the image to be processed:

[0074] Obtain the hue value and saturation of the image to be processed; obtain the average hue value and the average saturation value of N frames of the image to be processed; determine the degree of color difference of the image to be processed based on the difference between the hue value and the average hue value, and the difference between the saturation value and the average saturation value.

[0075] For example, the hue values ​​of N frames of images to be processed can be represented as B1, B2, ..., Bn, ..., BN. The saturation values ​​of N frames of images to be processed can be represented as C1, C2, ..., Cn, ..., CN. The calculation methods for hue values ​​and saturation are not specifically limited here.

[0076] Tonal mean of N frames of images to be processed It can be as shown in formula (2).

[0077] (2)

[0078] Mean saturation of N frames of images to be processed It can be as shown in formula (3).

[0079] (3)

[0080] For the first frame of N frames to be processed, calculate B1 and the hue mean. The difference between B2 and the mean tone value of the first frame of the image to be processed can be obtained. For the second frame of the N frames of images to be processed, calculate B2 and the mean tone value. The difference between Bn and the mean tone value of the second frame of the image to be processed can be obtained. It can be understood that for the nth frame of N frames of images to be processed, the difference between Bn and the mean tone value is calculated. The difference between the tonal value and the tonal mean of the nth frame of the image to be processed can be obtained, where n∈N. The calculation process of the difference between the tonal value and the tonal mean of the N frames of the image to be processed will not be listed here.

[0081] For the first frame of N frames to be processed, calculate C1 and the mean saturation value. The difference between C2 and the mean saturation value of the first frame of the image to be processed can be obtained. For the second frame of the N frames of images to be processed, calculate C2 and the mean saturation value. The difference between Cn and the mean saturation value of the second frame of the image to be processed can be obtained. It can be understood that, for the nth frame of the N frames to be processed, the difference between Cn and the mean saturation value is calculated. The difference between the saturation and the mean saturation of the nth frame of the image to be processed can be obtained, where n∈N. The calculation process of the difference between the saturation and the mean saturation of the N frames of the image to be processed will not be listed here.

[0082] As a specific example, another formula (4) is used to determine the degree of color difference corresponding to the image to be processed.

[0083] (4)

[0084] In formula (4), Di is the degree of color difference of the i-th frame of the N-frame images to be processed, Bi is the hue value of the i-th frame of the N-frame images to be processed, and Ci is the color saturation of the i-th frame of the N-frame images to be processed.

[0085] As a concrete example, for each of the N frames to be processed, the level of detail in that frame is used to characterize the overall detail richness of the image. The following steps can be used to determine the level of detail in the image to be processed:

[0086] For each pixel in the image to be processed, the dispersion of the pixel is determined based on the gray value of the pixel and the gray values ​​of a preset number of neighboring pixels. Based on the dispersion of each pixel in the image to be processed, the mean dispersion of the image to be processed is determined. The mean dispersion is used to characterize the richness of detail in the image to be processed.

[0087] For example, the preset number of neighboring pixels of a pixel in the image to be processed refers to the preset number of pixels closest to the pixel. The preset number can be set according to the actual application scenario and is not specifically limited here. For example, the preset number can be 4, which can select the nearest pixels in the four directions of up, down, left, and right of the pixel; the preset number can be 9, which can select the nearest pixels in the eight directions of up, down, left, right, top left, bottom left, top right, and bottom right of the pixel; optionally, when the preset number is larger, the nearest pixels can be selected, and there is no specific limit to the preset number here.

[0088] As a specific example, Figure 3 This is a schematic diagram illustrating the positional relationship between a pixel and its neighboring pixels, provided in an embodiment of this application. The preset number is 9, including pixel 301, and the neighboring pixels are shown in Figure 302.

[0089] It is understandable that when the discreteness of pixels at the edge needs to be calculated, the gray value corresponding to the position of the missing neighboring pixels can be set to the default value so that the number of neighboring pixels reaches the preset number. For example, the gray value can be set to 128. No specific restrictions are made here.

[0090] Optionally, the dispersion of each pixel can be determined by calculating the variance or standard deviation; no specific limitation is imposed here. After determining the dispersion of each pixel, the mean dispersion of pixels in each frame of the image to be processed can be determined.

[0091] In this embodiment, the mean dispersion is used to characterize the level of detail in the image to be processed. A larger mean dispersion indicates greater detail, while a smaller mean dispersion indicates less detail. Accordingly, when selecting a display image based on the level of detail, the image with the largest mean dispersion can be selected as the display image.

[0092] According to the embodiments of this application, the richness of detail is such that when selecting a playback image from N frames of images to be processed and replacing it with the target captured image, the visual effect changes caused by the difference in image details can be effectively reduced, the similarity between the playback image displayed in front and the target captured image displayed behind can be improved, and the difference between the two frames is too large, which can affect the user's judgment of shooting satisfaction.

[0093] In some embodiments, image feature information may include: brightness difference level, color difference level, and detail richness. Specifically, relating to step 240 above, determining the echo image from the N frames of images to be processed based on the image feature information corresponding to each of the N frames to be processed can be achieved by referring to the following steps: determining J frames of images to be processed from the N frames of images to be processed based on the brightness difference level corresponding to each of the N frames of images to be processed; determining K frames of images to be processed from the J frames of images to be processed based on the color difference level corresponding to each of the J frames of images to be processed; and determining the echo image from the K frames of images to be processed based on the detail richness corresponding to each of the K frames of images to be processed.

[0094] For example, when selecting images to be processed based on the degree of brightness difference, the J-frame images with the smallest brightness difference can be selected. In this way, after selecting J-frame images to be processed based on the degree of brightness difference, the display image is determined from the J-frame images to be processed. This can effectively reduce the visual effect of brightness jumps between the display image and the target image, which is conducive to improving the similarity between the display image displayed in front and the target image displayed behind, and avoids the difference between the two frames being too large, which would affect the user's judgment of shooting satisfaction.

[0095] When selecting images to be processed based on the degree of color difference, the K frames with the smallest color difference can be selected. After selecting K frames to be processed based on the degree of color difference, the displayed image is determined from the K frames. This ensures that the displayed image contains uniform colors, effectively reducing the visual effect of color jumps between the displayed image and the target image. It helps to improve the similarity between the displayed image and the target image, and avoids that the difference between the two frames is too large, which may affect the user's judgment of shooting satisfaction.

[0096] When selecting images to be processed based on the richness of detail, the image with the highest level of detail can be selected as the display image. This ensures that the display image has rich image detail information and effectively reduces the visual effect of abrupt changes in image detail between the display image and the target image. This improves the similarity between the display image displayed first and the target image displayed later, and avoids excessive differences between the two frames affecting the user's judgment of shooting satisfaction.

[0097] Based on the embodiments of this application, the echo image is selected from N frames of images to be processed based on the degree of brightness difference, the degree of color difference, and the richness of detail. This can effectively reduce the changes in visual effect after the echo image and the target captured image are replaced, and improve the similarity between the echo image displayed in front and the target captured image displayed behind.

[0098] The image processing method provided in this application can be executed by an image processing device. This application uses an image processing device executing the image processing method as an example to illustrate the image processing device provided in this application.

[0099] Figure 4 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application, combined with... Figure 4 As shown, the image processing device may include an acquisition module 410, a processing module 420, and a storage module 430.

[0100] The acquisition module 410 is used to acquire M frames of images to be processed, which are then fused to form the target image.

[0101] Processing module 420 is used to determine N frames to be processed from M frames of images to be processed based on the degree of inter-frame difference corresponding to each of the M frames of images to be processed.

[0102] The processing module 420 is also used to determine the display image from the N frames of images to be processed based on the image feature information corresponding to each of the N frames of images to be processed. The image feature information includes at least one of the following: brightness difference degree, color difference degree, and detail richness.

[0103] Storage module 430 is used to store the echoed image to the photo album;

[0104] The storage module 430 is also used to replace the displayed image in the album with the target captured image when the target captured image is obtained.

[0105] Based on the embodiments of this application, a display image that can be shown to the user can be quickly selected from multiple frames of images to be processed, which can meet the user's need to view the captured image immediately, and can improve the similarity between the display image displayed in front and the target captured image displayed later, avoiding the large difference between the two frames affecting the user's judgment of the shooting satisfaction.

[0106] In some embodiments, the acquisition module 410 is further configured to acquire the optical flow amplitude corresponding to each frame of the M frames of images to be processed, wherein the optical flow amplitude is used to characterize the degree of inter-frame difference between the image to be processed and the remaining images to be processed in the M frames of images to be processed.

[0107] The processing module 420 is further configured to determine the N frames of images to be processed from the M frames of images to be processed based on the optical flow amplitude corresponding to each frame of images to be processed, wherein the optical flow amplitudes corresponding to the N frames of images to be processed respectively meet the target screening conditions.

[0108] Based on the embodiments of this application, after determining N frames of images to be processed by preset filtering conditions, the echo image determined from the N frames of images to be processed can effectively reduce the visual effect of perspective jump between the echo image and the target captured image, improve the similarity between the echo image displayed in front and the target captured image displayed behind, and avoid the difference between the two frames being too large, which would affect the user's judgment of shooting satisfaction.

[0109] In some embodiments, the acquisition module 410 is further configured to acquire, for each frame of the M frames of images to be processed, the optical flow difference value between the image to be processed and each of the remaining images to be processed.

[0110] The acquisition module 410 is further configured to acquire the mean value of the optical flow difference between the image to be processed and the remaining images to be processed in each frame, and use the mean value as the optical flow amplitude corresponding to the image to be processed.

[0111] Based on the embodiments of this application, after determining N frames of images to be processed by preset filtering conditions, the echo image determined from the N frames of images to be processed can effectively reduce the visual effect of perspective jump between the echo image and the target captured image, improve the similarity between the echo image displayed in front and the target captured image displayed behind, and avoid the difference between the two frames being too large, which would affect the user's judgment of shooting satisfaction.

[0112] In some embodiments, the image feature information includes: brightness difference degree, color difference degree, and detail richness;

[0113] The processing module 420 is further configured to determine the J-frame image to be processed from the N-frame images to be processed based on the brightness difference degree corresponding to the N-frame images to be processed respectively;

[0114] The processing module 420 is further configured to determine the K-frame images to be processed from the J-frame images to be processed based on the degree of color difference corresponding to the J-frame images to be processed respectively;

[0115] The processing module 420 is further configured to determine the echo image from the K frames of images to be processed based on the level of detail corresponding to each of the K frames of images to be processed.

[0116] Based on the embodiments of this application, a playback image is selected from N frames of images to be processed based on the degree of brightness difference, the degree of color difference, and the richness of detail. This effectively improves the similarity between the playback image displayed in front and the target captured image displayed behind, and reduces the changes in visual effect that occur after replacing the playback image with the target captured image.

[0117] In some embodiments, for each of the N frames of images to be processed, the degree of brightness difference corresponding to the image to be processed is used to characterize the brightness difference between the image to be processed and the remaining images to be processed in the N frames of images to be processed.

[0118] The acquisition module 410 is also used to acquire the brightness value of the image to be processed;

[0119] The acquisition module 410 is also used to acquire the average brightness value of the N frames of images to be processed;

[0120] The processing module 420 is further configured to determine the degree of brightness difference of the image to be processed based on the difference between the brightness value of the image to be processed and the average brightness value.

[0121] This effectively reduces the visual effect of brightness jumps between the displayed image and the target image, improves the similarity between the displayed image and the target image, and avoids excessive differences between the two frames affecting the user's judgment of shooting satisfaction.

[0122] In some embodiments, for each of the N frames of images to be processed, the degree of color difference corresponding to the image to be processed is used to characterize the color difference between the image to be processed and the remaining images to be processed in the N frames of images to be processed.

[0123] The acquisition module 410 is also used to acquire the hue value and saturation of the image to be processed;

[0124] The acquisition module 410 is also used to acquire the average hue value of the N frames of images to be processed, and the average saturation value of the N frames of images to be processed.

[0125] The processing module 420 is further configured to determine the degree of color difference corresponding to the image to be processed based on the difference between the hue value and the hue mean of the image to be processed, and the difference between the saturation of the image to be processed and the saturation mean.

[0126] This effectively reduces the visual effect of color jumps between the displayed image and the target image, improves the similarity between the displayed image and the target image, and avoids excessive differences between the two frames affecting the user's judgment of shooting satisfaction.

[0127] In some embodiments, for each of the N frames of images to be processed, the detail richness of the image to be processed is used to characterize the image detail richness of the image to be processed.

[0128] The processing module 420 is further configured to determine the discreteness of each pixel in the image to be processed based on the gray value of the pixel and the gray values ​​of a preset number of neighboring pixels.

[0129] The processing module 420 is further configured to determine the mean of dispersion based on the dispersion of each pixel in the image to be processed, wherein the mean of dispersion is used to characterize the richness of detail in the image to be processed.

[0130] According to the embodiments of this application, the richness of detail is improved when selecting a playback image from N frames of images to be processed and replacing it with the target captured image. This can increase the similarity between the playback image displayed in front and the target captured image displayed behind, effectively reducing the visual effect changes caused by the difference in image details and avoiding the impact of excessive differences between the two frames on the user's judgment of shooting satisfaction.

[0131] The image processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0132] The image processing device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0133] The image processing apparatus provided in this application embodiment can implement the various processes implemented in the image processing method embodiment, and will not be described again here to avoid repetition.

[0134] Optionally, such as Figure 5 As shown, this application embodiment also provides an electronic device 500, including a processor 501 and a memory 502. The memory 502 stores a program or instructions that can run on the processor 501. When the program or instructions are executed by the processor 501, they implement the various steps of the above-described image processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0135] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0136] Figure 6 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0137] The electronic device 600 includes, but is not limited to, components such as: radio frequency unit 601, network module 602, audio output unit 603, input unit 604, sensor 605, display unit 606, user input unit 607, interface unit 608, memory 609, and processor 610.

[0138] Those skilled in the art will understand that the electronic device 600 may also include a power supply (such as a battery) for powering various components. The power supply can be logically connected to the processor 610 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The electronic device structure shown in Figure x does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0139] Processor 610 is used to acquire M frames of images to be processed, and the M frames of images to be processed are used for fusion processing to obtain images of the target.

[0140] Processor 610 is used to determine N frames to be processed from M frames to be processed based on the degree of inter-frame difference corresponding to each of the M frames to be processed;

[0141] The processor 610 is also used to determine the echo image from the N frames of images to be processed based on the image feature information corresponding to each of the N frames of images to be processed. The image feature information includes at least one of the following: brightness difference degree, color difference degree, and detail richness.

[0142] Memory 609 is used to store the echoed image to the photo album;

[0143] The memory 609 is also used to replace the displayed image in the album with the target image when the target image is acquired.

[0144] Based on the embodiments of this application, a display image that can be shown to the user can be quickly selected from multiple frames of images to be processed, which can meet the user's need to view the captured image immediately, and can improve the similarity between the display image displayed in front and the target captured image displayed later, avoiding the large difference between the two frames affecting the user's judgment of the shooting satisfaction.

[0145] In some embodiments, the processor 610 is further configured to obtain the optical flow amplitude corresponding to each frame of the M frames of images to be processed, wherein the optical flow amplitude is used to characterize the degree of inter-frame difference between the image to be processed and the remaining images to be processed in the M frames of images to be processed.

[0146] The processor 610 is further configured to determine the N frames of images to be processed from the M frames of images to be processed based on the optical flow amplitude corresponding to each frame of images to be processed, wherein the optical flow amplitudes corresponding to the N frames of images to be processed respectively satisfy the target screening conditions.

[0147] Based on the embodiments of this application, after determining N frames of images to be processed by preset filtering conditions, the echo image determined from the N frames of images to be processed can improve the similarity between the echo image displayed in front and the target captured image displayed behind. This can effectively reduce the visual effect of perspective jump between the echo image and the target captured image, and avoid the difference between the two frames being too large, which would affect the user's judgment of shooting satisfaction.

[0148] In some embodiments, the processor 610 is further configured to, for each of the M frames of images to be processed, obtain an optical flow difference value between the image to be processed and each of the remaining images to be processed in each frame.

[0149] The processor 610 is further configured to obtain the mean value of the optical flow difference between the image to be processed and the remaining images to be processed in each frame, and use the mean value as the optical flow amplitude corresponding to the image to be processed.

[0150] Based on the embodiments of this application, after determining N frames of images to be processed by preset filtering conditions, the echo image determined from the N frames of images to be processed can improve the similarity between the echo image displayed in front and the target captured image displayed behind. This can effectively reduce the visual effect of perspective jump between the echo image and the target captured image, and avoid the difference between the two frames being too large, which would affect the user's judgment of shooting satisfaction.

[0151] In some embodiments, the image feature information includes: brightness difference degree, color difference degree, and detail richness;

[0152] The processor 610 is further configured to determine the J-frame image to be processed from the N-frame images to be processed based on the degree of brightness difference corresponding to the N-frame images to be processed respectively;

[0153] The processor 610 is further configured to determine a K-frame image to be processed from the J-frame images to be processed based on the degree of color difference corresponding to the J-frame images to be processed respectively;

[0154] The processor 610 is further configured to determine the echo image from the K frames to be processed based on the level of detail corresponding to each of the K frames to be processed.

[0155] Based on the embodiments of this application, the echo image is selected from N frames of images to be processed based on the degree of brightness difference, the degree of color difference, and the richness of detail. This can improve the similarity between the echo image displayed in front and the target image displayed behind, effectively reduce the changes in visual effect after replacing the echo image with the target image, and avoid the difference between the two frames being too large, which would affect the user's judgment of shooting satisfaction.

[0156] In some embodiments, for each of the N frames of images to be processed, the degree of brightness difference corresponding to the image to be processed is used to characterize the brightness difference between the image to be processed and the remaining images to be processed in the N frames of images to be processed.

[0157] The processor 610 is also used to acquire the brightness value of the image to be processed;

[0158] The processor 610 is also used to obtain the average brightness value of the N frames of images to be processed;

[0159] The processor 610 is further configured to determine the degree of brightness difference corresponding to the image to be processed based on the difference between the brightness value of the image to be processed and the average brightness value.

[0160] This improves the similarity between the displayed echo image and the target image, effectively reducing visual effects of brightness jumps between the echo image and the target image, and preventing excessive differences between the two frames from affecting the user's judgment of shooting satisfaction.

[0161] In some embodiments, for each of the N frames of images to be processed, the degree of color difference corresponding to the image to be processed is used to characterize the color difference between the image to be processed and the remaining images to be processed in the N frames of images to be processed.

[0162] The processor 610 is also configured to acquire the hue value and saturation of the image to be processed;

[0163] The processor 610 is also configured to obtain the average hue value of the N frames of images to be processed, and the average saturation value of the N frames of images to be processed.

[0164] The processor 610 is further configured to determine the degree of color difference corresponding to the image to be processed based on the difference between the hue value and the hue mean of the image to be processed, and the difference between the saturation of the image to be processed and the saturation mean of the image to be processed.

[0165] This improves the similarity between the displayed echo image and the target image, effectively reducing visual color jumps between the echo image and the target image, and preventing excessive differences between the two frames from affecting the user's judgment of shooting satisfaction.

[0166] In some embodiments, for each of the N frames of images to be processed, the detail richness of the image to be processed is used to characterize the image detail richness of the image to be processed.

[0167] The processor 610 is further configured to, for each pixel in the image to be processed, determine the discreteness corresponding to the pixel based on the gray value of the pixel and the gray values ​​of a preset number of neighboring pixels;

[0168] The processor 610 is further configured to determine the mean of dispersion based on the dispersion corresponding to each pixel in the image to be processed, wherein the mean of dispersion is used to characterize the richness of detail in the image to be processed.

[0169] According to the embodiments of this application, the richness of detail is improved when selecting a playback image from N frames of images to be processed and replacing it with the target captured image. This can increase the similarity between the playback image displayed in front and the target captured image displayed behind, effectively reducing the visual effect changes caused by the difference in image details and avoiding the large difference between the two frames affecting the user's judgment of shooting satisfaction.

[0170] It should be understood that, in this embodiment, the input unit 604 may include a graphics processing unit (GPU) 6041 and a microphone 6042. The GPU 6041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 606 may include a display panel 6061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 607 includes at least one of a touch panel 6071 and other input devices 6072. The touch panel 6071 is also called a touch screen. The touch panel 6071 may include a touch detection device and a touch controller. Other input devices 6072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0171] The memory 609 can be used to store software programs and various data. The memory 609 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 609 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 609 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0172] Processor 610 may include one or more processing units; optionally, processor 610 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 610.

[0173] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described image processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0174] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0175] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described image processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0176] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0177] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described image processing method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0178] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0179] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0180] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image processing method, characterized in that, The method includes: Acquire M frames of images to be processed, which are then fused into a target image. Based on the degree of inter-frame difference corresponding to the M frames to be processed, N frames to be processed are determined from the M frames to be processed; the N frames to be processed are the images to be processed with the smallest degree of inter-frame difference among the M frames to be processed. Based on the image feature information corresponding to the N frames of images to be processed, a display image is determined from the N frames of images to be processed. The image feature information includes at least one of the following: brightness difference degree, color difference degree, and detail richness. Wherein, if the image feature information includes the brightness difference degree, the display image is determined from the image to be processed with the smallest brightness difference degree; if the image feature information includes the color difference degree, the display image is determined from the image to be processed with the smallest color difference degree; if the image feature information includes the detail richness degree, the image to be processed with the largest detail richness degree is determined as the display image. Store the echoed image to the photo album; If the target image is obtained, the displayed image in the album is replaced with the target image.

2. The method according to claim 1, characterized in that, The step of determining N frames to be processed from the M frames to be processed based on the degree of inter-frame difference corresponding to each of the M frames to be processed includes: For each frame of the M frames to be processed, the optical flow amplitude corresponding to the image to be processed is obtained. The optical flow amplitude is used to characterize the degree of inter-frame difference between the image to be processed and the remaining images to be processed in the M frames. Based on the optical flow amplitude corresponding to each frame of the image to be processed, the N frames of the image to be processed are determined from the M frames of the image to be processed, and the optical flow amplitudes corresponding to the N frames of the image to be processed respectively meet the target screening conditions.

3. The method according to claim 2, characterized in that, The step of obtaining the optical flow amplitude corresponding to each frame of the M frames to be processed includes: For each of the M frames of images to be processed, obtain the optical flow difference value between the image to be processed and the remaining images to be processed in each frame; The mean value of the optical flow difference between the image to be processed and the remaining images to be processed in each frame is obtained, and the mean value is used as the optical flow amplitude corresponding to the image to be processed.

4. The method according to claim 1, characterized in that, When the image feature information includes the degree of brightness difference, the degree of color difference, and the richness of detail; The step of determining the echo image from the N frames of images to be processed based on the image feature information corresponding to each of the N frames includes: Based on the brightness difference corresponding to the N frames of images to be processed, determine the J frames of images to be processed from the N frames of images to be processed; Based on the degree of color difference corresponding to the J frames to be processed, determine the K frames to be processed from the J frames to be processed; The echo image is determined from the K frames of images to be processed based on the level of detail corresponding to each of the K frames.

5. The method according to claim 1, characterized in that, For each of the N frames of images to be processed, the degree of brightness difference corresponding to the image to be processed is used to characterize the brightness difference between the image to be processed and the remaining images to be processed in the N frames of images to be processed; The method further includes: Obtain the brightness value of the image to be processed; Obtain the average brightness value of the N frames of images to be processed; The degree of brightness difference in the image to be processed is determined based on the difference between the brightness value of the image to be processed and the average brightness value.

6. The method according to claim 1, characterized in that, For each of the N frames of images to be processed, the degree of color difference corresponding to the image to be processed is used to characterize the color difference between the image to be processed and the remaining images to be processed in the N frames of images to be processed; The method further includes: Obtain the hue and saturation values ​​of the image to be processed; Obtain the average hue value and the average saturation value of the N frames of images to be processed; The degree of color difference in the image to be processed is determined based on the difference between the hue value and the mean hue value of the image to be processed, and the difference between the saturation value and the mean saturation value of the image to be processed.

7. The method according to claim 1, characterized in that, For each of the N frames of images to be processed, the level of detail corresponding to the image to be processed is used to characterize the level of image detail of the image to be processed; the method further includes: For each pixel in the image to be processed, the dispersion of the pixel is determined based on the gray value of the pixel and the gray values ​​of a preset number of neighboring pixels; the mean dispersion is determined based on the dispersion of each pixel in the image to be processed, and the mean dispersion is used to characterize the richness of detail in the image to be processed.

8. An image processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire M frames of images to be processed, which are then fused into a target image. The processing module is used to determine N frames to be processed from the M frames to be processed based on the degree of inter-frame difference corresponding to each of the M frames to be processed; the N frames to be processed are the images to be processed with the smallest degree of inter-frame difference among the M frames to be processed. The processing module is further configured to determine a display image from the N frames of images to be processed based on the image feature information corresponding to each of the N frames of images to be processed. The image feature information includes at least one of the following: brightness difference degree, color difference degree, and detail richness; wherein, if the image feature information includes the brightness difference degree, the display image is determined from the image to be processed with the smallest brightness difference degree; if the image feature information includes the color difference degree, the display image is determined from the image to be processed with the smallest color difference degree; if the image feature information includes the detail richness degree, the image to be processed with the largest detail richness degree is determined as the display image. A storage module is used to store the echoed image to a photo album; The storage module is further configured to replace the displayed image in the album with the target image when the target image is obtained.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the image processing method as described in any one of claims 1-7.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the image processing method as described in any one of claims 1-7.

Images